Flag write_json includes 3-D keypoints

doc/3d_reconstruction_demo.md

@@ -97,7 +97,14 @@ It should be similar to the following image.
 
 
 ## Using a Different Camera Brand
-You can copy and modify the OpenPose 3-D demo to use any camera brand, by modifying the frames producer. For that, you would need to provide your custom code to retrieve synchronized images from your cameras, as well as their intrinsic and extrinsic camera parameters.
+You can copy and modify the OpenPose 3-D demo to use any camera brand by:
+
+1. You can optionally turn off the `WITH_FLIR_CAMERA` while compiling CMake.
+2. Copy any of the `examples/tutorial_wrapper/*.cpp` examples (we recommend `2_user_synchronous.cpp`).
+3. Modify `WUserInput` and add your custom code there. Your code should fill `Datum::name`, `Datum::cameraMatrix`, `Datum::cvInputData`, and `Datum::cvOutputData` (fill cvOutputData = cvInputData).
+4. Remove `WUserPostProcessing` and `WUserOutput` (unless you want to have your custom post-processing and/or output).
+
+Note that your custom code should retrieve synchronized images from your cameras or any other source, as well as their intrinsic and extrinsic camera parameters.
 
 
 

doc/demo_overview.md

@@ -213,7 +213,7 @@ Each flag is divided into flag name, default value, and description.
 - DEFINE_string(write_images,             "",             "Directory to write rendered frames in `write_images_format` image format.");
 - DEFINE_string(write_images_format,      "png",          "File extension and format for `write_images`, e.g. png, jpg or bmp. Check the OpenCV function cv::imwrite for all compatible extensions.");
 - DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
-- DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+- DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 - DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 - DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag must be enabled.");
 - DEFINE_string(write_heatmaps_format,    "png",          "File extension and format for `write_heatmaps`, analogous to `write_images_format`. For lossless compression, recommended `png` for integer `heatmaps_scale` and `float` for floating values.");

doc/quick_start.md

@@ -125,13 +125,20 @@ build\x64\Release\OpenPoseDemo.exe --flir_camera --3d --number_people_max 1
 build\x64\Release\OpenPoseDemo.exe --flir_camera --3d --number_people_max 1 --face --hand
 ```
 
-2. Saving stereo camera images for a later post-processing:
+2. Saving 3-D keypoints
 ```
 # Ubuntu (same flags for Windows version)
-./build/examples/openpose/openpose.bin --flir_camera --3d --number_people_max 1 --num_gpu 0 --write_images output_folder_path/
+./build/examples/openpose/openpose.bin --flir_camera --3d --number_people_max 1 --write_json output_folder_path/
 ```
 
-3. Reading previouly saved stereo camera images and processing them:
+3. Saving stereo camera images fast (without keypoint detection) for later post-processing
+```
+# Ubuntu (same flags for Windows version)
+# Note: saving in PNG rather than JPG will improve image quality, but slow down FPS (depending on hard disk writing speed and camera number)
+./build/examples/openpose/openpose.bin --flir_camera --num_gpu 0 --write_images output_folder_path/
+```
+
+4. Reading and processing previouly saved stereo camera images
 ```
 # Ubuntu (same flags for Windows version)
 # Optionally add `--face` and/or `--hand` to include face and/or hands

doc/release_notes.md

@@ -185,6 +185,8 @@ OpenPose Library - Release Notes
     13. 3-D reconstruction module forces the user to set `number_people_max 1` to avoid errors (as it assumes only 1 person per image).
     14. Removed old `windows/` version. CMake is the only Windows version available.
     15. Camera parameters (flir camera) are read from disk at runtime rather than being compiled.
+    16. 3-D reconstruction module can be implemented with different camera brands or custom image sources.
+    16. Flag `--write_json` includes 3-D keypoints.
 2. Functions or parameters renamed:
     1. Flag `no_display` renamed as `display`, able to select between `NoDisplay`, `Display2D`, `Display3D`, and `DisplayAll`.
     2. 3-D reconstruction demo is now inside the OpenPose demo binary.

examples/openpose/openpose.cpp

@@ -194,7 +194,7 @@ DEFINE_string(write_images_format,      "png",          "File extension and form
 DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the"
                                                         " final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
 DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose"
-                                                        " keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+                                                        " keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag"
                                                         " must be enabled.");

examples/tutorial_add_module/1_custom_post_processing.cpp

@@ -201,7 +201,7 @@ DEFINE_string(write_images_format,      "png",          "File extension and form
 DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the"
                                                         " final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
 DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose"
-                                                        " keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+                                                        " keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag"
                                                         " must be enabled.");

examples/tutorial_wrapper/1_user_asynchronous_output.cpp

@@ -184,7 +184,7 @@ DEFINE_string(write_images_format,      "png",          "File extension and form
 DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the"
                                                         " final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
 DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose"
-                                                        " keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+                                                        " keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag"
                                                         " must be enabled.");

examples/tutorial_wrapper/2_user_synchronous.cpp

@@ -165,7 +165,7 @@ DEFINE_string(write_images_format,      "png",          "File extension and form
 DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the"
                                                         " final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
 DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose"
-                                                        " keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+                                                        " keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag"
                                                         " must be enabled.");

examples/tutorial_wrapper/3_user_asynchronous.cpp

@@ -165,7 +165,7 @@ DEFINE_string(write_images_format,      "png",          "File extension and form
 DEFINE_string(write_video,              "",             "Full file path to write rendered frames in motion JPEG video format. It might fail if the"
                                                         " final path does not finish in `.avi`. It internally uses cv::VideoWriter.");
 DEFINE_string(write_json,               "",             "Directory to write OpenPose output in JSON format. It includes body, hand, and face pose"
-                                                        " keypoints, as well as pose candidates (if `--part_candidates` enabled).");
+                                                        " keypoints (2-D and 3-D), as well as pose candidates (if `--part_candidates` enabled).");
 DEFINE_string(write_coco_json,          "",             "Full file path to write people pose data with JSON COCO validation format.");
 DEFINE_string(write_heatmaps,           "",             "Directory to write body pose heatmaps in PNG format. At least 1 `add_heatmaps_X` flag"
                                                         " must be enabled.");

include/openpose/filestream/wPeopleJsonSaver.hpp

@@ -66,10 +66,16 @@ namespace op
                     const auto fileName = baseFileName + (i != 0 ? "_" + std::to_string(i) : "");
 
                     const std::vector<std::pair<Array<float>, std::string>> keypointVector{
+                        // 2D
                         std::make_pair(tDatum.poseKeypoints, "pose_keypoints"),
                         std::make_pair(tDatum.faceKeypoints, "face_keypoints"),
                         std::make_pair(tDatum.handKeypoints[0], "hand_left_keypoints"),
-                        std::make_pair(tDatum.handKeypoints[1], "hand_right_keypoints")
+                        std::make_pair(tDatum.handKeypoints[1], "hand_right_keypoints"),
+                        // 3D
+                        std::make_pair(tDatum.poseKeypoints3D, "pose_keypoints_3d"),
+                        std::make_pair(tDatum.faceKeypoints3D, "face_keypoints_3d"),
+                        std::make_pair(tDatum.handKeypoints3D[0], "hand_left_keypoints_3d"),
+                        std::make_pair(tDatum.handKeypoints3D[1], "hand_right_keypoints_3d")
                     };
                     // Save keypoints
                     spPeopleJsonSaver->save(keypointVector, tDatum.poseCandidates, fileName, humanReadable);

include/openpose/gui/wGui.hpp

@@ -59,22 +59,22 @@ namespace op
             // tDatums might be empty but we still wanna update the GUI
             if (tDatums != nullptr)
             {
-                // Check tDatums->size() == 1
-                if (tDatums->size() > 1)
-                    error("Only implemented for tDatums->size() == 1", __LINE__, __FUNCTION__, __FILE__);
                 // Debugging log
                 dLog("", Priority::Low, __LINE__, __FUNCTION__, __FILE__);
                 // Profiling speed
                 const auto profilerKey = Profiler::timerInit(__LINE__, __FUNCTION__, __FILE__);
-                // T* to T
-                auto& tDatumsNoPtr = *tDatums;
-                // Set image
-                const auto cvOutputData = (!tDatumsNoPtr.empty() ? tDatumsNoPtr[0].cvOutputData : cv::Mat());
-                spGui->setImage(cvOutputData);
+                // Update cvMat
+                if (!tDatums->empty())
+                {
+                    std::vector<cv::Mat> cvOutputDatas;
+                    for (auto& tDatum : *tDatums)
+                        cvOutputDatas.emplace_back(tDatum.cvOutputData);
+                    spGui->setImage(cvOutputDatas);
+                }
                 // Refresh GUI
                 spGui->update();
                 // Profiling speed
-                if (!tDatumsNoPtr.empty())
+                if (!tDatums->empty())
                 {
                     Profiler::timerEnd(profilerKey);
                     Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__);

include/openpose/gui/wGui3D.hpp

@@ -63,21 +63,24 @@ namespace op
                 dLog("", Priority::Low, __LINE__, __FUNCTION__, __FILE__);
                 // Profiling speed
                 const auto profilerKey = Profiler::timerInit(__LINE__, __FUNCTION__, __FILE__);
+                // Update cvMat & keypoints
                 if (!tDatums->empty())
                 {
-                    // T* to T
-                    auto& tDatum = (*tDatums)[0];
                     // Update cvMat
                     std::vector<cv::Mat> cvOutputDatas;
                     for (auto& tDatum : *tDatums)
                         cvOutputDatas.emplace_back(tDatum.cvOutputData);
                     spGui3D->setImage(cvOutputDatas);
                     // Update keypoints
+                    auto& tDatum = (*tDatums)[0];
                     spGui3D->setKeypoints(tDatum.poseKeypoints3D, tDatum.faceKeypoints3D, tDatum.handKeypoints3D[0],
                                           tDatum.handKeypoints3D[1]);
-                    // Refresh/update GUI
-                    spGui3D->update();
-                    // Profiling speed
+                }
+                // Refresh/update GUI
+                spGui3D->update();
+                // Profiling speed
+                if (!tDatums->empty())
+                {
                     Profiler::timerEnd(profilerKey);
                     Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__);
                 }

src/openpose/filestream/fileStream.cpp

@@ -40,21 +40,22 @@ namespace op
                 {
                     const auto& keypoints = keypointVector[vectorIndex].first;
                     const auto& keypointName = keypointVector[vectorIndex].second;
-                    const auto numberBodyParts = keypoints.getSize(1);
+                    const auto numberElementsPerRaw = keypoints.getSize(1) * keypoints.getSize(2);
                     jsonOfstream.key(keypointName);
                     jsonOfstream.arrayOpen();
                     // Body parts
-                    for (auto bodyPart = 0 ; bodyPart < numberBodyParts ; bodyPart++)
+                    if (numberElementsPerRaw > 0)
                     {
-                        const auto finalIndex = 3*(person*numberBodyParts + bodyPart);
-                        jsonOfstream.plainText(keypoints[finalIndex]);
-                        jsonOfstream.comma();
-                        jsonOfstream.plainText(keypoints[finalIndex+1]);
-                        jsonOfstream.comma();
-                        jsonOfstream.plainText(keypoints[finalIndex+2]);
-                        if (bodyPart < numberBodyParts-1)
+                        const auto finalIndex = person*numberElementsPerRaw;
+                        for (auto element = 0 ; element < numberElementsPerRaw - 1 ; element++)
+                        {
+                            jsonOfstream.plainText(keypoints[finalIndex + element]);
                             jsonOfstream.comma();
+                        }
+                        // Last element (no comma)
+                        jsonOfstream.plainText(keypoints[finalIndex + numberElementsPerRaw - 1]);
                     }
+                    // Close array
                     jsonOfstream.arrayClose();
                     if (vectorIndex < keypointVector.size()-1)
                         jsonOfstream.comma();
@@ -313,7 +314,11 @@ namespace op
             JsonOfstream jsonOfstream{fileName, humanReadable};
             jsonOfstream.objectOpen();
             // Add version
-            jsonOfstream.version("1.1");
+            // Version 0.1: Body keypoints (2-D)
+            // Version 1.0: Added face and hands (2-D)
+            // Version 1.1: Added candidates
+            // Version 1.2: Added body, face, and hands (3-D)
+            jsonOfstream.version("1.2");
             jsonOfstream.comma();
             // Add people keypoints
             addKeypointsToJson(jsonOfstream, keypointVector);

src/openpose/producer/flirReader.cpp

@@ -102,6 +102,7 @@ namespace op
     {
         try
         {
+            mFrameNameCounter++; // Simple counter: 0,1,2,3,...
             return mSpinnakerWrapper.getRawFrames();
         }
         catch (const std::exception& e)

src/openpose/producer/webcamReader.cpp

@@ -15,6 +15,7 @@ namespace op
         {
             if (isOpened())
             {
+                mFrameNameCounter = 0;
                 if (webcamResolution != Point<int>{})
                 {
                     set(CV_CAP_PROP_FRAME_WIDTH, webcamResolution.x);