Added ZeroToOneFixedAspect and PlusMinusOneFixedAspect to ScaleMode

README.md

@@ -184,7 +184,7 @@ Just comment on GitHub or make a pull request and we will answer as soon as poss
 
 
 ## Citation
-Please cite these papers in your publications if it helps your research (the face keypoint detector was trained using the procedure described in [Simon et al. 2017] for hands):
+Please cite these papers in your publications if it helps your research. The body-foot model and any additional functionality (calibration, 3-D reconstruction, etc.) use `[Cao et al. 2018]`; the hand and face keypoint detectors use `[Simon et al. 2017]` (the face detector was trained using the same procedure than for hands); and the old (deprecated) body-only model uses `[Cao et al. 2017]`.
 
     @inproceedings{cao2018openpose,
       author = {Zhe Cao and Gines Hidalgo and Tomas Simon and Shih-En Wei and Yaser Sheikh},
@@ -193,17 +193,17 @@ Please cite these papers in your publications if it helps your research (the fac
       year = {2018}
     }
 
-    @inproceedings{cao2017realtime,
-      author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
+    @inproceedings{simon2017hand,
+      author = {Tomas Simon and Hanbyul Joo and Iain Matthews and Yaser Sheikh},
       booktitle = {CVPR},
-      title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
+      title = {Hand Keypoint Detection in Single Images using Multiview Bootstrapping},
       year = {2017}
     }
 
-    @inproceedings{simon2017hand,
-      author = {Tomas Simon and Hanbyul Joo and Iain Matthews and Yaser Sheikh},
+    @inproceedings{cao2017realtime,
+      author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
       booktitle = {CVPR},
-      title = {Hand Keypoint Detection in Single Images using Multiview Bootstrapping},
+      title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
       year = {2017}
     }
 
@@ -217,8 +217,8 @@ Please cite these papers in your publications if it helps your research (the fac
 Links to the papers:
 
 - [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://arxiv.org/abs/1812.08008)
-- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://arxiv.org/abs/1611.08050)
 - [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://arxiv.org/abs/1704.07809)
+- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://arxiv.org/abs/1611.08050)
 - [Convolutional Pose Machines](https://arxiv.org/abs/1602.00134)
 
 

doc/release_notes.md

@@ -391,6 +391,7 @@ OpenPose Library - Release Notes
     4. Default OpenCV version for Windows upgraded to version 4.1.1, extracted from their oficial website: section `Releases`, subsection `OpenCV - 4.1.1`, `Windows` version.
     5. In all `*.cpp` files, their include of their analog `*.hpp` file has been moved to the first line of those `*.cpp` files to slightly speed up compiling time.
     6. String is used in `include/openpose/wrapper/` to avoid std::string to cause errors for using diferent std DLLs.
+    7. Added `ScaleMode::ZeroToOneFixedAspect` and `ScaleMode::PlusMinusOneFixedAspect`. Compared to `ZeroToOne` and `PlusMinusOne`, the new ones also preserve the aspect ratio of each axis.
 2. Functions or parameters renamed:
     1. All headers moved into `openpose_private`, all 3rd-party library calls in headers, and std::string calls in `include/openpose/wrapper/`.
     2. Renamed `dLog()` as `opLogIfDebug()`, `log()` as `opLog()`, `check()` as `checkBool()`, and also renamed all the `checkX()` functions in `include/openpose/utilities/check.hpp`. This avoids compiling crashes when exporting OpenPose to other projects which contain other 3rd-party libraries that define functions with the same popular names with `#define`.

examples/tests/handFromJsonTest.cpp

@@ -52,7 +52,7 @@ int handFromJsonTest()
         op::WrapperStructPose wrapperStructPose{
             op::PoseMode::Disabled, op::flagsToPoint("656x368"), op::flagsToPoint("1280x720"), op::ScaleMode::InputResolution,
             FLAGS_num_gpu, FLAGS_num_gpu_start, 1, 0.15f, op::RenderMode::None, op::PoseModel::BODY_25, true, 0.f, 0.f,
-            0, "models/", {}, op::ScaleMode::ZeroToOne, false, 0.05f, -1, false};
+            0, "models/", {}, op::ScaleMode::ZeroToOneAspectRatio, false, 0.05f, -1, false};
         wrapperStructPose.modelFolder = op::String(FLAGS_model_folder);
         // Hand configuration (use op::WrapperStructHand{} to disable it)
         const op::WrapperStructHand wrapperStructHand{

include/openpose/core/enumClasses.hpp

@@ -9,7 +9,9 @@ namespace op
         NetOutputResolution,
         OutputResolution,
         ZeroToOne, // [0, 1]
+        ZeroToOneFixedAspect, // [0, 1]
         PlusMinusOne, // [-1, 1]
+        PlusMinusOneFixedAspect, // [-1, 1]
         UnsignedChar, // [0, 255]
         NoScale,
     };

include/openpose/face/faceExtractorCaffe.hpp

@@ -21,7 +21,7 @@ namespace op
         FaceExtractorCaffe(const Point<int>& netInputSize, const Point<int>& netOutputSize,
                            const std::string& modelFolder, const int gpuId,
                            const std::vector<HeatMapType>& heatMapTypes = {},
-                           const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne,
+                           const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect,
                            const bool enableGoogleLogging = true);
 
         virtual ~FaceExtractorCaffe();

include/openpose/face/faceExtractorNet.hpp

@@ -20,7 +20,7 @@ namespace op
          */
         explicit FaceExtractorNet(const Point<int>& netInputSize, const Point<int>& netOutputSize,
                                   const std::vector<HeatMapType>& heatMapTypes = {},
-                                  const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne);
+                                  const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect);
 
         /**
          * Virtual destructor of the HandExtractor class.

include/openpose/hand/handExtractorCaffe.hpp

@@ -27,7 +27,7 @@ namespace op
                            const std::string& modelFolder, const int gpuId,
                            const int numberScales = 1, const float rangeScales = 0.4f,
                            const std::vector<HeatMapType>& heatMapTypes = {},
-                           const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne,
+                           const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect,
                            const bool enableGoogleLogging = true);
 
         /**

include/openpose/hand/handExtractorNet.hpp

@@ -24,7 +24,7 @@ namespace op
         explicit HandExtractorNet(const Point<int>& netInputSize, const Point<int>& netOutputSize,
                                   const int numberScales = 1, const float rangeScales = 0.4f,
                                   const std::vector<HeatMapType>& heatMapTypes = {},
-                                  const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne);
+                                  const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect);
 
         /**
          * Virtual destructor of the HandExtractorNet class.

include/openpose/pose/poseExtractorCaffe.hpp

@@ -19,7 +19,7 @@ namespace op
         PoseExtractorCaffe(
             const PoseModel poseModel, const std::string& modelFolder, const int gpuId,
             const std::vector<HeatMapType>& heatMapTypes = {},
-            const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne,
+            const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect,
             const bool addPartCandidates = false, const bool maximizePositives = false,
             const std::string& protoTxtPath = "", const std::string& caffeModelPath = "",
             const float upsamplingRatio = 0.f, const bool enableNet = true,

include/openpose/pose/poseExtractorNet.hpp

@@ -13,7 +13,7 @@ namespace op
     public:
         PoseExtractorNet(const PoseModel poseModel,
                          const std::vector<HeatMapType>& heatMapTypes = {},
-                         const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOne,
+                         const ScaleMode heatMapScaleMode = ScaleMode::ZeroToOneFixedAspect,
                          const bool addPartCandidates = false,
                          const bool maximizePositives = false);
 

include/openpose/wrapper/wrapperStructPose.hpp

@@ -43,7 +43,7 @@ namespace op
          * Final scale of the Array<float> Datum.poseKeypoints and the writen pose data.
          * The final Datum.poseKeypoints can be scaled with respect to input size (ScaleMode::InputResolution), net
          * output size (ScaleMode::NetOutputResolution), output rendering size (ScaleMode::OutputResolution), from 0 to
-         * 1 (ScaleMode::ZeroToOne), and -1 to 1 (ScaleMode::PlusMinusOne).
+         * 1 (ScaleMode::ZeroToOne(FixedAspect)), and -1 to 1 (ScaleMode::PlusMinusOne(FixedAspect)).
          */
         ScaleMode keypointScaleMode;
 
@@ -129,8 +129,8 @@ namespace op
 
         /**
          * Scale of the Datum.heatmaps.
-         * Select ScaleMode::ZeroToOne for range [0,1], ScaleMode::PlusMinusOne for [-1,1] and ScaleMode::UnsignedChar
-         * for [0, 255].
+         * Select ScaleMode::ZeroToOne(FixedAspect) for range [0,1], ScaleMode::PlusMinusOne(FixedAspect) for [-1,1]
+         * and ScaleMode::UnsignedChar for [0, 255].
          * If heatMapTypes.empty(), then this parameters makes no effect.
          */
         ScaleMode heatMapScaleMode;

src/openpose/core/keypointScaler.cpp

 #include <openpose/core/keypointScaler.hpp>
+#include <openpose/utilities/fastMath.hpp>
 #include <openpose/utilities/keypoint.hpp>
 
 namespace op
@@ -19,10 +20,22 @@ namespace op
             else if (scaleMode == ScaleMode::ZeroToOne)
                 return Rectangle<float>{0.f, 0.f, 1.f / ((float)producerSize.x - 1.f),
                                         1.f / ((float)producerSize.y - 1.f)};
+            // [0,1]
+            else if (scaleMode == ScaleMode::ZeroToOneFixedAspect)
+            {
+                const float invMaxProducerSize = 1.f / ((float)fastMax(producerSize.x, producerSize.y) - 1.f);
+                return Rectangle<float>{0.f, 0.f, invMaxProducerSize, invMaxProducerSize};
+            }
             // [-1,1]
             else if (scaleMode == ScaleMode::PlusMinusOne)
                 return Rectangle<float>{-1.f, -1.f, 2.f / ((float)producerSize.x - 1.f),
                                         2.f / ((float)producerSize.y - 1.f)};
+            // [-1,1]
+            else if (scaleMode == ScaleMode::PlusMinusOneFixedAspect)
+            {
+                const float invMaxProducerSize = 2.f / ((float)fastMax(producerSize.x, producerSize.y) - 1.f);
+                return Rectangle<float>{-1.f, -1.f, invMaxProducerSize, invMaxProducerSize};
+            }
             // InputResolution
             else if (scaleMode == ScaleMode::InputResolution)
                 return Rectangle<float>{0.f, 0.f, 1.f, 1.f};

src/openpose/face/faceExtractorCaffe.cpp

@@ -57,7 +57,8 @@ namespace op
                     std::copy(heatMapsGpuPtr, heatMapsGpuPtr + volumeBodyParts, heatMapsPtr);
                 #endif
                 // Change from [0,1] to [-1,1]
-                if (heatMapScaleMode == ScaleMode::PlusMinusOne)
+                if (heatMapScaleMode == ScaleMode::PlusMinusOne
+                    || heatMapScaleMode == ScaleMode::PlusMinusOneFixedAspect)
                     for (auto i = 0u ; i < volumeBodyParts ; i++)
                         heatMapsPtr[i] = fastTruncate(heatMapsPtr[i]) * 2.f - 1.f;
                 // [0, 255]

src/openpose/face/faceExtractorNet.cpp

@@ -15,10 +15,13 @@ namespace op
         try
         {
             // Error check
-            if (mHeatMapScaleMode != ScaleMode::ZeroToOne && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+            if (mHeatMapScaleMode != ScaleMode::ZeroToOne
+                && mHeatMapScaleMode != ScaleMode::ZeroToOneFixedAspect
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOneFixedAspect
                 && mHeatMapScaleMode != ScaleMode::UnsignedChar)
-                error("The ScaleMode heatMapScaleMode must be ZeroToOne, PlusMinusOne or UnsignedChar.",
-                      __LINE__, __FUNCTION__, __FILE__);
+                error("The ScaleMode heatMapScaleMode must be ZeroToOne(FixedAspect), PlusMinusOne(FixedAspect)"
+                    " or UnsignedChar.", __LINE__, __FUNCTION__, __FILE__);
             checkEqual(
                 netOutputSize.x, netInputSize.x, "Net input and output size must be equal.",
                 __LINE__, __FUNCTION__, __FILE__);

src/openpose/hand/handExtractorCaffe.cpp

@@ -141,7 +141,8 @@ namespace op
                     std::copy(heatMapsGpuPtr, heatMapsGpuPtr + volumeBodyParts, heatMapsPtr);
                 #endif
                 // Change from [0,1] to [-1,1]
-                if (heatMapScaleMode == ScaleMode::PlusMinusOne)
+                if (heatMapScaleMode == ScaleMode::PlusMinusOne
+                    || heatMapScaleMode == ScaleMode::PlusMinusOneFixedAspect)
                     for (auto i = 0u ; i < volumeBodyParts ; i++)
                         heatMapsPtr[i] = fastTruncate(heatMapsPtr[i]) * 2.f - 1.f;
                 // [0, 255]

src/openpose/hand/handExtractorNet.cpp

@@ -17,10 +17,13 @@ namespace op
         try
         {
             // Error check
-            if (mHeatMapScaleMode != ScaleMode::ZeroToOne && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+            if (mHeatMapScaleMode != ScaleMode::ZeroToOne
+                && mHeatMapScaleMode != ScaleMode::ZeroToOneFixedAspect
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOneFixedAspect
                 && mHeatMapScaleMode != ScaleMode::UnsignedChar)
-                error("The ScaleMode heatMapScaleMode must be ZeroToOne, PlusMinusOne or UnsignedChar.",
-                      __LINE__, __FUNCTION__, __FILE__);
+                error("The ScaleMode heatMapScaleMode must be ZeroToOne, ZeroToOneFixedAspect, PlusMinusOne,"
+                    " PlusMinusOneFixedAspect or UnsignedChar.", __LINE__, __FUNCTION__, __FILE__);
             checkEqual(
                 netOutputSize.x, netInputSize.x, "Net input and output size must be equal.",
                 __LINE__, __FUNCTION__, __FILE__);

src/openpose/pose/poseExtractorNet.cpp

@@ -57,10 +57,13 @@ namespace op
         try
         {
             // Error check
-            if (mHeatMapScaleMode != ScaleMode::ZeroToOne && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+            if (mHeatMapScaleMode != ScaleMode::ZeroToOne
+                && mHeatMapScaleMode != ScaleMode::ZeroToOneFixedAspect
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOne
+                && mHeatMapScaleMode != ScaleMode::PlusMinusOneFixedAspect
                 && mHeatMapScaleMode != ScaleMode::UnsignedChar && mHeatMapScaleMode != ScaleMode::NoScale)
-                error("The ScaleMode heatMapScaleMode must be ZeroToOne, PlusMinusOne, UnsignedChar, or NoScale.",
-                      __LINE__, __FUNCTION__, __FILE__);
+                error("The ScaleMode heatMapScaleMode must be ZeroToOne, ZeroToOneFixedAspect, PlusMinusOne,"
+                    " PlusMinusOneFixedAspect or UnsignedChar.", __LINE__, __FUNCTION__, __FILE__);
 
             // Properties - Init to 0
             for (auto& property : mProperties)
@@ -136,7 +139,8 @@ namespace op
                     if (mHeatMapScaleMode != ScaleMode::NoScale)
                     {
                         // Change from [0,1] to [-1,1]
-                        if (mHeatMapScaleMode == ScaleMode::PlusMinusOne)
+                        if (mHeatMapScaleMode == ScaleMode::PlusMinusOne
+                            || mHeatMapScaleMode == ScaleMode::PlusMinusOneFixedAspect)
                             for (auto i = 0u ; i < volumeBodyParts ; i++)
                                 heatMaps[i] = fastTruncate(heatMaps[i]) * 2.f - 1.f;
                         // [0, 255]
@@ -168,7 +172,8 @@ namespace op
                         if (mHeatMapScaleMode != ScaleMode::NoScale)
                         {
                             // Change from [0,1] to [-1,1]
-                            if (mHeatMapScaleMode == ScaleMode::PlusMinusOne)
+                            if (mHeatMapScaleMode == ScaleMode::PlusMinusOne
+                                || mHeatMapScaleMode == ScaleMode::PlusMinusOneFixedAspect)
                                 for (auto i = 0u ; i < channelOffset ; i++)
                                     heatMapsPtr[i] = fastTruncate(heatMapsPtr[i]) * 2.f - 1.f;
                             // [0, 255]
@@ -205,7 +210,8 @@ namespace op
                     if (mHeatMapScaleMode != ScaleMode::NoScale)
                     {
                         // Change from [-1,1] to [0,1]. Note that PAFs are in [-1,1]
-                        if (mHeatMapScaleMode == ScaleMode::ZeroToOne)
+                        if (mHeatMapScaleMode == ScaleMode::ZeroToOne
+                            || mHeatMapScaleMode == ScaleMode::ZeroToOneFixedAspect)
                             for (auto i = 0u ; i < volumePAFs ; i++)
                                 heatMapsPtr[i] = fastTruncate(heatMapsPtr[i], -1.f) * 0.5f + 0.5f;
                         // [0, 255]

src/openpose/utilities/flagsToOpenPose.cpp

@@ -90,10 +90,15 @@ namespace op
                 return ScaleMode::ZeroToOne;
             else if (keypointScaleMode == 4)
                 return ScaleMode::PlusMinusOne;
+            else if (keypointScaleMode == 5)
+                return ScaleMode::ZeroToOneAspectRatio;
+            else if (keypointScaleMode == 6)
+                return ScaleMode::PlusMinusOneAspectRatio;
             // else
-            const std::string message = "Integer does not correspond to any scale mode: (0, 1, 2, 3, 4) for"
-                                        " (InputResolution, NetOutputResolution, OutputResolution, ZeroToOne,"
-                                        " PlusMinusOne).";
+            const std::string message = "Integer does not correspond to any scale mode: set to (0, 1, 2, 3, 4, 5, 6)"
+                                        " for (InputResolution, NetOutputResolution, OutputResolution, ZeroToOne,"
+                                        " PlusMinusOne, ZeroToOneAspectRatio, PlusMinusOneAspectRatio),"
+                                        " respectively.";
             error(message, __LINE__, __FUNCTION__, __FILE__);
             return ScaleMode::InputResolution;
         }
@@ -117,16 +122,21 @@ namespace op
                 return ScaleMode::UnsignedChar;
             else if (heatMapScaleMode == 3)
                 return ScaleMode::NoScale;
+            else if (heatMapScaleMode == 4)
+                return ScaleMode::ZeroToOneAspectRatio;
+            else if (heatMapScaleMode == 5)
+                return ScaleMode::PlusMinusOneAspectRatio;
             // else
-            const std::string message = "Integer does not correspond to any scale mode: (0, 1, 2, 3) for"
-                                        " (PlusMinusOne, ZeroToOne, UnsignedChar, NoScale).";
+            const std::string message = "Integer does not correspond to any scale mode: set to (0, 1, 2, 3, 4, 5)"
+                                        " for (PlusMinusOne, ZeroToOne, UnsignedChar, NoScale, PlusMinusOneAspectRatio,"
+                                        " ZeroToOneAspectRatio), respectively.";
             error(message, __LINE__, __FUNCTION__, __FILE__);
-            return ScaleMode::PlusMinusOne;
+            return ScaleMode::PlusMinusOneAspectRatio;
         }
         catch (const std::exception& e)
         {
             error(e.what(), __LINE__, __FUNCTION__, __FILE__);
-            return ScaleMode::PlusMinusOne;
+            return ScaleMode::PlusMinusOneAspectRatio;
         }
     }