dnn_pose_estimation

Estimate the human poses with MediaPipe person detection and pose estimation.

Human pose estimation with MediaPipe BlazePose of opencv_zoo (Apache 2.0): the persons are detected by the person detection model, then the 33 landmarks of each person are estimated by the pose estimation model, on the image rotated so that the person stands upright. As MediaPipe, the persons are tracked by the auxiliary landmarks of their poses in the previous image, and the person detection runs every detection_interval images to find the new persons, or when no person is tracked. The landmarks of the tracked persons are smoothed by One Euro filter, as MediaPipe. The models need OpenCV 4.9 or later (OpenCV 4.6 cannot read them). ~skeletons has a Skeleton for each person: label “person”, id the id while tracked, score the confidence of the pose, rect the bounding box of the landmarks, and keypoints the 33 landmarks in the image coordinates with their visibility and presence, in the order of MediaPipe (LANDMARK_NAMES).

opencv_apps

3.0.0 and later

ROS

lyrical and later; not ROS One (OpenCV 4.6), jazzy (OpenCV 4.6), kilted (OpenCV 4.6)

OpenCV

OpenCV 4.9 or later, which reads the MediaPipe models

Group

Human and hand poses, the nodes of which publish ~/skeletons (SkeletonArrayStamped)
demo of dnn_pose_estimation

Usage

# ROS 2
ros2 launch opencv_apps dnn_pose_estimation.launch.xml image:=/camera/image_raw debug_view:=false
# ROS 1
roslaunch opencv_apps dnn_pose_estimation.launch image:=/camera/image_raw debug_view:=false

Try it with the demo video

Download the input video of the demo above, publish it as image with image_publisher, and run the node on it, in the terminals:

wget https://ros-perception.github.io/opencv_apps/media/clips/tos_bridge.mp4
# terminal 1: publish the video as image in a loop
ros2 run image_publisher image_publisher_node tos_bridge.mp4 --ros-args -p publish_rate:=10.0 -r image_raw:=image
# terminal 2: run the node
ros2 launch opencv_apps dnn_pose_estimation.launch.xml image:=image
# terminal 3: view the result
ros2 run rqt_image_view rqt_image_view /dnn_pose_estimation/image

Needs OpenCV 4.9 or later; recorded with OpenCV 4.12 on CUDA.

In ROS 1:

# terminal 1
rosrun image_publisher image_publisher tos_bridge.mp4 _publish_rate:=10 __name:=camera
# terminal 2
roslaunch opencv_apps dnn_pose_estimation.launch image:=camera/image_raw

See Credits of the demo videos for the license of the video.

Subscribed topics

Topic

Type

image

sensor_msgs/msg/Image

Published topics

Topic

Type

~/image

sensor_msgs/msg/Image

~/skeletons

opencv_apps/msg/SkeletonArrayStamped

Parameters

See also the common parameters.

Name

Type

Default (range)

Changed at run time

Description

backend

string

“auto”

yes

Backend of cv::dnn: auto, cpu, opencl, opencl_fp16, cuda or cuda_fp16. auto uses CUDA when OpenCV is built with CUDA and a CUDA device is found, and the CPU otherwise.

confidence_threshold

double

0.5 (0 - 1)

yes

Threshold of the confidence to detect the persons and to estimate their poses

detection_interval

int

10 (1 - 1000)

yes

Interval of the person detection to find the new persons while tracking

nms_threshold

double

0.3 (0 - 1)

yes

Threshold of the overlap of the persons to suppress, by non maximum suppression

person_detection_model

string

“”

yes

Path or URL of the MediaPipe person detection model (onnx), empty for person_detection_mediapipe_2023mar.onnx of opencv_zoo. The URL is downloaded to ~/.ros/opencv_apps/models/ at the first time

pose_estimation_model

string

“”

yes

Path or URL of the MediaPipe pose estimation model (onnx), empty for pose_estimation_mediapipe_2023mar.onnx of opencv_zoo. The URL is downloaded to ~/.ros/opencv_apps/models/ at the first time

queue_size

int

3

no, restart the node

Specify queue_size of input image subscribers

smoothing

bool

true

yes

Smooth the landmarks of the tracked persons by One Euro filter

smoothing_beta

double

80.0 (0 - 10000)

yes

Speed coefficient of One Euro filter, larger follows the fast motion with less lag

smoothing_derivative_cutoff

double

1.0 (0.001 - 100)

yes

Cutoff frequency (Hz) of the velocity of One Euro filter

smoothing_min_cutoff

double

0.05 (0.001 - 100)

yes

Minimum cutoff frequency (Hz) of One Euro filter, smaller is smoother when still

tracking

bool

true

yes

Track the persons by their poses in the previous image, as MediaPipe, and run the person detection every detection_interval images, otherwise every image

Source

src/nodelet/dnn_pose_estimation_nodelet.cpp, launch/dnn_pose_estimation.launch.xml