dnn_hand_pose_estimation

Estimate the hand poses with MediaPipe palm detection and hand pose estimation.

Hand pose estimation with MediaPipe Hands of opencv_zoo (Apache 2.0): the palms are detected by the palm detection model, then the 21 landmarks of each hand are estimated by the hand pose estimation model, on the image rotated so that the fingers point up. As MediaPipe, the hands are tracked by the palm landmarks of their poses in the previous image, and the palm detection runs every detection_interval images to find the new hands, or when a tracked hand is lost. The landmarks of the tracked hands are smoothed by One Euro filter, as MediaPipe. The hand pose estimation model needs OpenCV 4.9 or later (OpenCV 4.6 cannot read it). ~skeletons has a Skeleton for each hand: label “left” or “right” (the handedness of the model, which assumes the mirrored image of a front camera as MediaPipe), id the id while tracked, score the confidence of the hand, rect the bounding box of the landmarks, and keypoints the 21 landmarks in the image coordinates, in the order of MediaPipe (LANDMARK_NAMES), whose visibility and presence are the confidence of the hand.

opencv_apps	3.0.0 and later
ROS	lyrical and later; not ROS One (OpenCV 4.6), jazzy (OpenCV 4.6), kilted (OpenCV 4.6)
OpenCV	OpenCV 4.9 or later, which reads the MediaPipe hand pose model
Group	Human and hand poses, the nodes of which publish `~/skeletons` (SkeletonArrayStamped)

Usage

# ROS 2
ros2 launch opencv_apps dnn_hand_pose_estimation.launch.xml image:=/camera/image_raw debug_view:=false
# ROS 1
roslaunch opencv_apps dnn_hand_pose_estimation.launch image:=/camera/image_raw debug_view:=false

Try it with the demo video

Download the input video of the demo above, publish it as image with image_publisher, and run the node on it, in the terminals:

wget https://ros-perception.github.io/opencv_apps/media/clips/sign.mp4
# terminal 1: publish the video as image in a loop
ros2 run image_publisher image_publisher_node sign.mp4 --ros-args -p publish_rate:=10.0 -r image_raw:=image
# terminal 2: run the node
ros2 launch opencv_apps dnn_hand_pose_estimation.launch.xml image:=image
# terminal 3: view the result
ros2 run rqt_image_view rqt_image_view /dnn_hand_pose_estimation/image

Needs OpenCV 4.9 or later; recorded with OpenCV 4.12 on CUDA.

In ROS 1:

# terminal 1
rosrun image_publisher image_publisher sign.mp4 _publish_rate:=10 __name:=camera
# terminal 2
roslaunch opencv_apps dnn_hand_pose_estimation.launch image:=camera/image_raw

See Credits of the demo videos for the license of the video.

Subscribed topics

Topic	Type
`image`	`sensor_msgs/msg/Image`

Published topics

Topic	Type
`~/image`	`sensor_msgs/msg/Image`
`~/skeletons`	opencv_apps/msg/SkeletonArrayStamped

Parameters

See also the common parameters.

Name	Type	Default (range)	Changed at run time	Description
`backend`	string	“auto”	yes	Backend of cv::dnn: auto, cpu, opencl, opencl_fp16, cuda or cuda_fp16. auto uses CUDA when OpenCV is built with CUDA and a CUDA device is found, and the CPU otherwise.
`confidence_threshold`	double	0.5 (0 - 1)	yes	Threshold of the confidence to detect the palms and to estimate the hands
`detection_interval`	int	10 (1 - 1000)	yes	Interval of the palm detection to find the new hands while tracking
`hand_pose_estimation_model`	string	“”	yes	Path or URL of the MediaPipe hand pose estimation model (onnx), empty for handpose_estimation_mediapipe_2023feb.onnx of opencv_zoo. The URL is downloaded to ~/.ros/opencv_apps/models/ at the first time
`nms_threshold`	double	0.3 (0 - 1)	yes	Threshold of the overlap of the hands to suppress, by non maximum suppression
`palm_detection_model`	string	“”	yes	Path or URL of the MediaPipe palm detection model (onnx), empty for palm_detection_mediapipe_2023feb.onnx of opencv_zoo. The URL is downloaded to ~/.ros/opencv_apps/models/ at the first time
`queue_size`	int	3	no, restart the node	Specify queue_size of input image subscribers
`smoothing`	bool	true	yes	Smooth the landmarks of the tracked hands by One Euro filter
`smoothing_beta`	double	80.0 (0 - 10000)	yes	Speed coefficient of One Euro filter, larger follows the fast motion with less lag
`smoothing_derivative_cutoff`	double	1.0 (0.001 - 100)	yes	Cutoff frequency (Hz) of the velocity of One Euro filter
`smoothing_min_cutoff`	double	0.05 (0.001 - 100)	yes	Minimum cutoff frequency (Hz) of One Euro filter, smaller is smoother when still
`tracking`	bool	true	yes	Track the hands by their poses in the previous image, as MediaPipe, and run the palm detection every detection_interval images, otherwise every image

Source

src/nodelet/dnn_hand_pose_estimation_nodelet.cpp, launch/dnn_hand_pose_estimation.launch.xml