Depth Sweep Regression Forests for Estimating 3D Human Pose from Images
Ilya Kostrikov and Juergen Gall
Abstract
In this work we address the problem of estimating the 3D human pose from a single RGB image, which is a challenging problem since different 3D poses may have similar 2D projections. Following the success of regression forests for 3D pose estimation from depth data or 2D pose estimation from RGB images, we extend regression forests to infer missing depth data of image features and 3D pose simultaneously. Since we do not observe depth for inference or training directly, we hypothesize the depth of the features by sweeping with a plane through the 3D volume of potential joint locations. The regression forests are then combined with a pictorial structure framework, which is extended to 3D. The approach is evaluated on two challenging benchmarks where state-of- the-art performance is achieved.
Images
3D joint location probabilities from a 2D image are estimated using depth sweep pose regression forests. The two left images depict the front and side view of the inferred 3D pose (green) and compare it with the ground-truth (red). A 3D pictorial structure framework is then used to enforce kinematic constraints and improve the 3D pose estimates (right).
Illustration of a depth sweep regression forest for 3D pose estimation from a 2D image. Left: Patches sampled from different depths project onto the image with different scale. Middle: The projected patches traverse the tree evaluating splitting functions in the intermediate nodes (black and red) until they reach a leaf node (blue). A leaf node contains 3D offsets that point to locations of a joint with associated weights. Right: Based on the offsets, the patches sampled in the 3D volume cast 3D votes for several joint locations.
Publications
Kostrikov I. and Gall J., Depth Sweep Regression Forests for Estimating 3D Human Pose from Images (PDF), British Machine Vision Conference (BMVC'14), 2014.
Dantone M., Gall J., Leistner C., and van Gool L., Body Parts Dependent Joint Regressors for Human Pose Estimation in Still Images (PDF), IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 36, No. 11, 2131-2143, 2014. ©IEEE
Dantone M., Gall J., Leistner C., and van Gool L., Human Pose Estimation using Body Parts Dependent Joint Regressors (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13), 3041-3048, 2013. ©IEEE