Learning Monocular Reconstruction from Multi-view Images

Accurate 3D human pose estimation from single images is possible with sophisticated deep-net architectures that have been trained on very large datasets. However, this still leaves open the problem of capturing motions for which no such database exists. Manual annotation is tedious, slow, and error-prone.

In this project, we propose to replace most of the annotations by the use of multiple views, at training time only. We propose the following two alternative methods.

Learning Monocular 3D Human Pose Estimation from Multi-view Images

In this pose-centered approach, we train the system to predict the same pose articulated pose in all available views. Such a consistency constraint is necessary but not sufficient to predict accurate poses. We therefore complement it with a supervised loss aiming to predict the correct pose in a small set of labeled images, and with a regularization term that penalizes drift from initial predictions. Furthermore, we propose a method to estimate camera pose jointly with human pose, which lets us utilize multi-view footage where calibration is difficult, e.g., for pan-tilt or moving handheld cameras. We demonstrate the effectiveness of our approach on established benchmarks, as well as on a new Ski dataset with rotating cameras and expert ski motion, for which annotations are truly hard to obtain. Learning an unsupervised representation

The new Ski-pose PTZ-camera dataset is available here: Ski-PosePTZ-Dataset.
Pre-print: https://arxiv.org/abs/1803.04775

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

Learning an unsupervised representation

In this approach, we propose to overcome remaining problems by learning a geometry-aware body representation from multi-view images without any 3D annotations. To this end, we use an encoder-decoder that predicts an image from one viewpoint given an image from another viewpoint. Because this representation encodes 3D geometry, using it in a semi-supervised setting makes it easier to learn a mapping from it to 3D human pose. As evidenced by our experiments, our approach significantly outperforms fully-supervised methods given the same amount of labeled data, and improves over the first semi-supervised method while using as little as 1% of the labeled data.

Pre-print: https://arxiv.org/abs/1804.01110
Pytorch network definition and training code: github.com/hrhodin/


LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals

A. Gosztolai; S. Gunel; V. Lobato-Rios; M. Pietro Abrate; D. Morales et al. 

Nature Methods. 2021-08-01. Vol. 18, num. 8, p. 975-+. DOI : 10.1038/s41592-021-01226-z.

PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers

F. Yu; M. Salzmann; P. Fua; H. Rhodin 

2021. Conference on Computer Vision and Pattern Recognition (CVPR), Online, June 19-25, 2021.

ActiveMoCap: Optimized Viewpoint Selection for Active Human Motion Capture

S. Kiciroglu; H. Rhodin; S. N. Sinha; M. Salzmann; P. Fua 

2020-06-01. Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, Washington, USA, June 14-19, 2020. p. 100-109. DOI : 10.1109/CVPR42600.2020.00018.

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

S. Günel; H. Rhodin; D. Morales; J. H. Campagnolo; P. Ramdya et al. 

eLife. 2019-10-04. Vol. 8, p. e48571. DOI : 10.7554/eLife.48571.

Neural Scene Decomposition for Multi-Person Motion Capture

H. Rhodin; V. Constantin; I. Katircioglu; M. Salzmann; P. Fua 

2019-06-20. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, Jun 16-20, 2019. p. 7695-7705. DOI : 10.1109/CVPR.2019.00789.

What Face and Body Shapes Can Tell Us About Height

S. Gunel; H. Rhodin; P. Fua 

2019-01-01. IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, SOUTH KOREA, Oct 27-Nov 02, 2019. p. 1819-1827. DOI : 10.1109/ICCVW.2019.00226.

Learning Monocular 3D Human Pose Estimation from Multi-view Images

H. Rhodin; J. Sporri; I. Katircioglu; V. Constantin; F. Meyer et al. 

2018-01-01. Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, Jun 18-23, 2018. p. 8437-8446. DOI : 10.1109/CVPR.2018.00880.

Unsupervised Geometry-Aware Representation Learning for 3D Human Pose Estimation

H. Rhodin; M. Salzmann; P. Fua 

2018. European Conference on Computer Vision (ECCV).