Description
Recent parametric human body models, such as SMPL [1] or GHUM [2] provide rich and natural yet very sparse prior for realistic body modelling. All parameters have semantic meaning and allow to render diverse set of human bodies in the form of triangular meshes with known topology. Usually, the set of parameters consists of disentangled shape and pose parts, where the former accounts for the appearance, e.g., tall, slim, etc., and the latter – for the posing in terms of rotations in the joints, e.g., left hand is up, head is turned left, etc.
Despite great advances and capabilities of the proposed models, none of them has an intrinsic rectification module that would correct unrealistic bodies. For example, this is easy to imagine a body with the arms directed towards the body but the arms inside the body is definitely considered unrealistic, yet there is no control of this effect in the model.
We propose to explore effective ways of correcting such bodies, e.g. via using trainable neural flows. If such “body correction” module exists, it would allow to fix any set of parameters to produce bodies that are always realistic. Moreover, if such module exists and is differentiable, it could be used as an additional layer in the neural network to provide an additional realistic body prior during training or inference time.
In the core, the proposed method explores the capabilities and robustness of so-called Neural ODE flow models [3]. The idea is to build a neural flow that would move from the fixed body of simple (but realistic) pose towards a given one that might be infeasible. As any ODE always obeys the Picard-Lindelöf theorem [4] (does not matter whether it is modelled with a closed-form formula or with a neural network), the resulting motions of each point in the body, i.e. per-vertex flows, cannot intersect each other. That said, the resulting body would be realistic and at the same time would not lie too far from the original unrealistic target body.
Besides developing our main solution algorithm, it is also important to provide an assessment of the existing solutions for the self-penetration problem with their pros and cons.
Prerequisites
The candidate should have programming experience in Python. It is equally important to have strong intuition and deep understanding of linear algebra and basic differential equations courses. Calculus and functional calculus are also appreciated.
Previous experience with neural networks, especially PyTorch framework, is a plus.
The main requirements are curiosity to learn new and willingness to overcome difficulties.
Contact
If you are interested in this project, please send an email to Andrey Davydov. More details and explanations on this project can be provided in-person or via Zoom.
References
[1] SMPL-X model: Expressive Body Capture: 3D Hands, Face, and Body from a Single Image, G. Pavlakos et al, CVPR, 2019.
[2] GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models, H. Xu et al., CVPR, 2020.
[3] Neural ODEs: breakdown of another deep learning breakthrough, A. Honchar, Towards Data Science blogpost.
[4] The Picard–Lindelöf theorem.