Abstract
Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. Here we extend the Laplacian formalism, which was first introduced in the Graphics community to regularize 3D meshes, to achieve this goal given correspondences with a reference image in which the shape is known.
Our approach allows us to reduce the dimensionality of the surface reconstruction problem without sacrificing accuracy, thus allowing for real-time implementations.
The algorithm runs in realtime on tablets. It can detect which deformable object appears in the camera stream, then track and reconstruct the corresponding 3D shape as the object deforms over time.
We also use our algorithm to build an augmented reality coloring book App, which runs in realtime on tablets. The App tracks a possibly non-planar drawing page, precisely takes its colors to texture a virtual animated character. This work was done in colloboration with Disney Research Zurich.
Results
Real-Time application on Tablets
Real-Time Demo on PC
Modeling the Impact of a Baseball
Reprojection of 3D reconstructed mesh on image plane
3D reconstruction of the baseball during the collision with a bat
On-contact vertices during the impact
Code and Supplementary Material
Code and instructions for running it are available here.
Supplementary material can be downloaded as a zip-file supplementary.zip or viewed on a webpage README.html.