Motion2fusion: real-time volumetric performance capture

• Published in: ACM transactions on graphics 2017-12, Vol.36 (6), p.1-16
• Main Authors: Dou, Mingsong, Davidson, Philip, Fanello, Sean Ryan, Khamis, Sameh, Kowdle, Adarsh, Rhemann, Christoph, Tankovich, Vladimir, Izadi, Shahram
• Format: Article
• Language: English
• ISSN: 0730-0301; 1557-7368
• DOI: 10.1145/3130800.3130801

We present Motion2Fusion, a state-of-the-art 360 performance capture system that enables *real-time* reconstruction of arbitrary non-rigid scenes. We provide three major contributions over prior work: 1) a new non-rigid fusion pipeline allowing for far more faithful reconstruction of high frequency geometric details, avoiding the over-smoothing and visual artifacts observed previously. 2) a high speed pipeline coupled with a machine learning technique for 3D correspondence field estimation reducing tracking errors and artifacts that are attributed to fast motions. 3) a backward and forward non-rigid alignment strategy that more robustly deals with topology changes but is still free from scene priors. Our novel performance capture system demonstrates real-time results nearing 3x speed-up from previous state-of-the-art work on the exact same GPU hardware. Extensive quantitative and qualitative comparisons show more precise geometric and texturing results with less artifacts due to fast motions or topology changes than prior art.