Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion, Shape and Reflectance

• Published in: International journal of computer vision 2002-09, Vol.49 (2-3), p.175
• Main Authors: Carceroni, Rodrigo L, Kutulakos, Kiriakos N
• Format: Article
• Language: English
• Subjects: Parameter estimation; Spacetime; Studies
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1023/A:1020145606604

In this paper we study the problem of recovering the 3D shape, reflectance, and non-rigid motion properties of a dynamic 3D scene. Because these properties are completely unknown and because the scene's shape and motion may be non-smooth, our approach uses multiple views to build a piecewise-continuous geometric and radiometric representation of the scene's trace in space-time. A basic primitive of this representation is the dynamic surfel, which (1) encodes the instantaneous local shape, reflectance, and motion of a small and bounded region in the scene, and (2) enables accurate prediction of the region's dynamic appearance under known illumination conditions. We show that complete surfel-based reconstructions can be created by repeatedly applying an algorithm called Surfel Sampling that combines sampling and parameter estimation to fit a single surfel to a small, bounded region of space-time. Experimental results with the Phong reflectancemodel and complex real scenes (clothing, shiny objects, skin) illustrate our method's ability to explain pixels and pixel variations in terms of their underlying causes--shape, reflectance, motion, illumination, and visibility.[PUBLICATION ABSTRACT]