VR-NeRF: High-Fidelity Virtualized Walkable Spaces

We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields. To this end, we designed and built a custom multi-camera rig to densely capture walkable spaces in high fidelity and with m...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-11
Main Authors: Xu, Linning, Agrawal, Vasu, Laney, William, Garcia, Tony, Bansal, Aayush, Kim, Changil, Samuel Rota Bulò, Porzi, Lorenzo, Kontschieder, Peter, Božič, Aljaž, Lin, Dahua, Zollhöfer, Michael, Richardt, Christian
Format: Article
Language:English
Subjects:
Accuracy
Datasets
Image quality
Image reconstruction
Radiance
Rendering
Virtual reality
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields. To this end, we designed and built a custom multi-camera rig to densely capture walkable spaces in high fidelity and with multi-view high dynamic range images in unprecedented quality and density. We extend instant neural graphics primitives with a novel perceptual color space for learning accurate HDR appearance, and an efficient mip-mapping mechanism for level-of-detail rendering with anti-aliasing, while carefully optimizing the trade-off between quality and speed. Our multi-GPU renderer enables high-fidelity volume rendering of our neural radiance field model at the full VR resolution of dual 2K\(\times\)2K at 36 Hz on our custom demo machine. We demonstrate the quality of our results on our challenging high-fidelity datasets, and compare our method and datasets to existing baselines. We release our dataset on our project website.
ISSN:2331-8422
DOI:10.48550/arxiv.2311.02542