MDS-NeRF: Neural Radiance Fields with Marigold Depth Supervision
Abstract
Neural Radiance Fields (NeRF) achieved impressive results in novel view synthesis given dense multi-view images. However, NeRF face challenges of fitting incorrect geometries when given an insufficient number of input views. In this project, we employ Marigold, a state-of-the-art affine-invariant depth prediction method, to supervise NeRF training with few views. Marigold depth could provide geometry priors for NeRF training, which helps to improve the performance of scene reconstruction.
Method
Marigold only predict affine-invariant depth which exists the scale and offset difference from metric depth, so it could not be directly used for depth supervision. In this project, we jointly optimize scales, offsets of views and NeRF representation based on mutli-view geometry. Our method could recover metric depth and realize high-fidelty scene reconstruction simultaneously. We use nerfacto as our NeRF backbone due to its fast speed.
Results
Nerfacto (left) vs Ours (right) on LLFF dataset. Scene trained on 3 views. Try selecting scenes!
Nerfacto (left) vs Ours (right) on Record3D dataset. Scene trained on 9 views. Try selecting scenes!
Acknowledgements
I would like to thank our supervisors Mihai Dusmanu and Zuria Bauer for their valuable guidance in this 3DV project.
The website template was borrowed from Michaël Gharbi and Ref-NeRF.