SOMSI: Spherical Novel View Synthesis
with Soft Occlusion Multi-Sphere Images
Tewodros Habtegebrial1,3
Christiano Gava1,3
Marcel Rogge1,3
Didier Stricker1,3
Varun Jampani2
1 TU Kaiserslautern 2 Google Research 3 DFKI Kaiserslautern

IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) 2022


Spherical novel view synthesis (SNVS) is the task of estimating 360 views at dynamic novel views given a set of 360 input views. Prior arts learn multi-sphere image (MSI) representations that enables fast rendering times but are only limited to modelling low-dimensional color values. Modelling high-dimensional appearance features in MSI can result in better view synthesis, but it is not feasible to represent high-dimensional features in a large number (>64) of MSI spheres. We propose a novel MSI representation called Soft Occlusion MSI (SOMSI) that enables modelling high-dimensional appearance features in MSI while retaining the fast rendering times of a standard MSI. Our key insight is to model appearance features in a smaller set (e.g. 3) of occlusion levels instead of larger number of MSI levels. Experiments on both synthetic and real-world scenes demonstrate that using SOMSI can provide a good balance between accuracy and runtime. SOMSI can produce considerably better results compared to MSI based MatryODSHka, while having similar fast rendering time. SOMSI view synthesis quality is on-par with state-of-the-art NeRF like model while being 2 orders of magnitude faster.


Supplementary Video

 [Data: click here to download the dataset]


We thank Koretaka Ogata and Soichiro Yokota, for preparing the Residential Area scenes.
Tewodros is supported by the Google PhD Fellowship program.
This page is borrowed from Richard Zhang.