Method
SceneDINO is trained using multi-view self-supervision and learns a feature field composed of high-dimensional DINO features and 3D geometry. Distilling SceneDINO's feature field results in unsupervised semantic prediction. During inference, SceneDINO performs SSC, given a single RGB input image.
(a) Inference: Given a single input image \(\mathbf{I}_{0}\) during inference, a 2D encoder-decoder \(\xi\) produces the embedding \(\mathbf{E}\) from which the local embedding \(\mathbf{e}_{\mathbf{u}}\) is interpolated. The MLP encoder \(\phi\) takes in \(\mathbf{e}_{\mathbf{u}}\) and 3D position \(\mathbf{x}_{i}\), and predicts both the density \(\sigma_{\mathbf{x}_{i}}\) and the 3D feature \(f_{\mathbf{x}_{i}}\). Using a lightweight unsupervised segmentation head \(h\), we can obtain semantic predictions \(p_{\mathbf{x}_{i}}\) using \(f_{\mathbf{x}_{i}}\).
(b) Rendering: Our feature field allows for volume rendering by shooting a ray through it, yielding depth \(\hat{d}\) and \(\hat{f}\) in 2D. Color \(c_{i}\) is sampled from an another view (e.g., \(\mathbf{I}_{1}\)) using \(\mathbf{u}_{s}\) and rendered to obtain the reconstructed color \(\hat{c}\).
(c) Multi-view training: We render 2D views (features & images) from our feature field and reconstruct the training views.
3D Visualization
We provide visualizations of SceneDINO's feature field and semantic scene completion for three scenes from KITTI-360, alongside the corresponding 3D ground truth. We visualize the top 3 PCA components of the feature field. Click on any input image below to view SceneDINO's predictions. We only visualize surface voxels within the field of view for the sake of clarity.
2D Visualization
We compare the original dense DINO features with our SceneDINO features rendered back into the image plane. The visualized PCA components are mapped to RGB channels. Interactively adjust which components to display.