TL;DR: We introduce the task of unsupervised video panoptic segmentation, along with a unified evaluation protocol and four competitive baselines. We propose VideoCUPS, the first method that directly tackles it, generating temporally consistent panoptic pseudo-labels from monocular scene-centric videos using depth, motion, and visual cues, then training an accurate VPS model with a novel Video DropLoss.

Abstract

Video panoptic segmentation (VPS) aims to jointly detect, segment, and track all objects while partitioning the video into semantically consistent regions. We introduce the task setting of unsupervised VPS, omitting any human supervision. Existing unsupervised scene understanding works mainly focused on image segmentation tasks; the video domain remains underexplored. We propose VideoCUPS, the first unsupervised VPS approach. VideoCUPS generates temporally consistent panoptic video pseudo-labels from scene-centric videos by exploiting unsupervised depth, motion, and visual cues. Training on these pseudo-labels using a novel Video DropLoss yields an accurate, unsupervised VPS model. To benchmark progress, we introduce a comprehensive evaluation protocol and four competitive baselines, extending state-of-the-art unsupervised panoptic image and instance video segmentation models to VPS. VideoCUPS outperforms all baselines and demonstrates strong label-efficient learning. With VideoCUPS, our evaluation protocol, and baselines, we provide a strong foundation for future research on unsupervised VPS.

Method

VideoCUPS generates temporally consistent video panoptic pseudo-labels from monocular scene-centric videos. Instance pseudo-labels come from motion-based region growing on self-supervised optical flow (SMURF) and depth (DynamoDepth). Semantic pseudo-labels come from distilled DINO features (k-means) refined by depth-guided inference. Temporally, instances are propagated and tracked via warped-IoU + Hungarian matching, semantics are smoothed by a 3-frame majority vote, and semantic↔instance assignments are aligned per clip. A pixel-ratio threshold splits things from stuff. The resulting pseudo-labels supervise a Panoptic Cascade MaskTrack R-CNN with DINO ResNet-50, trained with our Video DropLoss and self-enhanced video copy-paste augmentation.

Quantitative results

We compare VideoCUPS against four competitive baselines that pair state-of-the-art unsupervised semantic, video-instance, or panoptic image segmentation methods with unsupervised tracking, evaluated with STQ, AQ, and SQ (all in %, higher is better) on Cityscapes-VPS, KITTI-STEP, Waymo, and the out-of-domain MOTS benchmark. Despite training only on monocular videos, VideoCUPS outperforms every baseline on every dataset and metric, including CUPS + SORT, which uses stereo video at training time.

^† CUPS trained with monocular videos. Supervised reference in gray.

Label-efficient learning

We fine-tune a VideoCUPS-pretrained model on varying fractions of Cityscapes-VPS labels and compare against the same architecture initialized with DINO. With only 10% of the labels, VideoCUPS already reaches the STQ of a randomly initialized supervised model trained on the full Cityscapes-VPS train set, and improves over the DINO-initialized baseline by 4.6% STQ. At 100% labels, VideoCUPS still improves over DINO by +2.6% STQ, +2.3% AQ, and +3.5% SQ.

Qualitative comparisons

Below, we showcase video panoptic predictions on Cityscapes demo video sequences. Where the baselines struggle with temporal consistency, VideoCUPS produces stable, temporally coherent panoptic masks.

BibTeX

@inproceedings{Reich:2026:VideoCUPS,
  title     = {Scene-Centric Unsupervised Video Panoptic Segmentation},
  author    = {Reich, Christoph and Hahn, Oliver and Araslanov, Nikita and
               Leal-Taix{\'e}, Laura and Rupprecht, Christian and
               Cremers, Daniel and Roth, Stefan},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}