• Home
  • Search
  • STEP: Segmenting and Tracking Every Pixel
  • Cite Icon39
  • https://doi.org/10.48550/arxiv.2102.11859Copy DOI Icon

STEP: Segmenting and Tracking Every Pixel

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.

Similar Papers
  • Research Article
  • Citations3

An Active Flow Control Approach for Spatially Growing Mixing Layer

  • Mar 30, 2022
  • Journal of Fluids Engineering
  • Upender K Kaul
  • Research Article
  • Citations4

Hospitalization prediction from the emergency department using computer vision AI with short patient video clips

  • Dec 19, 2024
  • npj Digital Medicine
  • Wui Ip +17
  • Research Article
  • Citations56

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

  • Jan 08, 2021
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Jianwen Xie +2
  • Conference Article
  • Citations64

An iterative weighted regularized algorithm for improving the resolution of video sequences

  • Oct 26, 1997
  • Min-Cheol Hong +2
  • Research Article
  • Citations29

Robust Online Tracking via Contrastive Spatio-Temporal Aware Network

  • Jan 01, 2021
  • IEEE Transactions on Image Processing
  • Siyuan Yao +5
  • Research Article
  • Citations10

Factor analysis in both spatial and temporal domains of color blooming artifacts in ultrasound investigations utilizing contrast agents

  • Mar 14, 2004
  • Computerized Medical Imaging and Graphics
  • Chung-Yuo Wu +5
  • Research Article
  • Citations27

Robust quantitative single-exposure laser speckle imaging with true flow speckle contrast in the temporal and spatial domains.

  • Jul 17, 2019
  • Biomedical Optics Express
  • Chenge Wang +6
  • Research Article
  • Citations3

The effect of cue type on directive-following in children with moderate to severe autism spectrum disorder

  • Jul 03, 2021
  • Augmentative and Alternative Communication
  • Anna A Allen +3
  • PDF
  • Research Article
  • Citations2

Stereoscopic Video Quality Assessment Using Oriented Local Gravitational Force Statistics

  • Jan 01, 2020
  • IEEE Access
  • Yujian Hou +3
  • PDF
  • Research Article
  • Citations19

Video Denoising Based on a Spatiotemporal Kalman-Bilateral Mixture Model

  • Jan 01, 2013
  • The Scientific World Journal
  • Chenglin Zuo +4
  • Research Article
  • Citations13

Metaheuristic-based vector quantization approach: a new paradigm for neural network-based video compression

  • Oct 28, 2020
  • Multimedia Tools and Applications
  • Saad M Darwish +1
  • Conference Article
  • Citations5

Pixelwise adaptive prediction for lossless video compression

  • Oct 01, 2012
  • Yuhai Lu +3
  • Conference Article
  • Citations38

<title>Regularized multichannel restoration approach for globally optimal high-resolution video sequence</title>

  • Jan 10, 1997
  • Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE
  • Min-Cheol Hong +2
  • Research Article
  • Citations16

Preserving interactions among moving objects in surveillance video synopsis

  • Aug 27, 2020
  • Multimedia Tools and Applications
  • Namitha K +1
  • Book Chapter
  • Citations1

Ensemble Methods for Tracking and Segmentation (Abstract)

  • Jan 01, 2011
  • Shai Avidan
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.