Track, Check, Repeat: An EM Approach to Unsupervised Tracking

We propose an unsupervised method for detecting and tracking moving objects in 3D, in unlabelled RGB-D videos. The method begins with classic handcrafted techniques for segmenting objects using motion cues: we estimate optical flow and camera motion, and conservatively segment regions that appear to...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2021-04
Main Authors: Harley, Adam W, Zuo, Yiming, Wen, Jing, Mangal, Ayush, Potdar, Shubhankar, Chaudhry, Ritwick, Fragkiadaki, Katerina
Format: Article
Language:English
Subjects:
Algorithms
Data augmentation
Labels
Maximization
Modules
Moving object recognition
Optical flow (image analysis)
Optimization
Segments
Tracking
Video
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose an unsupervised method for detecting and tracking moving objects in 3D, in unlabelled RGB-D videos. The method begins with classic handcrafted techniques for segmenting objects using motion cues: we estimate optical flow and camera motion, and conservatively segment regions that appear to be moving independently of the background. Treating these initial segments as pseudo-labels, we learn an ensemble of appearance-based 2D and 3D detectors, under heavy data augmentation. We use this ensemble to detect new instances of the "moving" type, even if they are not moving, and add these as new pseudo-labels. Our method is an expectation-maximization algorithm, where in the expectation step we fire all modules and look for agreement among them, and in the maximization step we re-train the modules to improve this agreement. The constraint of ensemble agreement helps combat contamination of the generated pseudo-labels (during the E step), and data augmentation helps the modules generalize to yet-unlabelled data (during the M step). We compare against existing unsupervised object discovery and tracking methods, using challenging videos from CATER and KITTI, and show strong improvements over the state-of-the-art.
ISSN:2331-8422