Self-Supervised Multi-View Person Association and its Applications

Reliable markerless motion tracking of people participating in a complex group activity from multiple moving cameras is challenging due to frequent occlusions, strong viewpoint and appearance variations, and asynchronous video streams. To solve this problem, reliable association of the same person a...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2021-08, Vol.43 (8), p.2794-2808
Main Authors: Vo, Minh, Yumer, Ersin, Sunkavalli, Kalyan, Hadap, Sunil, Sheikh, Yaser, Narasimhan, Srinivasa G.
Format: Article
Language:English
Subjects:
Algorithms
Cameras
Clustering
Descriptor adaptation
motion tracking
multi-angle video
Occlusion
Pattern recognition
people association
Reliability
self-supervised
Skeleton
Streaming media
Target tracking
Three-dimensional displays
Tracking
Video data
Websites
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reliable markerless motion tracking of people participating in a complex group activity from multiple moving cameras is challenging due to frequent occlusions, strong viewpoint and appearance variations, and asynchronous video streams. To solve this problem, reliable association of the same person across distant viewpoints and temporal instances is essential. We present a self-supervised framework to adapt a generic person appearance descriptor to the unlabeled videos by exploiting motion tracking, mutual exclusion constraints, and multi-view geometry. The adapted discriminative descriptor is used in a tracking-by-clustering formulation. We validate the effectiveness of our descriptor learning on WILDTRACK T. Chavdarova et al. , "WILDTRACK: A multi-camera HD dataset for dense unscripted pedestrian detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5030-5039. and three new complex social scenes captured by multiple cameras with up to 60 people "in the wild". We report significant improvement in association accuracy (up to 18 percent) and stable and coherent 3D human skeleton tracking (5 to 10 times) over the baseline. Using the reconstructed 3D skeletons, we cut the input videos into a multi-angle video where the image of a specified person is shown from the best visible front-facing camera. Our algorithm detects inter-human occlusion to determine the camera switching moment while still maintaining the flow of the action well. Website : http://www.cs.cmu.edu/~ILIM/projects/IM/Association4Tracking .
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2020.2974726