Unsupervised video object segmentation with distractor-aware online adaptation

• Published in: Journal of visual communication and image representation 2021-01, Vol.74, p.102953, Article 102953
• Main Authors: Wang, Ye, Choi, Jongmoo, Chen, Yueru, Li, Siyang, Huang, Qin, Zhang, Kaitai, Lee, Ming-Sui, Kuo, C.-C. Jay
• Format: Article
• Language: English
• Subjects: Hard negative mining; Motion saliency; Online adaptation; Pseudo ground truth; Unsupervised video object segmentation
• ISSN: 1047-3203; 1095-9076
• DOI: 10.1016/j.jvcir.2020.102953

Unsupervised video object segmentation is a crucial application in video analysis when there is no prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a video clip. In this paper, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatiotemporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and they are grouped by motion information as positives or hard negatives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced when there are no hard negatives in the sequence. The experimental results demonstrate these two kinds of negatives are complementary. Finally, we conduct DOA using positive, negative, and hard negative masks to update the foreground and background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, the DAVIS 2016 and the Freiburg-Berkeley motion segmentation (FBMS)-59.