Animal Pose Tracking: 3D Multimodal Dataset and Token-based Pose Optimization

Accurate tracking of the 3D pose of animals from video recordings is critical for many behavioral studies, yet there is a dearth of publicly available datasets that the computer vision community could use for model development. We here introduce the Rodent3D dataset that records animals exploring th...

Full description

Saved in:
Bibliographic Details
Published in:International journal of computer vision 2023-02, Vol.131 (2), p.514-530
Main Authors: Patel, Mahir, Gu, Yiwen, Carstensen, Lucas C., Hasselmo, Michael E., Betke, Margrit
Format: Article
Language:English
Subjects:
Analysis
Animal behavior
Animals
Artificial Intelligence
Cameras
Computational linguistics
Computer Imaging
Computer Science
Computer vision
Datasets
Image Processing and Computer Vision
Language processing
Machine vision
Natural language interfaces
Occlusion
Optimization
Pattern Recognition
Pattern Recognition and Graphics
Pose estimation
Special Issue on Computer Vision Approach for Animal Tracking and Modeling
Tracking
Vision
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Accurate tracking of the 3D pose of animals from video recordings is critical for many behavioral studies, yet there is a dearth of publicly available datasets that the computer vision community could use for model development. We here introduce the Rodent3D dataset that records animals exploring their environment and/or interacting with each other with multiple cameras and modalities (RGB, depth, thermal infrared). Rodent3D consists of 200 min of multimodal video recordings from up to three thermal and three RGB-D synchronized cameras (approximately 4 million frames). For the task of optimizing estimates of pose sequences provided by existing pose estimation methods, we provide a baseline model called OptiPose . While deep-learned attention mechanisms have been used for pose estimation in the past, with OptiPose , we propose a different way by representing 3D poses as tokens for which deep-learned context models pay attention to both spatial and temporal keypoint patterns. Our experiments show how OptiPose is highly robust to noise and occlusion and can be used to optimize pose sequences provided by state-of-the-art models for animal pose estimation.
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-022-01714-5