PASS: Patch Automatic Skip Scheme for Efficient On-Device Video Perception

• Published in: IEEE transactions on pattern analysis and machine intelligence 2024-05, Vol.46 (5), p.3938-3954
• Main Authors: Zhou, Qihua, Guo, Song, Pan, Jun, Liang, Jiacheng, Guo, Jingcai, Xu, Zhenda, Zhou, Jingren
• Format: Article
• Language: English
• Subjects: Computational modeling; Decoupling; Feature extraction; Logic gates; Model accuracy; Multiple target tracking; On-device processing systems; Optimization; Perception; Pose estimation; Streaming media; Streams; Task analysis; Termination of employment; video perception; visual analytics
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2024.3350380

Real-time video perception tasks are often challenging on resource-constrained edge devices due to the issues of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. These limitations motivate us to design a general and task-independent methodology, called Patch Automatic Skip Scheme (PASS), which supports diverse video perception settings by decoupling acceleration and tasks. The gist is to capture inter-frame correlations and skip redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. Specifically, we are the first to construct a self-supervisory procedure for gate optimization, which learns to extract contrastive representations from frame sequences. The pre-trained gates can serve as plug-and-play modules to implement patch-skippable neural backbones, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming state-of-the-art MobileHumanPose in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 \times 9.43× and 12.19 \times 12.19× speedups, respectively, on NVIDIA Jetson Nano devices.