Continuous, Real-Time Object Detection on Mobile Devices without Offloading

• Main Authors: Liu, Miaomiao, Ding, Xianzhong, Du, Wan
• Format: Conference Proceeding
• Language: English
• Subjects: Adaptation models; Degradation; Mobile devices; Mobile handsets; Object detection; Object tracking; Pipelines; Real-time systems; Streaming media; Video processing
• ISSN: 2575-8411
• DOI: 10.1109/ICDCS47774.2020.00085

This paper presents AdaVP, a continuous and real-time video processing system for mobile devices without offloading. AdaVP uses Deep Neural Network (DNN) based tools like YOLOv3 for object detection. Since DNN computation is time-consuming, multiple frames may be captured by the camera during the processing of one frame. To support real-time video processing, we develop a mobile parallel detection and tracking (MPDT) pipeline that executes object detection and tracking in parallel. When the object detector is processing a new frame, a light-weight object tracker is used to track the objects in the accumulated frames. As the tracking accuracy decreases gradually, due to the accumulation of tracking error and the appearance of new objects, new object detection results are used to calibrate the tracking accuracy periodically. In addition, a large DNN model produces high accuracy, but requires long processing latency, resulting in a great degradation for tracking accuracy. Based on our experiments, we find that the tracking accuracy degradation is also related to the variation of video content, e.g., for a dynamically changing video, the tracking accuracy degrades fast. A model adaptation algorithm is thus developed to adapt the DNN models according to the change rate of video content. We implement AdaVP on Jetson TX2 and conduct a variety of experiments on a large video dataset. The experiment results reveal that AdaVP improves the accuracy of the state-of-the-art solution by up to 43.9%.