PVNAS: 3D Neural Architecture Search With Point-Voxel Convolution

3D neural networks are widely used in real-world applications (e.g., AR/VR headsets, self-driving cars). They are required to be fast and accurate; however, limited hardware resources on edge devices make these requirements rather challenging. Previous work processes 3D data using either voxel-based...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2022-11, Vol.44 (11), p.8552-8568
Main Authors: Liu, Zhijian, Tang, Haotian, Zhao, Shengyu, Shao, Kevin, Han, Song
Format: Article
Language:English
Subjects:
3D point cloud
Autonomous cars
autonomous driving
Computational modeling
Computer architecture
Convolution
efficient deep learning
Hardware
Memory management
Network latency
neural architecture search
Neural networks
Race cars
Racing
Random access memory
Solid modeling
Three dimensional models
Three-dimensional displays
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:3D neural networks are widely used in real-world applications (e.g., AR/VR headsets, self-driving cars). They are required to be fast and accurate; however, limited hardware resources on edge devices make these requirements rather challenging. Previous work processes 3D data using either voxel-based or point-based neural networks, but both types of 3D models are not hardware-efficient due to the large memory footprint and random memory access. In this paper, we study 3D deep learning from the efficiency perspective. We first systematically analyze the bottlenecks of previous 3D methods. We then combine the best from point-based and voxel-based models together and propose a novel hardware-efficient 3D primitive, Point-Voxel Convolution (PVConv) . We further enhance this primitive with the sparse convolution to make it more effective in processing large (outdoor) scenes. Based on our designed 3D primitive, we introduce 3D Neural Architecture Search (3D-NAS) to explore the best 3D network architecture given a resource constraint. We evaluate our proposed method on six representative benchmark datasets, achieving state-of-the-art performance with 1.8-23.7× measured speedup. Furthermore, our method has been deployed to the autonomous racing vehicle of MIT Driverless, achieving larger detection range, higher accuracy and lower latency.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2021.3109025