Computer vision-based excavator bucket fill estimation using depth map and faster R-CNN

• Published in: Automation in construction 2024-10, Vol.166, p.105592, Article 105592
• Main Authors: Helian, Bobo, Huang, Xiaoqian, Yang, Meng, Bian, Yongming, Geimer, Marcus
• Format: Article
• Language: English
• Subjects: Bucket volume estimation; Computer vision; Custom loss function; Depth map; Excavator automation; Faster R-CNN
• ISSN: 0926-5805
• DOI: 10.1016/j.autcon.2024.105592

Excavators are crucial in the construction industry, and developing autonomous excavator systems is vital for enhancing productivity and reducing the reliance on manual labor. Accurate estimation of the volume of the excavator bucket fill is key for monitoring and evaluating system automation performance. This paper presents the use of 2D depth maps as input to a Faster Region Convolutional Neural Network (Faster R-CNN) deep learning model for bucket volume estimation. This structure enables high estimation accuracy while maintaining fast processing speed. An excavator operation monitoring test bench was established, and the datasets used in the study were self-generated for training. A loss function is proposed, combining Cross Entropy with Root Mean Squared Error to improve generalization and precision. Comparative results indicate that the proposed approach achieves 96.91% accuracy in fill factor estimation and predicts in real-time at about 10 fps, highlighting its potential for practical use in automated excavator operations. •Tailored Faster R-CNN for accurate real-time excavator bucket fill estimation.•Use of 2D depth maps enhances processing speed without compromising accuracy.•Development of diverse image datasets matching excavator loading conditions.•Custom loss function integrates Cross Entropy with MSE, enhancing accuracy in bucket fill estimation.•Achieving 96.91% accuracy in fill estimation test, proving its benefits in automated monitoring.