Object recognition with human-in-the-loop assistance using error information

• Main Authors: Hu, Jie, Kimura, Nobutaka, Matsui, Takaharu
• Format: Conference Proceeding
• Language: English
• Subjects: Automation; Classification algorithms; Graphical user interfaces; Human in the loop; Image edge detection; Industries; Materials handling; Object recognition; Production; Service robots
• ISSN: 2161-8089
• DOI: 10.1109/CASE59546.2024.10711817

Object recognition is fundamental to many industrial automation operations such as robotic material handling. Currently, many industry sectors have seen growing needs to automate the handling of high mix objects, which is more challenging comparing to traditional low mix objects in terms achieving high object recognition success rate. To this end, we propose an object recognition workflow with human-in-the-loop features that is suitable for use in production. The object recognition is based on object boundary detection to obtain the segments corresponding to different objects. When problematic recognition occurs, human operators can provide assistance using the designed Graphic User Interface (GUI) remotely. To minimize the downtime, we prompt the human operators to indicate the recognition error information. The designed algorithms will use the error information to automatically adjust corresponding parameters to solve the recognition errors. We first evaluated the proposed workflow on two groups of real world datasets in terms of object recognition. Then we tested the designed GUI and proposed algorithms to correct the recognition errors from previous evaluations. We demonstrated that the proposed human-in-the-loop assistance is effective in solving commonly seen recognition errors and could be implemented in production to address the challenges that the object recognition system might encounter when handling high mix objects.