Generating Assembly Instructions Using Reinforcement Learning in Combination with Large Language Models

• Main Authors: Widulle, Niklas, Meyer, Frederic, Niggemann, Oliver
• Format: Conference Proceeding
• Language: English
• Subjects: Accuracy; Assembly; Informatics; Large language models; Manuals; Reinforcement learning; Solid modeling
• ISSN: 2378-363X
• DOI: 10.1109/INDIN58382.2024.10774545

The efficiency of manual assembly can be significantly improved by utilizing assistance systems that display as-sembly instructions. However, generating and maintaining these instructions require substantial effort, especially for low-volume or highly customized products. If neglected, this can lead to outdated instructions, frustrated operators, and the abandonment of the assistance platform. Recent advancements in Large Language Models (LLMs) have made it possible to generate high-quality assembly instructions given the right input. However, relying solely on LLMs to plan the assembly process risks producing unfeasible assembly sequences due to potential hallucinations by the models. To address this, Reinforcement Learning (RL) can be used in simulations to plan the assembly process, imposing restrictions on impractical movements. We propose a framework that integrates RL and LLMs to generate practical and accurate assembly instructions. In our framework, RL is used within a simulation to generate feasible assembly sequences. These sequences are then transformed into detailed assembly instructions by an LLM. We evaluate our framework on real-world product assemblies, generating comprehensive assembly instructions from corresponding CAD files. Our results demonstrate the potential of combining RL and LLMs to automate the generation of assembly instructions, thereby overcoming the limitations of current assistance systems and enhancing the efficiency of manual assembly processes.