Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm

• Published in: Computers & operations research 2022-08, Vol.144, p.105823, Article 105823
• Main Authors: Yan, Qi, Wang, Hongfeng, Wu, Fang
• Format: Article
• Language: English
• Subjects: Algorithms; Decision making; Digital twin; Digital twins; Double-resource flexible job shop; Heuristic methods; Job shop scheduling; Job shops; Machine learning; Manufacturing; Operations research; Optimization; Perturbation; Preventive maintenance; Production scheduling; Real time; Reinforcement learning; Scheduling; Uncertain disturbances
• ISSN: 0305-0548; 1873-765X; 0305-0548
• DOI: 10.1016/j.cor.2022.105823

•An integrated optimization problem of flexible job shop scheduling and preventive maintenance is investigated.•A flexible maintenance strategy that integrates time-based maintenance and condition-based maintenance is proposed.•Both the machine resource and the worker resource are under consideration.•Four dynamic disturbances are considered, and respective rescheduling processes are illustrated.•A double-layer Q-learning algorithm is integrated with the digital twin for effective dynamic scheduling. Dynamic scheduling methods are essential and critical to manufacturing systems because of uncertain events in the production process, such as new job insertions, order cancellations, worker absences, and machine breakdowns. Emerging digital twin (DT) technology can help detect disturbances by continuously comparing physical space with virtual space and triggering a rescheduling policy immediately after a disturbance. This enables dynamic scheduling and greatly reduces the deviation between preschedules and actual schedules. This study focuses on a DT-enabled integrated optimisation problem of flexible job shop scheduling and flexible preventive maintenance (PM) considering both machine and worker resources. A double-layer Q-learning algorithm (DLQL) is designed as the underlying key optimisation method to simultaneously learn the selection process of machines and operations to achieve efficient real-time scheduling. The superior solution performance of DLQL was verified by comparing it with two well-known metaheuristic algorithms and a single-layer Q-learning algorithm under several benchmarks. Furthermore, different disturbance settings were designed to illustrate the DLQL-based dynamic scheduling process in detail. The proposed reinforcement learning (RL)-driven DT enables efficient collaborative scheduling between production and maintenance departments and helps manufacturing companies improve the real-time decision-making process under uncertain perturbations.