Multi-modal sorting in plastic and wood waste streams

• Published in: Resources, conservation and recycling conservation and recycling, 2023-12, Vol.199, p.107244, Article 107244
• Main Authors: Konstantinidis, Fotios K., Sifnaios, Savvas, Arvanitakis, George, Tsimiklis, Georgios, Mouroutsos, Spyridon G., Amditis, Angelos, Gasteratos, Antonios
• Format: Article
• Language: English
• Subjects: Machine vision; Multi-modal learning; Multi-spectral; Plastic-wood wastes; Waste management; Waste sorting
• ISSN: 0921-3449; 1879-0658
• DOI: 10.1016/j.resconrec.2023.107244

Addressing the escalating waste crisis necessitates innovative waste management strategies, particularly valorisation techniques, the efficiency of which is dictated by the purity of the feedstock. In order to mitigate the segregation challenges encountered in complex and non-homogeneous waste streams, this work proposes a vision-based architecture ample for effective sorting of parts based on shape and material-related properties. The proposed work encapsulates a novel deep learning multi-modal approach, in which multiple parallel auto-encoders are used to extract spatio-spectral information from the RGB and multi-spectral sensors and project them in a common latent space. By decoding the latent space representations, the class of each object is picked out, thus guiding the robotic sub-system accordingly. To support the proposed deep architecture, a dataset, called Multispectral Mixed Waste Dataset (MMWD) was produced, containing multi-spectral data from the visible (16 bands), near-infrared (25 bands) regions of the electromagnetic spectrum and RGB (3 bands) data. The dataset includes the following seven plastic and wood wastes: Polypropylene (PP), PolyEthylene Terephthalate (PET), Low-Density PolyEthylene (LDPE), High-Density PolyEthylene (HDPE), Medium Density Fibreboard (MDF), Melamine Faced Chipboards (MFC), and Oak Veneer samples. For the localisation of waste along the conveyor belt, YOLO v8 is used to achieve 99.5% mean average precision (mAP50). In the classification task, where the multi-modal approach was followed, the overall accuracy achieved is 96% with the prediction recall being greater than 95% for the majority of classes under examination.