Artificial Intelligence of Things (AIoT) Enabled Virtual Shop Applications Using Self‐Powered Sensor Enhanced Soft Robotic Manipulator

• Published in: Advanced science 2021-07, Vol.8 (14), p.e2100230-n/a
• Main Authors: Sun, Zhongda, Zhu, Minglu, Zhang, Zixuan, Chen, Zhaocong, Shi, Qiongfeng, Shan, Xuechuan, Yeow, Raye Chen Hua, Lee, Chengkuo
• Format: Article
• Language: English
• Subjects: Anthropomorphism; artificial intelligence; Cameras; Coronaviruses; COVID-19; Electronic commerce; Flexibility; Fluorides; Haptics; Internet of Things; object recognition; Robotics; Robots; Sensors; Skin; soft manipulator; triboelectric; virtual/augmented reality
• ISSN: 2198-3844; 2198-3844
• DOI: 10.1002/advs.202100230

Rapid advancements of artificial intelligence of things (AIoT) technology pave the way for developing a digital‐twin‐based remote interactive system for advanced robotic‐enabled industrial automation and virtual shopping. The embedded multifunctional perception system is urged for better interaction and user experience. To realize such a system, a smart soft robotic manipulator is presented that consists of a triboelectric nanogenerator tactile (T‐TENG) and length (L‐TENG) sensor, as well as a poly(vinylidene fluoride) (PVDF) pyroelectric temperature sensor. With the aid of machine learning (ML) for data processing, the fusion of the T‐TENG and L‐TENG sensors can realize the automatic recognition of the grasped objects with the accuracy of 97.143% for 28 different shapes of objects, while the temperature distribution can also be obtained through the pyroelectric sensor. By leveraging the IoT and artificial intelligence (AI) analytics, a digital‐twin‐based virtual shop is successfully implemented to provide the users with real‐time feedback about the details of the product. In general, by offering a more immersive experience in human–machine interactions, the proposed remote interactive system shows the great potential of being the advanced human–machine interface for the applications of the unmanned working space. A smart soft robotic manipulator is developed with a self‐powered multifunctional sensory system for simultaneously deformation, tactile, and temperature perception. With machine learning analysis, automatic recognition of grasped objects can be realized with high accuracy. By leveraging the artificial intelligence of things (AIoT) technology, a digital‐twin‐based virtual shop is successfully implemented to provide users with a more immersive shopping experience.