Micro-object pose estimation with sim-to-real transfer learning using small dataset

• Published in: Communications physics 2022-04, Vol.5 (1), p.1-11, Article 80
• Main Authors: Zhang, Dandan, Barbot, Antoine, Seichepine, Florent, Lo, Frank P.-W., Bai, Wenjia, Yang, Guang-Zhong, Lo, Benny
• Format: Article
• Language: English
• Subjects: 639/624/1107/328; 639/766/25; Datasets; Deep learning; Feedback control; Generative adversarial networks; Imaging techniques; Mathematical models; Optical microscopes; Physics; Physics and Astronomy; Pose estimation; Scanning electron microscopy; Scanning tunneling microscopy; Training
• ISSN: 2399-3650; 2399-3650
• DOI: 10.1038/s42005-022-00844-z

Three-dimensional (3D) pose estimation of micro/nano-objects is essential for the implementation of automatic manipulation in micro/nano-robotic systems. However, out-of-plane pose estimation of a micro/nano-object is challenging, since the images are typically obtained in 2D using a scanning electron microscope (SEM) or an optical microscope (OM). Traditional deep learning based methods require the collection of a large amount of labeled data for model training to estimate the 3D pose of an object from a monocular image. Here we present a sim-to-real learning-to-match approach for 3D pose estimation of micro/nano-objects. Instead of collecting large training datasets, simulated data is generated to enlarge the limited experimental data obtained in practice, while the domain gap between the generated and experimental data is minimized via image translation based on a generative adversarial network (GAN) model. A learning-to-match approach is used to map the generated data and the experimental data to a low-dimensional space with the same data distribution for different pose labels, which ensures effective feature embedding. Combining the labeled data obtained from experiments and simulations, a new training dataset is constructed for robust pose estimation. The proposed method is validated with images from both SEM and OM, facilitating the development of closed-loop control of micro/nano-objects with complex shapes in micro/nano-robotic systems. High-resolution scanning tunnelling microscopy is a state-of-the-art imaging technique at the nanometer scale. This work presents a novel deep learning approach for 3D pose estimation of micro/nano-objects, particularly useful in regimes of limited experimental data.