TechnoSapiens: merging humans with technology in augmented reality

• Published in: The Visual computer 2024-02, Vol.40 (2), p.1021-1036
• Main Authors: Rudolph, Carsten, Brunnett, Guido, Bretschneider, Maximilian, Meyer, Bertolt, Asbrock, Frank
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Computer Graphics; Computer Science; Image Processing and Computer Vision; Original Article
• ISSN: 0178-2789; 1432-2315
• DOI: 10.1007/s00371-023-02829-7

We present a marker-less AR/DR system that can replace the arm of the user with a virtual bionic prosthesis in real time including finger tracking. For this, we use a mixed reality HMD that provides the user with a stereo image based on video-see-through (VST). We apply chroma-keying to remove the user’s arm from each captured image and input reconstructed background information into the removed pixels. Before rendering the prosthesis model into the image, we re-target motion capture data of the user’s hand to the kinematic skeleton of the prosthesis to match the current hand pose. This system opens new research possibilities on self- and other-perception of bionic bodies. In a first evaluation study of the system, we propose that users perceive the virtual prosthesis model as a part of their body (i.e., that they experience a sense of ownership). We tested this assumption in a laboratory study with 27 individuals who used the system to perform a series of simple tasks in AR with their prosthesis. We measured body ownership and other measures with self-reports. In support of the hypothesis, users experienced a sense of body ownership. Also, a feeling of self-presence is induced during the task, and participants rated the overall experience as positive.