Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

Embodied AI is one of the most popular studies in artificial intelligence and robotics, which can effectively improve the intelligence of real-world agents (i.e. robots) serving human beings. Scene knowledge is important for an agent to understand the surroundings and make correct decisions in the v...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering 2024-11, Vol.36 (11), p.6962-6976
Main Authors: Song, Yaoxian, Sun, Penglei, Liu, Haoyu, Li, Zhixu, Song, Wei, Xiao, Yanghua, Zhou, Xiaofang
Format: Article
Language:English
Subjects:
Artificial intelligence
embodied AI
Knowledge based systems
Knowledge engineering
Knowledge graphs
Multimodal knowledge graph
robotic intelligence
Robots
scene driven
Task analysis
Visualization
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Embodied AI is one of the most popular studies in artificial intelligence and robotics, which can effectively improve the intelligence of real-world agents (i.e. robots) serving human beings. Scene knowledge is important for an agent to understand the surroundings and make correct decisions in the varied open world. Currently, knowledge base for embodied tasks is missing and most existing work use general knowledge base or pre-trained models to enhance the intelligence of an agent. For conventional knowledge base, it is sparse, insufficient in capacity and cost in data collection. For pre-trained models, they face the uncertainty of knowledge and hard maintenance. To overcome the challenges of scene knowledge, we propose a scene-driven multimodal knowledge graph (Scene-MMKG) construction method combining conventional knowledge engineering and large language models. A unified scene knowledge injection framework is introduced for knowledge representation. To evaluate the advantages of our proposed method, we instantiate Scene-MMKG considering typical indoor robotic functionalities ( Manip ulation and Mob ility), named ManipMob-MMKG . Comparisons in characteristics indicate our instantiated ManipMob-MMKG has broad superiority on data-collection efficiency and knowledge quality. Experimental results on typical embodied tasks show that knowledge-enhanced methods using our instantiated ManipMob-MMKG can improve the performance obviously without re-designing model structures complexly.
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2024.3399746