Learning Relation Models to Detect Important People in Still Images

• Published in: IEEE transactions on multimedia 2023-01, Vol.25, p.1-15
• Main Authors: Qiu, Yu-Kun, Hong, Fa-Ting, Li, Wei-Hong, Zheng, Wei-Shi
• Format: Article
• Language: English
• Subjects: Data models; Faces; Feature extraction; Graph neural networks; Important People Detection; Modules; Predictive models; Task analysis; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3211390

Important people detection aims to identify the most important people ( i.e. , the people who play the main roles in scenes) in images, which is challenging since people's importance in images depends not only on their appearance but also on their interactions with others ( i.e. , relations among people) and their roles in the scene ( i.e. , relations between people and underlying events). In this work, we propose the People Relation Network (PRN) to solve this problem. PRN consists of three modules ( i.e. , the feature representation, relation and classification modules) to extract visual features, model relations and estimate people's importance, respectively. The relation module contains two submodules to model two types of relations, namely, the person-person relation submodule and the person-event relation submodule. The person-person relation submodule infers the relations among people from the interaction graph and the person-event relation submodule models the relations between people and events by considering the spatial correspondence between features. With the help of them, PRN can effectively distinguish important people from other individuals. Extensive experiments on the Multi-Scene Important People (MS) and NCAA Basketball Image (NCAA) datasets show that PRN achieves state-of-the-art performance and generalizes well when available data is limited.