MMDF-LDA: An improved Multi-Modal Latent Dirichlet Allocation model for social image annotation

• Published in: Expert systems with applications 2018-08, Vol.104, p.168-184
• Main Authors: Zheng, Liu, Caiming, Zhang, Caixian, Chen
• Format: Article
• Language: English
• Subjects: Data integration; Dirichlet problem; Geographical topic; Image annotation; Image enhancement; LDA model; Modal data; Multi-modal data fusion; Multimedia; Multisensor fusion; Patches (structures); Semantic annotation; Semantic web; Social image; Social networks; Tags; Web sites
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2018.03.014

•A multi-modal data fusion model for social images annotation is proposed.•A probability topic model is learned by fusing multi-modal metadata.•Geographical topics are generated from geographical region of social images.•Patches of social images are annotated by the proposed model.•Experiments demonstrate the effectiveness of the proposed solution. Social image annotation, which aims at inferring a set of semantic concepts for a social image, is an effective and straightforward way to facilitate social image search. Conventional approaches mainly demonstrated on adopting the visual features and tags, without considering other types of metadata. How to enhance the accuracy of social image annotation by fully exploiting multi-modal features is still an opening and challenging problem. In this paper, we propose an improved Multi-Modal Data Fusion based Latent Dirichlet Allocation (LDA) topic model (MMDF-LDA) to annotate social images via fusing visual content, user-supplied tags, user comments, and geographic information. When MMDF-LDA samples annotations for one data modality, all the other data modalities are exploited. In MMDF-LDA, geographical topics are generated from GPS locations of social images, and annotations have different probability to be used in different geographical regions. A social image is divided into several patches in advance, and then MMDF-LDA assigns annotations for the patches of social images by estimating the probability of annotation-patch assignment. Through experiments in social image annotation and retrieval on several datasets, we demonstrate the effectiveness of the proposed MMDF-LDA model in comparison with state-of-the-art methods.