Learning of Multimodal Representations With Random Walks on the Click Graph

In multimedia information retrieval, most classic approaches tend to represent different modalities of media in the same feature space. With the click data collected from the users' searching behavior, existing approaches take either one-to-one paired data (text-image pairs) or ranking examples...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2016-02, Vol.25 (2), p.630-642
Main Authors: Fei Wu, Xinyan Lu, Jun Song, Shuicheng Yan, Zhang, Zhongfei Mark, Yong Rui, Yueting Zhuang
Format: Article
Language:English
Subjects:
click log
cross-media search
Data models
deep learning
Electronic mail
Graph theory
Graphical representations
Graphs
Image processing
latent representation
Neural networks
Queries
Random walk
Retrieval
Retrieval performance measures
Search engines
Semantics
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In multimedia information retrieval, most classic approaches tend to represent different modalities of media in the same feature space. With the click data collected from the users' searching behavior, existing approaches take either one-to-one paired data (text-image pairs) or ranking examples (text-query-image and/or image-query-text ranking lists) as training examples, which do not make full use of the click data, particularly the implicit connections among the data objects. In this paper, we treat the click data as a large click graph, in which vertices are images/text queries and edges indicate the clicks between an image and a query. We consider learning a multimodal representation from the perspective of encoding the explicit/implicit relevance relationship between the vertices in the click graph. By minimizing both the truncated random walk loss as well as the distance between the learned representation of vertices and their corresponding deep neural network output, the proposed model which is named multimodal random walk neural network (MRW-NN) can be applied to not only learn robust representation of the existing multimodal data in the click graph, but also deal with the unseen queries and images to support cross-modal retrieval. We evaluate the latent representation learned by MRW-NN on a public large-scale click log data set Clickture and further show that MRW-NN achieves much better cross-modal retrieval performance on the unseen queries/images than the other state-of-the-art methods.
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2015.2507401