Multimodal Distributional Semantics

• Published in: The Journal of artificial intelligence research 2014-01, Vol.49, p.1-47
• Main Authors: Bruni, E., Tran, N. K., Baroni, M.
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Computational linguistics; Computer vision; Linguistics; Perceptions; Representations; Semantics; Simulation; Word meaning; Words (language)
• ISSN: 1076-9757; 1076-9757; 1943-5037
• DOI: 10.1613/jair.4135

Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete “visual words” in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.