Estimating the imageability of words by mining visual characteristics from crawled image data

Natural Language Processing and multi-modal analyses are key elements in many applications. However, the semantic gap is an everlasting problem, leading to unnatural results disconnected from the user’s perception. To understand semantics in multimedia applications, human perception needs to be take...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia tools and applications 2020-07, Vol.79 (25-26), p.18167-18199
Main Authors: Kastner, Marc A., Ide, Ichiro, Nack, Frank, Kawanishi, Yasutomo, Hirayama, Takatsugu, Deguchi, Daisuke, Murase, Hiroshi
Format: Article
Language:English
Subjects:
Annotations
Computer Communication Networks
Computer Science
Concrete
Crowdsourcing
Data mining
Data Structures and Information Theory
Dictionaries
Experiments
Imageability
Information retrieval
Language acquisition
Multimedia
Multimedia Information Systems
Multimodality
Natural language processing
Perception
Perceptions
Semantics
Social networks
Special Purpose and Application-Based Systems
Webs
Words (language)
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Natural Language Processing and multi-modal analyses are key elements in many applications. However, the semantic gap is an everlasting problem, leading to unnatural results disconnected from the user’s perception. To understand semantics in multimedia applications, human perception needs to be taken into consideration. Imageability is an approach originating from Pyscholinguistics to quantize the human perception of words. Research shows a relationship between language usage and the imageability of words, making it useful for multimodal applications. However, the creation of imageability datasets is often manual and labor-intensive. In this paper, we propose a method using image data mining of a variety of visual features to estimate the imageability of words. The main assumption is a relationship between the imageability of concepts, human perception, and the contents of Web-crawled images. Using a set of low- and high-level visual features from Web-crawled images, a model is trained to predict imageability. The evaluations show that the imageability can be predicted with both a sufficiently low error, and a high correlation to the ground-truth annotations. The proposed method can be used to increase the corpus of imageability dictionaries.
ISSN:1380-7501
1573-7721
DOI:10.1007/s11042-019-08571-4