A framework for creating natural language descriptions of video streams

• Published in: Information sciences 2015-05, Vol.303, p.61-82
• Main Authors: Khan, Muhammad Usman Ghani, Al Harbi, Nouf, Gotoh, Yoshihiko
• Format: Article
• Language: English
• Subjects: Categories; Coherence; Descriptions; Feature extraction; Human; Natural language generation; Natural language processing; Streams; Tasks; Video annotation; Video retrieval; Visual
• ISSN: 0020-0255; 1872-6291
• DOI: 10.1016/j.ins.2014.12.034

This contribution addresses generation of natural language descriptions for important visual content present in video streams. The work starts with implementation of conventional image processing techniques to extract high-level visual features such as humans and their activities. These features are converted into natural language descriptions using a template-based approach built on a context free grammar, incorporating spatial and temporal information. The task is challenging particularly because feature extraction processes are erroneous at various levels. In this paper we explore approaches to accommodating potentially missing information, thus creating a coherent description. Sample automatic annotations are created for video clips presenting humans’ close-ups and actions, and qualitative analysis of the approach is made from various aspects. Additionally a task-based scheme is introduced that provides quantitative evaluation for relevance of generated descriptions. Further, to show the framework’s potential for extension, a scalability study is conducted using video categories that are not targeted during the development.