Describing Complex Charts in Natural Language: A Caption Generation System

• Published in: Computational linguistics - Association for Computational Linguistics 1998, Vol.24 (3), p.431-467
• Main Authors: Mittal, Vibhu O, Moore, Johanna D, Carenini, Giuseppe, Roth, Steven
• Format: Article
• Language: English
• ISSN: 0891-2017

Graphical presentations can be used to communicate information in relational data sets succinctly & effectively. However, novel graphical presentations that represent many attributes & relationships are often difficult to understand completely until explained. Automatically generated graphical presentations must therefore either be limited to generating simple, conventionalized graphical presentations, or risk incomprehensibility. A possible solution to this problem would be to extend automatic graphical presentation systems to generate explanatory captions in natural language to enable users to understand the information expressed in the graphic. This paper presents a system that uses a text planner to determine the content & structure of the captions based on (1) a representation of the structure of the graphical presentation & its mapping to the data it depicts, (2) a framework for identifying the perceptual complexity of graphical elements, & (3) the structure of the data expressed in the graphic. The output of the planner is further processed regarding issues such as ordering, aggregation, centering, generating referring expressions, & lexical choice. We discuss the architecture of our system & its strengths & limitations. Our implementation is currently limited to 2-D charts & maps, but, except for lexical information, it is completely domain independent. We illustrate our discussion with figures & generated captions about housing sales in Pittsburgh. 1 Table, 22 Figures, 1 Appendix, 49 References. Adapted from the source document