Natural Language to Visualization by Neural Machine Translation

Supporting the translation from natural language (NL) query to visualization (NL2VIS) can simplify the creation of data visualizations because if successful, anyone can generate visualizations by their natural language from the tabular data. The state-of-the-art NL2VIS approaches ( e.g. , NL4DV and...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on visualization and computer graphics 2022-01, Vol.28 (1), p.217-226
Main Authors: Luo, Yuyu, Tang, Nan, Li, Guoliang, Tang, Jiawei, Chai, Chengliang, Qin, Xuedi
Format: Article
Language:English
Subjects:
Algorithms
Artificial neural networks
Bars
chart template
Data visualization
Deep learning
Heuristic methods
Machine learning
Machine translation
Natural language
Natural language interface
Natural languages
neural machine translation
Query languages
Tables (data)
Transformers
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Supporting the translation from natural language (NL) query to visualization (NL2VIS) can simplify the creation of data visualizations because if successful, anyone can generate visualizations by their natural language from the tabular data. The state-of-the-art NL2VIS approaches ( e.g. , NL4DV and FlowSense) are based on semantic parsers and heuristic algorithms, which are not end-to-end and are not designed for supporting (possibly) complex data transformations. Deep neural network powered neural machine translation models have made great strides in many machine translation tasks, which suggests that they might be viable for NL2VIS as well. In this paper, we present ncNet , a Transformer-based sequence-to-sequence model for supporting NL2VIS, with several novel visualization-aware optimizations, including using attention-forcing to optimize the learning process, and visualization-aware rendering to produce better visualization results. To enhance the capability of machine to comprehend natural language queries, ncNet is also designed to take an optional chart template ( e.g. , a pie chart or a scatter plot) as an additional input, where the chart template will be served as a constraint to limit what could be visualized. We conducted both quantitative evaluation and user study, showing that ncNet achieves good accuracy in the nvBench benchmark and is easy-to-use.
ISSN:1077-2626
1941-0506
DOI:10.1109/TVCG.2021.3114848