Recognition of visual scene elements from a story text in Persian natural language

• Published in: Natural language engineering 2023-05, Vol.29 (3), p.693-719
• Main Authors: Hashemi-Namin, Mojdeh, Jahed-Motlagh, Mohammad Reza, Torkaman Rahmani, Adel
• Format: Article
• Language: English
• Subjects: Accuracy; Animation; Conditional random fields; Datasets; Error analysis; Labeling; Language; Machine learning; Mapping; Natural language; Natural language processing; Persian language; Representations; Semantic analysis; Semantics
• ISSN: 1351-3249; 1469-8110
• DOI: 10.1017/S1351324922000390

Text-to-scene conversion systems map natural language text to formal representations required for visual scenes. The difficulty involved in this mapping is one of the most critical challenges for developing these systems. The current study mapped Persian natural language text as the headmost system to a conceptual scene model. This conceptual scene model is an intermediate semantic representation between natural language and the visual scene and contains descriptions of visual elements of the scene. It will be used to produce meaningful animation based on an input story in this ongoing study. The mapping task was modeled as a sequential labeling problem, and a conditional random field (CRF) model was trained and tested for sequential labeling of scene model elements. To the best of the authors’ knowledge, no dataset for this task exists; thus, the required dataset was collected for this task. The lack of required off-the-shelf natural language processing modules and a significant error rate in the available corpora were important challenges to dataset collection. Some features of the dataset were manually annotated. The results were evaluated using standard text classification metrics, and an average accuracy of 85.7% was obtained, which is satisfactory.