Text Model for the Automatic Scoring of Business Letter Writing

• Published in: Automatic control and computer sciences 2023-12, Vol.57 (7), p.828-840
• Main Authors: Zafievsky, D. D., Lagutina, N. S., Melnikova, O. A., Poletaev, A. Y.
• Format: Article
• Language: English
• Subjects: Algorithms; Automatic text analysis; Computer Science; Control Structures and Microprogramming; Criteria; Dictionaries; English language; Graphical representations; Mathematical models; Numerical methods; Parameters; Sentences; Teachers; Texts
• ISSN: 0146-4116; 1558-108X
• DOI: 10.3103/S0146411623070167

This article describes a text model designed to automatically score a cohesive text in the form of a letter on a theme. The scoring parameters are formulated and formalized in the form of 14 criteria with the help of expert English language teachers. The criteria include parameters related to the analysis of vocabulary, including the features of the data domain, text subject, writing style and format, and logical connection in sentences. The authors have developed algorithms for determining the corresponding numerical characteristics using methods and tools for automatic text analysis. The algorithms are based on the analysis of the composition and structure of sentences, using data from specialized dictionaries. The characteristics are focused on checking business e-mails, but can be adapted to the analysis of other written texts, for example, by replacing dictionaries. Based on the developed algorithms, a system for automatic text scoring is created. An experiment is carried out to analyze the results of this system’s operation on a corpus of 20 texts, previously marked up by English teachers. Automatic scoring and the scoring of experts are compared using heat maps and the the UMAP two-dimensional representation of vectors applied to the characteristic text vectors. In most cases, there are no significant differences between the scores; moreover, automatic scoring turns out to be more objective. Thus, the developed model successfully copes with this task and can be used to evaluate texts written by humans. The results will be used for automatic student language profiling. The advantages of the model lie in the good interpretability of the results, credibility, and development prospects.