Generation of Coherent Multi-Sentence Texts with a Coherence Mechanism

• Published in: Computer speech & language 2023-03, Vol.78, p.101457, Article 101457
• Main Authors: Zhao, Qingjuan, Niu, Jianwei, Liu, Xuefeng, He, Wenbo, Tang, Shaojie
• Format: Article
• Language: English
• Subjects: Abstractive summarization; Attention mechanism; Coherence mechanism; Multiple sentences summarization; Text generation
• ISSN: 0885-2308; 1095-8363
• DOI: 10.1016/j.csl.2022.101457

Automatic generation of long texts containing multiple sentences has many applications in the field of Natural Language Processing (NLP) including question answering, machine translation, and paraphrase generation, etc. However, in terms of readability, the long texts generated by machines are not comparable to those organized by human beings. Through statistics, we observed that human-organized texts generally have a special property: one or more of the words (particularly nouns and pronouns) appeared in one sentence will reappear in the next one in the same or a different form. This repetition of words in consecutive sentences can greatly improve the readability. Based on this observation, we propose CMST, a deep neural network model for generating Coherent Multi-Sentence Texts. CMST explicitly incorporates a training strategy of coherence mechanism to evaluate the repetition of words in consecutive sentences. We evaluate the performance of the CMST on the CNN/Daily Mail dataset. The experimental results show that, compared with the baseline models, CMST not only improves the readability of the generated texts, but achieves higher METEOR and ROUGE values. •The reappearance of words in adjacent sentences could make the text read coherently.•The coherence level is defined and incorporated into the objective function.•The results show that the generated texts achieve higher evaluation scores.