EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation

• Published in: arXiv.org 2023-10
• Main Authors: Wang, You, Wu, Wenshan, Liang, Yaobo, Mao, Shaoguang, Wu, Chenfei, Cao, Maosong, Cai, Yuzhe, Guo, Yiduo, Xia, Yan, Furu Wei, Duan, Nan
• Format: Article
• Language: English
• Subjects: Iterative methods; Large language models; Narratives; Natural language processing
• ISSN: 2331-8422

Plan-and-Write is a common hierarchical approach in long-form narrative text generation, which first creates a plan to guide the narrative writing. Following this approach, several studies rely on simply prompting large language models for planning, which often yields suboptimal results. In this paper, we propose a new framework called Evaluation-guided Iterative Plan Extraction for long-form narrative text generation (EIPE-text), which extracts plans from the corpus of narratives and utilizes the extracted plans to construct a better planner. EIPE-text has three stages: plan extraction, learning, and inference. In the plan extraction stage, it iteratively extracts and improves plans from the narrative corpus and constructs a plan corpus. We propose a question answer (QA) based evaluation mechanism to automatically evaluate the plans and generate detailed plan refinement instructions to guide the iterative improvement. In the learning stage, we build a better planner by fine-tuning with the plan corpus or in-context learning with examples in the plan corpus. Finally, we leverage a hierarchical approach to generate long-form narratives. We evaluate the effectiveness of EIPE-text in the domains of novels and storytelling. Both GPT-4-based evaluations and human evaluations demonstrate that our method can generate more coherent and relevant long-form narratives. Our code will be released in the future.