Corruption Is Not All Bad: Incorporating Discourse Structure Into Pre-Training via Corruption for Essay Scoring

Existing approaches for automated essay scoring and document representation learning typically rely on discourse parsers to incorporate discourse structure into text representation. However, the performance of parsers is not always adequate, especially when they are used on noisy texts, such as stud...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on audio, speech, and language processing speech, and language processing, 2021-01, Vol.29, p.2202-2215
Main Authors: Mim, Farjana Sultana, Inoue, Naoya, Reisert, Paul, Ouchi, Hiroki, Inui, Kentaro
Format: Article
Language:English
Subjects:
Annotations
Automated Essay Scoring
Bit error rate
Coherence
Cohesion
Context modeling
Corruption
Discourse
Essays
Natural Language Processing
Organizations
Parsers
Pre-training
Representations
Speech processing
Task analysis
Training
Unsupervised Learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Existing approaches for automated essay scoring and document representation learning typically rely on discourse parsers to incorporate discourse structure into text representation. However, the performance of parsers is not always adequate, especially when they are used on noisy texts, such as student essays. In this paper, we propose an unsupervised pre-training approach to capture discourse structure of essays in terms of coherence and cohesion that does not require any discourse parser or annotation. We introduce several types of token, sentence and paragraph-level corruption techniques for our proposed pre-training approach and augment masked language modeling pre-training with our pre-training method to leverage both contextualized and discourse information. Our proposed unsupervised approach achieves a new state-of-the-art result on the task of essay Organization scoring.
ISSN:2329-9290
2329-9304
DOI:10.1109/TASLP.2021.3088223