Automated Scoring for the "TOEFL Junior"® Comprehensive Writing and Speaking Test. Research Report. ETS RR-15-09

This report describes the initial automated scoring results that were obtained using the constructed responses from the Writing and Speaking sections of the pilot forms of the "TOEFL Junior"® Comprehensive test administered in late 2011. For all of the items except one (the edit item in th...

Full description

Saved in:
Bibliographic Details
Published in:ETS research report series 2015-06
Main Authors: Evanini, Keelan, Heilman, Michael, Wang, Xinhao, Blanchard, Daniel
Format: Article
Language:English
Subjects:
Audio Equipment
Automation
Children
Computer Assisted Testing
Correlation
English (Second Language)
Experiments
Language Tests
Man Machine Systems
Middle School Students
Narration
Pictorial Stimuli
Reading Aloud to Others
Scoring
Second Language Learning
Speech Tests
Test of English as a Foreign Language
Writing Tests
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This report describes the initial automated scoring results that were obtained using the constructed responses from the Writing and Speaking sections of the pilot forms of the "TOEFL Junior"® Comprehensive test administered in late 2011. For all of the items except one (the edit item in the Writing section), existing automated scoring capabilities were used with only minor modifications to obtain a baseline benchmark for automated scoring performance on the TOEFL Junior task types; for the edit item in the Writing section, a new automated scoring capability based on string matching was developed. A generic scoring model from the "e-rater"® automated essay scoring engine was used to score the email, opinion, and listen-write items in the Writing section, and the form-level results based on the five responses in the Writing section from each test taker showed a human-machine correlation of r = 0.83 (compared to a human-human correlation of r = 0.90). For scoring the Speaking section, new automated speech recognition models were first trained, and then item-specific scoring models were built for the read-aloud picture narration, and listen-speak items using preexisting features from the "SpeechRater"[SM] automated speech scoring engine (with the addition of a new content feature for the listen-speak items). The form-level results based on the five items in the Speaking section from each test taker showed a human-machine correlation of r =0.81 (compared to a human-human correlation of r = 0.89).
ISSN:2330-8516