Scoring Constructed Responses Using Expert Systems

The use of constructed-response items in large scale standardized testing has been hampered by the costs and difficulties associated with obtaining reliable scores. The advent of expert systems may signal the eventual removal of this impediment. This study investigated the accuracy with which expert...

Full description

Saved in:
Bibliographic Details
Published in:Journal of educational measurement 1990-06, Vol.27 (2), p.93-108
Main Authors: Braun, Henry I., Bennett, Randy Elliot, Frye, Douglas, Soloway, Elliot
Format: Article
Language:English
Subjects:
Advanced placement
Computer programming
Computer software
Expert systems
High school students
Knowledge bases
Proportions
Rain
Software bugs
Standardized tests
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The use of constructed-response items in large scale standardized testing has been hampered by the costs and difficulties associated with obtaining reliable scores. The advent of expert systems may signal the eventual removal of this impediment. This study investigated the accuracy with which expert systems could score a new, nonmultiple-choice item type. The item type presents a faulty solution to a computer programming problem and asks the student to correct the solution. This item type was administered to a sample of high school seniors enrolled in an Advanced Placement course in Computer Science who also took the Advanced Placement Computer Science (APCS) examination. Results indicated that the expert systems were able to produce scores for between 82% and 95% of the solutions encountered and to display high agreement with a human reader on the correctness of the solutions. Diagnoses of the specific errors produced by students were less accurate. Correlations with scores on the objective and free-response sections of the APCS examination were moderate. Implications for additional research and for testing practice are offered.
ISSN:0022-0655
1745-3984
DOI:10.1111/j.1745-3984.1990.tb00736.x