Standard setting of script concordance tests using an adapted Nedelsky approach

• Published in: Medical teacher 2013-04, Vol.35 (4), p.314-319
• Main Authors: Linn, Andrew M.J., Tonkin, Anne, Duggan, Paul
• Format: Article
• Language: English
• Subjects: Achievement; Achievement tests; Assessment; College students; Comparative analysis; Concordance; Criteria; Education, Medical, Undergraduate - standards; Educational Measurement - standards; Exercise; Feasibility; Feasibility Studies; Grouping (Instructional Purposes); Health education; Humans; Medical students; Scripts; Standard Setting; Standard Setting (Scoring); Standardized tests; Statistical Data
• ISSN: 0142-159X; 1466-187X
• DOI: 10.3109/0142159X.2012.746446

Background: Standard setting in assessment seeks to apply meaning of achievement to an assessment score. Appropriate standard setting for script concordance tests (SCTs) remains a challenge, with existing methods representing norm-referenced approaches. Aims: To develop a criterion-referenced standard setting approach for SCT using an adapted Nedelsky approach, to pilot feasibility, and to compare failure rates with two other methods. Methods: Second- and third-year medical students were administered a 45-question SCT and results collated. Standard setting was applied using three approaches: (1) norm-referenced (student cohorts), (2) expert-referenced (student cohort compared to expert mean), and (3) adapted Nedelsky approach using answer key normalization. Feasibility and failure rates were measured. Results: All standard setting approaches were feasible, with 60 additional minutes required for the Nedelsky standard setting exercise. Failure rates between the three approaches were similar (Year 2: 8.0-9.8% and Year 3: 2.1-7.6%), with the adapted Nedelsky approach representing an intermediate option (Year 2: 8.0% and Year 3: 3.5%). Conclusion: Standard setting SCT using the criterion-referenced method of an adapted Nedelsky approach was found to be both logically justifiable and logistically simple, and produced failure rates comparable to other currently utilized and less objective approaches.