Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality

Over recent years, UK medical schools have moved to more integrated summative examinations. This paper analyses data from the written assessment of undergraduate medical students to investigate two key psychometric aspects of this type of high-stakes assessment. Firstly, the strength of the relation...

Full description

Saved in:
Bibliographic Details
Published in:Assessment and evaluation in higher education 2012-11, Vol.37 (7), p.787-804
Main Authors: Homer, Matt, Darling, Jonathan, Pell, Godfrey
Format: Article
Language:English
Subjects:
Academic Achievement
Accuracy
College students
Correlation
Difficulty Level
Ebel ratings
Educational evaluation
Examiners
Foreign Countries
High Stakes Tests
integrated examinations
Item Response Theory
Medical Schools
Medical Students
Psychometrics
Quantitative psychology
Standard Setting
Standard Setting (Scoring)
Student behavior
Summative Evaluation
Test Items
Undergraduate Students
unidimensionality
United Kingdom
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Over recent years, UK medical schools have moved to more integrated summative examinations. This paper analyses data from the written assessment of undergraduate medical students to investigate two key psychometric aspects of this type of high-stakes assessment. Firstly, the strength of the relationship between examiner predictions of item performance (as required under the Ebel standard setting method employed) and actual item performance ('facility') in the examination is explored. It is found that there is a systematic pattern of difference between these two measures, with examiners tending to underestimate the difficulty of items classified as relatively easy, and overestimating that of items classified harder. The implications of these differences for standard setting are considered. Secondly, the integration of the assessment raises the question as to whether the student total score in the exam can provide a single meaningful measure of student performance across a broad range of medical specialties. Therefore, Rasch measurement theory is employed to evaluate psychometric characteristics of the examination, including its dimensionality. Once adjustment is made for item interdependency, the examination is shown to be unidimensional with fit to the Rasch model implying that a single underlying trait, clinical knowledge, is being measured.
ISSN:0260-2938
1469-297X
DOI:10.1080/02602938.2011.573843