Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test

• Published in: Medical science educator 2015-12, Vol.25 (4), p.439-446
• Main Authors: Wilson, Adam B., Pike, Gary R., Humbert, Aloysius J.
• Format: Article
• Language: English
• Subjects: Education; Medical Education; Original Research
• ISSN: 2156-8650; 2156-8650
• DOI: 10.1007/s40670-015-0178-7

Background To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods This retrospective data analysis utilized scores from medical students ( n  = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.