Sources of difficulty in assessment: example of PISA science items

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our res...

Full description

Saved in:
Bibliographic Details
Published in:International journal of science education 2017-03, Vol.39 (4), p.468-487
Main Authors: Le Hebel, Florence, Montpied, Pascale, Tiberghien, Andrée, Fontanieu, Valérie
Format: Article
Language:English
Subjects:
Achievement Tests
assessment
Difficulty Level
Education
Foreign Countries
Humanities and Social Sciences
International Assessment
Item Analysis
Linguistics
Science Achievement
Science education
Science Instruction
Science Tests
Scientific Literacy
Scores
Secondary School Students
Statistical Analysis
Students' proficiencies
Test Items
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item's proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item's proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students' low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.
ISSN:0950-0693
1464-5289
DOI:10.1080/09500693.2017.1294784