Modeling the interaction between resilience and ability in assessments with allowances for multiple attempts

• Published in: Computers in human behavior 2021-09, Vol.122, p.106847, Article 106847
• Main Authors: Zhang, Susu, Bergner, Yoav, DiTrapani, Jack, Jeon, Minjeong
• Format: Article
• Language: English
• Subjects: Assessments; Colleges & universities; Computer-based assessment; Learning; Multiple attempt; Persistence; Resilience; Summative assessment; Tree-based item response models
• ISSN: 0747-5632; 1873-7692
• DOI: 10.1016/j.chb.2021.106847

Many computer-based assessments, in particular those administered in learning settings, provide immediate item-level feedback and permit respondents to retry questions following incorrect responses. Doing so supports learning and facilitates the measurement of partial mastery. However, persistence through attempts may introduce new factors, such as resilience, affecting a student's test score. The current study proposes a tree-based approach to jointly model item responses and reattempt decisions in multiple-attempt assessments, allowing for the separate measurement of latent ability and reattempt propensity, or resilience. The proposed model is implemented on two datasets collected from online homework assignments, one from a university for-credit course and the other from a massive open online course. The results shed light on the relationship between resilience and ability both at the individual level and at the item level, where threshold parameters can be interpreted as difficulty and repeatability. Importantly, resilience was found to affect certain commonly used performance scores, such as eventual correctness proportion, posing a validity threat for summative assessment use-cases. The tree-based model not only resolves the problem but also opens the door to ecologically valid measures of resilience that are not reliant on self-reports. •Eventual correctness on multiple-attempt test items depends on persistence, potentially complicating score interpretation. .•An IRTree model for multiple-attempt items parses out persistence from achievement and, thus, latent resilience from ability. .•Resilience, found uncorrelated with ability in online homework settings, should not be ignored as a factor in its own right. .•While some items decrease in difficulty on subsequent tries, most items do not.•Learners are less likely to reattempt difficult problems following incorrect response.