Multidimensional Nonadditivity in One-Facet G-Theory Designs: A Profile Analytic Approach

• Published in: Psychological methods 2023-06, Vol.28 (3), p.651-663
• Main Authors: Grochowalski, Joseph H., Ayturk, Ezgi, Hendrickson, Amy
• Format: Article
• Language: English
• Subjects: Analysis of Variance; Error of Measurement; Neurocognition; Profiles (Measurement); Simulation; Statistical Analysis; Test Items; Test Scores
• ISSN: 1082-989X; 1939-1463
• DOI: 10.1037/met0000452

We introduce a new method for estimating the degree of nonadditivity in a one-facet generalizability theory design. One-facet G-theory designs have only one observation per cell, such as persons answering items in a test, and assume that there is no interaction between facets. When there is interaction, the model becomes nonadditive, and G-theory variance estimates and reliability coefficients are likely biased. We introduce a multidimensional method for detecting interaction and nonadditivity in G-theory that has less bias and smaller error variance than methods that use the one-degree of freedom method based on Tukey's test for nonadditivity. The method we propose is more flexible and detects a greater variety of interactions than the formulation based on Tukey's test. Further, the proposed method is descriptive and illustrates the nature of the facet interaction using profile analysis, giving insight into potential interaction like rater biases, DIF, threats to test security, and other possible sources of systematic construct-irrelevant variance. We demonstrate the accuracy of our method using a simulation study and illustrate its descriptive profile features with a real data analysis of neurocognitive test scores. Translational AbstractPsychological researchers often analyze scores from psychometric tests, and test scores are typically unidimensional measures of psychological traits. Researchers using test scores-and score reliability coefficients like Cronbach's alpha or from generalizability theory-assume there is no interaction effect between items and persons, which is why item responses can be generalized into a single unidimensional score. In this article we introduce a method for psychologists to assess the degree of item and person interaction that their scores conceal, which can broaden the utility of scores while increasing score reliability and validity. The method uses generalizability theory and nonadditivity analysis to estimate the degree of interaction in scores, and the estimated interaction is analyzed via profile analysis to identify the potential sources and nature of the interaction. The interaction profiles provide utility in research for purposes like diagnosis, subgroup identification, invariance analysis, validity research, or for improving score reliability. For this article we conducted two simulation analyses, evaluated our method's accuracy under different conditions, and illustrated the method using a neuropsychometric ana