Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores part 1: Algorithms, framework, and implementation

• Published in: International journal of psychophysiology 2021-08, Vol.166, p.174-187
• Main Authors: Clayson, Peter E., Carbine, Kaylie A., Baldwin, Scott A., Olsen, Joseph A., Larson, Michael J.
• Format: Article
• Language: English
• Subjects: ERP Reliability Analysis (ERA) Toolbox; Event-related potentials; Generalizability theory; Psychometrics; Reliability
• ISSN: 0167-8760; 1872-7697
• DOI: 10.1016/j.ijpsycho.2021.01.006

The reliability of event-related brain potential (ERP) scores depends on study context and how those scores will be used, and reliability must be routinely evaluated. Many factors can influence ERP score reliability; generalizability (G) theory provides a multifaceted approach to estimating the internal consistency and temporal stability of scores that is well suited for ERPs. G theory's approach possesses a number of advantages over classical test theory that make it ideal for pinpointing sources of error in observed scores. The current primer outlines the G-theory approach to estimating internal consistency (coefficients of equivalence) and test-retest reliability (coefficients of stability). This approach is used to evaluate the reliability of ERP measurements. The primer outlines how to estimate reliability coefficients that consider the impact of the number of trials, events, occasions, and groups. The uses of two different G-theory reliability coefficients (i.e., generalizability and dependability) in ERP research are elaborated, and a dataset from the companion manuscript, which examines N2 amplitudes to Go/NoGo stimuli, is used as an example of the application of these coefficients to ERPs. The developed algorithms are implemented in the ERP Reliability Analysis (ERA) Toolbox, which is open-source software designed for estimating score reliability using G theory. The toolbox facilitates the application of G theory in an effort to simplify the study-by-study evaluation of ERP score reliability. The formulas provided in this primer should enable researchers to pinpoint the sources of measurement error in ERP scores from multiple recording sessions and subsequently plan studies that optimize score reliability. •Psychometric reliability is critical for studies of individual differences.•Generalizability theory (GT) can be used to assess test-retest reliability.•GT simultaneously evaluates internal consistency and test-retest reliability.•This is a primer on how to use GT to evaluate event related potentials (ERPs).•The ERP Reliability Analysis (ERA) Toolbox calculates reliability using GT.