Modeling Approaches for Cross-Sectional Integrative Data Analysis: Evaluations and Recommendations

• Published in: Psychological methods 2023-02, Vol.28 (1), p.242-261
• Main Authors: Wilcox, Kenneth Tyler, Wang, Lijuan
• Format: Article
• Language: English
• Subjects: Computer Simulation; Cross-Sectional Studies; Estimation; Freedom; Humans; Models, Statistical; Multilevel Analysis; Sample Size; Simulation; Statistical Analysis; Test Bias; Type I Errors
• ISSN: 1082-989X; 1939-1463
• DOI: 10.1037/met0000397

Integrative data analysis (IDA) jointly models participant-level data from multiple studies. In psychology, IDA has been conducted using different fixed-effects and multilevel modeling (MLM) approaches. However, evaluations regarding the performance of these models in an IDA context are limited. The goal of this article is to evaluate three fixed-effects models (aggregated vs. disaggregated vs. study-specific coefficients regressions) and two MLMs (fixed-slope vs. random-slopes MLM) for cross-sectional IDA. Using a simulation study with study sample sizes and numbers of studies consistent with applied IDA (e.g., two to 35 studies), we evaluated estimation bias and type I error rates for participant-level and study-level effects and variance components for these models; for the MLMs, we evaluated different estimation methods (i.e., constrained vs. unconstrained variance estimation and five degrees of freedom methods). Disaggregated and study-specific coefficients regressions and both MLMs yielded fixed effects estimates with ignorable bias, but only the random-slopes MLM fully modeled between-study heterogeneity and, consequently, provided well-controlled type I error rates for testing both fixed effects. Overall, we found that MLMs could be feasible under IDA conditions with three to six studies and well-chosen estimation methods. A real-data IDA example is used to illustrate and compare the application of the five models. We hope our results will help researchers choose appropriate modeling methods when conducting IDA. Translational AbstractIntegrative data analysis (IDA) is an alternative to meta-analysis that combines participant-level data from multiple studies for cumulative analysis. Two approaches, fixed effects models (FEM) and multilevel models (MLM), have been used in psychological applications of IDA but have not been fully evaluated. Because IDA combines data from multiple studies, two different kinds of fixed effects can be studied in IDA: study-level and participant-level effects. Furthermore, between-study differences need to be modeled carefully. For IDA with cross-sectional data, we reviewed three FEMs and two MLMs and theoretically discussed whether and how they can estimate and test participant-level and study-level fixed effects with sufficient data. We also evaluated the performance of these models and different MLM estimation methods in a simulation study under realistic IDA conditions (e.g., fewer than 30 studies). Although two of th