Research experiences instrument: Validation evidence for an instrument to assess the research experiences of engineering PhD students' professional practice opportunities

• Published in: Journal of engineering education (Washington, D.C.) D.C.), 2022-04, Vol.111 (2), p.420-445
• Main Authors: Holloway, Eric A., Douglas, Kerrie A., Radcliffe, David F., Oakes, William C.
• Format: Article
• Language: English
• Subjects: assessment; Doctoral Students; Engineering; Engineering Education; engineering PhD students; Ethnicity; Evidence; Factor Analysis; Factor Structure; Graduate Students; Measures (Individuals); Men; Opportunities; Professional practice; Professionals; Psychometrics; Race; Readiness; Reliability analysis; Reliability aspects; research experiences; Scores; Sex; Student Research; Students; Test Reliability; Test Validity; Validity
• ISSN: 1069-4730; 2168-9830
• DOI: 10.1002/jee.20451

Background There are long‐held concerns about how graduate research programs prepare engineering PhD students for professional practice. Suitable instruments are lacking to effectively assess how research experiences contribute to the success of graduate students becoming professionals. Purpose The purpose of this work is to examine evidence of internal reliability and validity of using the research experiences instrument (REI) scores as a measure of engineering PhD students' professional practice opportunities in their research experiences. Method REI was constructed using an ontological framework. REI was administered twice to engineering PhD students, once to a single university (n = 236) and once to multiuniversities (n = 215). Psychometric analyses were conducted related to validity and reliability evidence, including exploratory and confirmatory factor analyses, and group score comparisons between genders, race/ethnicity, and engineering disciplines. Results Results of both factor analyses aligned with the theoretical five‐factor structure, with a second‐order Opportunity factor. Mean scores were the same between women and men, and for three engineering disciplines, but significantly lower for racially/ethnically minoritized groups. Factor scores indicate that students often lack opportunities to engage with professionals, a likely cause of students' professional practice struggles. Conclusions Evidence of validity has been provided to justify the use of REI to assess the unique research experiences of engineering PhD students as these relate to preparedness for professional practice. REI scores can be used for gender and race/ethnicity comparisons and are generalizable across engineering disciplines. Where students' REI scores show a lack of opportunities, remedial strategies can be implemented.