Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances

• Published in: Computers and education. Artificial intelligence 2023, Vol.5, p.100172, Article 100172
• Main Authors: Bernabei, Margherita, Colabianchi, Silvia, Falegnami, Andrea, Costantino, Francesco
• Format: Article
• Language: English
• Subjects: ChatGPT; Essay generation; Higher education; LLM
• ISSN: 2666-920X; 2666-920X
• DOI: 10.1016/j.caeai.2023.100172

The accessibility of advanced Artificial Intelligence-based tools, like ChatGPT, has made Large Language Models (LLMs) readily available to students. These LLMs can generate original written content to assist students in their academic assessments. With the rapid adoption of LLMs, exemplified by the popularity of OpenAI's ChatGPT, there is a growing need to explore their application in education. Few studies examine students' use of LLMs as learning tools. This paper focuses on the application of ChatGPT in engineering higher education through an in-depth case study. It investigates whether engineering students can generate high-quality university essays with LLMs assistance, whether existing LLMs identification systems can detect essays produced with LLMs, and how students perceive the usefulness and acceptance of LLMs in learning. The research adopts a deductive/inductive approach, combining conceptualization and empirical evidence analysis. The study involves mechanical and management engineering students, who compose essays using LLMs. The essay assessment showed good results, but some recommendations emerged for teachers and students. Thirteen LLMs detectors were tested without achieving satisfactory results, suggesting to avoid LLMs ban. In addition, students were administered a questionnaire based on constructs and items that follow the technology acceptance models available in the literature. The results contribute to qualitative evidence by highlighting possible future research and educational practices. •Exploration of LLMs, like ChatGPT, in engineering education and assessment of students' acceptance, perceptions, efficacy, and detection chances.•Engineering students produce high-quality essays with LLM assistance, maintaining academic performance.•LLM assistance benefits the students' understanding of the topics.•Thirteen LLM detectors tested achieved unsatisfactory results, and limitations.•LLMs availability emphasizes the importance of developing students' critical thinking and problem-solving skills.