Comparing Large Language Models for supervised analysis of students' lab notes

We compare the application of Bag of Words, BERT, and various flavors of LLaMA machine learning models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-12
Main Authors: Fussell, Rebeckah K, Flynn, Megan, Damle, Anil, Fox, Michael F J, Holmes, N G
Format: Article
Language:English
Subjects:
Education
Large language models
Machine learning
Performance measurement
Skills
Students
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We compare the application of Bag of Words, BERT, and various flavors of LLaMA machine learning models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling. We evaluate the models based on their resource use, performance metrics, and research outcomes when identifying skills in lab notes. We find that higher-resource models often, but not necessarily, perform better than lower-resource models. We also find that all models estimate similar trends in research outcomes, although the absolute values of the estimated measurements are not always within uncertainties of each other. We use the results to discuss relevant considerations for education researchers seeking to select a model type to use as a classifier.
ISSN:2331-8422