GPT-4 as an X data annotator: Unraveling its performance on a stance classification task

• Published in: PloS one 2024-08, Vol.19 (8), p.e0307741
• Main Authors: Liyanage, Chandreen R, Gokani, Ravi, Mago, Vijay
• Format: Article
• Language: English
• Subjects: Accuracy; Annotations; Artificial intelligence; Biology and Life Sciences; Chatbots; Classification; Cognition & reasoning; Computational linguistics; Computer and Information Sciences; Creative ability; Datasets; Dictionaries; Digital media; Humans; Investigations; Labeling; Labels; Language processing; Large language models; Machine Learning; Metadata; Natural language interfaces; Natural Language Processing; Prompt engineering; Social Media; Social networks; Social Sciences
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0307741

Data annotation in NLP is a costly and time-consuming task, traditionally handled by human experts who require extensive training to enhance the task-related background knowledge. Besides, labeling social media texts is particularly challenging due to their brevity, informality, creativity, and varying human perceptions regarding the sociocultural context of the world. With the emergence of GPT models and their proficiency in various NLP tasks, this study aims to establish a performance baseline for GPT-4 as a social media text annotator. To achieve this, we employ our own dataset of tweets, expertly labeled for stance detection with full inter-rater agreement among three annotators. We experiment with three techniques: Zero-shot, Few-shot, and Zero-shot with Chain-of-Thoughts to create prompts for the labeling task. We utilize four training sets constructed with different label sets, including human labels, to fine-tune transformer-based large language models and various combinations of traditional machine learning models with embeddings for stance classification. Finally, all fine-tuned models undergo evaluation using a common testing set with human-generated labels. We use the results from models trained on human labels as the benchmark to assess GPT-4's potential as an annotator across the three prompting techniques. Based on the experimental findings, GPT-4 achieves comparable results through the Few-shot and Zero-shot Chain-of-Thoughts prompting methods. However, none of these labeling techniques surpass the top three models fine-tuned on human labels. Moreover, we introduce the Zero-shot Chain-of-Thoughts as an effective strategy for aspect-based social media text labeling, which performs better than the standard Zero-shot and yields results similar to the high-performing yet expensive Few-shot approach.