A Tool for Introducing Computer Science with Automatic Formative Assessment

• Published in: IEEE transactions on learning technologies 2018-04, Vol.11 (2), p.179-192
• Main Authors: Benotti, Luciana, Martinez, Maria Cecilia, Schapachnik, Fernando
• Format: Article
• Language: English
• Subjects: Automata; automatic formative assessment; Competition; Computer Games; Computer Science; Computer Science Education; Computer Software; Education; Elementary Secondary Education; Feedback (Response); Foreign Countries; Formative Evaluation; Games; Gender Differences; High School Students; Interactive learning environments; Introductory Courses; K-12 education; Learner Engagement; Mixed Methods Research; Natural Language Processing; Natural languages; Pilot Projects; Pretests Posttests; Programming; Programming profession; Video Games
• ISSN: 1939-1382; 1939-1382; 2372-0050
• DOI: 10.1109/TLT.2017.2682084

In this paper we present a software platform called Chatbot designed to introduce high school students to Computer Science (CS) concepts in an innovative way: by programming chatbots. A chatbot is a bot that can be programmed to have a conversation with a human or robotic partner in some natural language such as English or Spanish. While programming their chatbots, students use fundamental CS constructs such as variables, conditionals, and finite state automata, among others. Chatbot uses pattern matching, state of the art lemmatization techniques, and finite state automata in order to provide automatic formative assessment to the students. When an error is found, the formative feedback generated is immediate and task-level. We evaluated Chatbot in two observational studies. An online nation-wide competition where more than 10,000 students participated. And, a mandatory in-class 15-lesson pilot course in three high schools. We measured indicators of student engagement (task completion, participation, self reported interest, etc.) and found that girls' engagement with Chatbot was higher than boys' for most indicators. Also, in the online competition, the task completion rate for the students that decided to use Chatbot was five times higher than for the students that chose to use the renowned animation and game programming tool Alice. Our results suggest that the availability of automatic formative assessment may have an impact on task completion and other engagement indicators among high school students.