Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems

Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue...

Full description

Saved in:
Bibliographic Details
Published in:International journal of speech technology 2016-06, Vol.19 (2), p.373-383
Main Authors: AbuShawar, Bayan, Atwell, Eric
Format: Article
Language:English
Subjects:
Architecture
Artificial Intelligence
Black boxes
Cognitive style
Criteria
Customers
Engineering
Genre
Glass
Human-computer interaction
Languages
Natural language
Natural language processing
Naturalness
Platforms
Signal,Image and Speech Processing
Social Sciences
Special Issue Article
Speech
Systems analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: “usefulness” in terms of a user’s qualitative needs, “localizability” to new genres and languages, “humanness” or “naturalness” compared to human–human dialogues, and “language benefit” compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.
ISSN:1381-2416
1572-8110
DOI:10.1007/s10772-015-9330-4