Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition

We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the t...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-03
Main Authors: den Hengst, Floris, Wolter, Ralf, Altmeyer, Patrick, Kaygan, Arda
Format: Article
Language:English
Subjects:
Classification
Confidence intervals
Questions
Recognition
User experience
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level. By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection. In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection. CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
ISSN:2331-8422