Prosodic and temporal features for language modeling for dialog

• Published in: Speech communication 2012-02, Vol.54 (2), p.161-174
• Main Authors: Ward, Nigel G., Vega, Alejandro, Baumann, Timo
• Format: Article
• Language: English
• Subjects: Applied sciences; Dialog dynamics; Dialog state; Errors; Exact sciences and technology; Information, signal and communications theory; Inspiration; Interlocutor behavior; Mathematical models; Miscellaneous; Perplexity; Prediction; Prosody; Signal processing; Speech; Speech processing; Speech recognition; Spontaneous; Switchboard corpus; Switchboards; Telecommunications and information theory; Temporal logic; Time measurements; Verbmobil corpus; Word probabilities
• ISSN: 0167-6393; 1872-7182
• DOI: 10.1016/j.specom.2011.07.009

► Speakers’ underlying cognitive processes and states may be revealed by prosody. ► Features of the local prosodic context can help predict what words are likely next. ► Speaking rate, volume, pitch and time-until-utterance-end features were informative. ► A 8.4% perplexity reduction on the Switchboard corpus was obtained. ► In a recognizer, this gave up to a 1.0% relative reduction in word error rate. If we can model the cognitive and communicative processes underlying speech, we should be able to better predict what a speaker will do. With this idea as inspiration, we examine a number of prosodic and timing features as potential sources of information on what words the speaker is likely to say next. In spontaneous dialog we find that word probabilities do vary with such features. Using perplexity as the metric, the most informative of these included recent speaking rate, volume, and pitch, and time until end of utterance. Using simple combinations of such features to augment trigram language models gave up to a 8.4% perplexity benefit on the Switchboard corpus, and up to a 1.0% relative reduction in word error rate (0.3% absolute) on the Verbmobil II corpus.