Conditional leaving-one-out and cross-validation for discount estimation in Kneser-Ney-like extensions

The smoothing of n-gram models is a core technique in language modelling (LM). Modified Kneser-Ney (mKN) ranges among one of the best smoothing techniques. This technique discounts a fixed quantity from the observed counts in order to approximate the Turing-Good (TG) counts. Despite the TG counts op...

Full description

Saved in:
Bibliographic Details
Main Authors: Andres-Ferrer, J., Sundermeyer, M., Ney, H.
Format: Conference Proceeding
Language:English
Subjects:
Approximation methods
Computational modeling
Cross Validation
Estimation
Language Modelling
Leaving-One-Out
modified Kneser-Ney smoothing
Optimization
Smoothing methods
Training
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The smoothing of n-gram models is a core technique in language modelling (LM). Modified Kneser-Ney (mKN) ranges among one of the best smoothing techniques. This technique discounts a fixed quantity from the observed counts in order to approximate the Turing-Good (TG) counts. Despite the TG counts optimise the leaving-one-out (L1O) criterion, the discounting parameters introduced in mKN do not. Moreover, the approximation to the TG counts for large counts is heavily simplified. In this work, both ideas are addressed: the estimation of the discounting parameters by L1O and better functional forms to approximate larger TG counts. The L1O performance is compared with cross-validation (CV) and mKN baseline in two large vocabulary tasks.
ISSN:1520-6149
2379-190X
DOI:10.1109/ICASSP.2012.6289046