Exact marginal inference in Latent Dirichlet Allocation

• Published in: arXiv.org 2020-03
• Main Author: Maennel, Hartmut
• Format: Article
• Language: English
• Subjects: Algorithms; Bayesian analysis; Dirichlet problem; Upper bounds
• ISSN: 2331-8422

Assume we have potential "causes" \(z\in Z\), which produce "events" \(w\) with known probabilities \(\beta(w|z)\). We observe \(w_1,w_2,...,w_n\), what can we say about the distribution of the causes? A Bayesian estimate will assume a prior on distributions on \(Z\) (we assume a Dirichlet prior) and calculate a posterior. An average over that posterior then gives a distribution on \(Z\), which estimates how much each cause \(z\) contributed to our observations. This is the setting of Latent Dirichlet Allocation, which can be applied e.g. to topics "producing" words in a document. In this setting usually the number of observed words is large, but the number of potential topics is small. We are here interested in applications with many potential "causes" (e.g. locations on the globe), but only a few observations. We show that the exact Bayesian estimate can be computed in linear time (and constant space) in \(|Z|\) for a given upper bound on \(n\) with a surprisingly simple formula. We generalize this algorithm to the case of sparse probabilities \(\beta(w|z)\), in which we only need to assume that the tree width of an "interaction graph" on the observations is limited. On the other hand we also show that without such limitation the problem is NP-hard.