On the efficiency of data collection for multiple Naïve Bayes classifiers

• Published in: Artificial intelligence 2019-10, Vol.275, p.356-378
• Main Authors: Manino, Edoardo, Tran-Thanh, Long, Jennings, Nicholas R.
• Format: Article
• Language: English
• Subjects: Accuracy; Active learning; Adaptive sampling; Classification; Classifiers; Crowdsourcing; Data collection; Decay rate; Empirical analysis; Naïve Bayes; Policies; Random walk; Upper bounds
• ISSN: 0004-3702; 1872-7921
• DOI: 10.1016/j.artint.2019.06.010

Many classification problems are solved by aggregating the output of a group of distinct predictors. In this respect, a popular choice is to assume independence and employ a Naïve Bayes classifier. When we have not just one but multiple classification problems at the same time, the question of how to assign the limited pool of available predictors to the individual classification problems arises. Empirical studies show that the policies we use to perform such assignments have a strong impact on the accuracy of the system. However, to date there is little theoretical understanding of this phenomenon. To help rectify this, in this paper we provide the first theoretical explanation of the accuracy gap between the most popular policies: the non-adaptive uniform allocation, and the adaptive allocation schemes based on uncertainty sampling and information gain maximisation. To do so, we propose a novel representation of the data collection process in terms of random walks. Then, we use this tool to derive new lower and upper bounds on the accuracy of the policies. These bounds reveal that the tradeoff between the number of available predictors and the accuracy has a different exponential rate depending on the policy used. By comparing them, we are able to quantify the advantage that the two adaptive policies have over the non-adaptive one for the first time, and prove that the probability of error of the former decays at more than double the exponential rate of the latter. Furthermore, we show in our analysis that this result holds both in the case where we know the accuracy of each individual predictor, and in the case where we only have access to a noisy estimate of it.