Speeding up prediction performance of BDT-based models

• Published in: Journal of physics. Conference series 2018-09, Vol.1085 (4), p.42009
• Main Authors: Khairullin, E., Ustyuzhanin, A.
• Format: Article
• Language: English
• Subjects: Algorithms; Decision trees; Machine learning; Physics; Prediction models
• ISSN: 1742-6588; 1742-6596
• DOI: 10.1088/1742-6596/1085/4/042009

The outcome of a machine learning algorithm is a prediction model. Typically, these models are computationally expensive, where improving of the quality the prediction leads to a decrease in the inference speed. However it is not always tradeoff between quality and speed. In this paper we show it is possible to speed up the model by using additional memory without losing significat prediction quality for a novel boosted trees algorithm called CatBoost. The idea is to combine two approaches: training fewer trees and merging trees into a kind of hashmaps called DecisionTensors. The proposed method allows for pareto-optimal reduction of the computational complexity of the decision tree model with regard to the quality of the model. In the considered example the number of lookups was decreased from 5000 to only 6 (speedup factor of 1000) while AUC score of the model was reduced by less than 10−3.