Scanner data in inflation measurement: From raw data to price indices

• Published in: Statistical journal of the IAOS 2021, Vol.37 (4), p.1315-1336
• Main Authors: Białek, Jacek, Berȩsewicz, Maciej
• Format: Article
• Language: English
• Subjects: Data processing; Mathematical analysis; Product names; Scanners
• ISSN: 1874-7655; 1875-9254
• DOI: 10.3233/SJI-210816

Scanner data offer new opportunities for CPI or HICP calculation. They can be obtained from a wide variety of retailers (supermarkets, home electronics, Internet shops, etc.) and provide information at the level of the barcode. One of advantages of using scanner data is the fact that they contain complete transaction information, i.e. prices and quantities for every sold item. After clearing data and unifying product names, products should be carefully classified (e.g. into COICOP 5 or below), matched, filtered and aggregated. One of new challenges connected with scanner data is the appropriate choice of the index formula. In this article we present a proposal for the implementation of individual stages of handling scanner data. We also point out potential problems during scanner data processing and their solutions. We compare a large number of price index methods based on real scanner data sets and we verify their sensitivity on adopted data filtering and aggregating methods. One of the aims is also to compare calculations of multilateral indices in terms of how time-consuming they are. Finally, the paper investigates distances between these indices and the theoretical, expected value of the price share when prices are log-normally distributed. It is a new approach to providing an additional criterion in the price index selection.