Ensembling Models for the Generation of Queries to an Altering Search Engine Using Reinforcement Learning

• Main Authors: Shodiev, Damir, Kopanichuk, Ilia, Chashchin, Artem, Grabovoy, Andrey, Kildyakov, Aleksandr, Chekhovich, Yury
• Format: Conference Proceeding
• Language: English
• Subjects: Indexes; information retrieval; Measurement; natural language processing; Natural languages; Reinforcement learning; Search engines; Search problems; upper confidence bound algorithm; Web pages
• ISSN: 2767-9535
• DOI: 10.1109/ISPRAS60948.2023.10508170

The automatic generation of queries to a search engine based on the incoming text is important for question-answering, recommendation, and text reuse detection systems. Every such query requires resources from a user and a search engine itself. A method of ensembling query generation models that maximizes the search completeness metric for the minimum number of queries could be useful. The task of selecting the best model or an ensemble of models is trivial for the case of a fixed search engine. However, real search engines are constantly changing their behavior, learning on incoming data, changing their index of web pages and documents. They are black boxes for a user. In this paper we propose an approach to ensemble query generation models based on reinforcement learning. By reformulating the problem so that the agent selects a sequence of models rather than a single query generation model, we guarantee maximum retrieval recall even when the worst possible action is selected. As a reward, we introduce a discount recall metric that penalizes the agent for each extra step of a model request. We modify the UCB learning algorithm so that the re-initialization of the recidivism penalty matrix occurs independently of the engine index state. In this way, we ensure that the top 3 best actions (i.e. sequences of generation model requests) are found in just 5 epochs, each epoch contains 1050 documents. The model ensemble maintains a stable performance even when the index alters in a way that the ensemble was not informed about.