WhittleSearch: Interactive Image Search with Relative Attribute Feedback

• Published in: International journal of computer vision 2015-11, Vol.115 (2), p.185-210
• Main Authors: Kovashka, Adriana, Parikh, Devi, Grauman, Kristen
• Format: Article
• Language: English
• Subjects: Algorithms; Analysis; Artificial Intelligence; Computational efficiency; Computer Imaging; Computer Science; Feedback; Image Processing and Computer Vision; Image retrieval; Mathematical analysis; Mathematical models; Pattern Recognition; Pattern Recognition and Graphics; Query processing; Searches; Searching; Semantics; Studies; Vision; Vision systems
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-015-0814-0

We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought. For example, perusing image results for a query “black shoes”, the user might state, “Show me shoe images like these, but sportier .” Offline, our approach first learns a set of ranking functions, each of which predicts the relative strength of a nameable attribute in an image (e.g., sportiness ). At query time, the system presents the user with a set of exemplar images, and the user relates them to his/her target image with comparative statements. Using a series of such constraints in the multi-dimensional attribute space, our method iteratively updates its relevance function and re-ranks the database of images. To determine which exemplar images receive feedback from the user, we present two variants of the approach: one where the feedback is user-initiated and another where the feedback is actively system-initiated. In either case, our approach allows a user to efficiently “whittle away” irrelevant portions of the visual feature space, using semantic language to precisely communicate her preferences to the system. We demonstrate our technique for refining image search for people, products, and scenes, and we show that it outperforms traditional binary relevance feedback in terms of search speed and accuracy. In addition, the ordinal nature of relative attributes helps make our active approach efficient—both computationally for the machine when selecting the reference images, and for the user by requiring less user interaction than conventional passive and active methods.