Towards Open-ended Visual Quality Comparison

Comparative settings (e.g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses. In this work, we extend the e...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-03
Main Authors: Wu, Haoning, Zhu, Hanwei, Zhang, Zicheng, Zhang, Erli, Chen, Chaofeng, Liao, Liang, Li, Chunyi, Wang, Annan, Sun, Wenxiu, Yan, Qiong, Liu, Xiaohong, Zhai, Guangtao, Wang, Shiqi, Lin, Weisi
Format: Article
Language:English
Subjects:
Benchmarks
Image quality
Quality assessment
Teachers
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Comparative settings (e.g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses. In this work, we extend the edge of emerging large multi-modality models (LMMs) to further advance visual quality comparison into open-ended settings, that 1) can respond to open-range questions on quality comparison; 2) can provide detailed reasonings beyond direct answers. To this end, we propose the Co-Instruct. To train this first-of-its-kind open-source open-ended visual quality comparer, we collect the Co-Instruct-562K dataset, from two sources: (a) LLM-merged single image quality description, (b) GPT-4V "teacher" responses on unlabeled data. Furthermore, to better evaluate this setting, we propose the MICBench, the first benchmark on multi-image comparison for LMMs. We demonstrate that Co-Instruct not only achieves in average 30% higher accuracy than state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher), on both existing related benchmarks and the proposed MICBench. Our model is published at https://huggingface.co/q-future/co-instruct.
ISSN:2331-8422