Mobile image restoration via prior quantization

• Published in: Pattern recognition letters 2023-10, Vol.174, p.64-70
• Main Authors: Chen, Shiqi, Zhou, Jingwen, Li, Menghao, Chen, Yueting, Jiang, Tingting
• Format: Article
• Language: English
• Subjects: Computational photography; Deep learning; Image restoration; Mobile ISP systems
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2023.08.017

In the photograph of mobile terminal, image degradation is a multivariate problem, where the spectral of the scene, the lens imperfections, the sensor noise, and the field of view together contribute to the results. Besides eliminating it at the hardware level, the post-processing system, which utilizes various prior information, is significant for correction. However, due to the content differences among priors, the pipeline that directly aligns these factors shows limited efficiency and unoptimized restoration. Here, we propose a prior quantization model to correct the degradation introduced in the image formation pipeline. To integrate the multivariate messages, we encode various priors into a latent space and quantify them by the learnable codebooks. After quantization, the prior codes are fused with the image restoration branch to realize targeted optical degradation correction. Moreover, we propose a comprehensive synthetic flow to acquire data pairs in a relative low computational overhead. Comprehensive experiments demonstrate the flexibility of the proposed method and validate its potential to accomplish targeted restoration for mass-produced mobile terminals. Furthermore, our model promises to analyze the influence of various priors and the degradation of devices, which is helpful for joint soft-hardware design. •Proposing a prior quantization model to utilize physical properties of camera.•Using a synthetic flow for generating the data pairs from corresponding situation.•Realizing flexible image restoration for mass-produced mobile terminals.•Providing a new perspective to assess the quality of the camera.