Enhanced Residue Prediction for Lossless Coding of Multimodal Image Pairs Based on Image-to-Image Translation

• Main Authors: Nicolau, Daniel S., Parracho, Joao O., Thomaz, Lucas A., Tavora, Luis M. N., Faria, Sergio M. M.
• Format: Conference Proceeding
• Language: English
• Subjects: Computed tomography; Encoding; Generative adversarial networks; Generative predictive coding; Image coding; Learning based prediction; Lossless image coding; Multimodal image coding; Prediction methods; Versatile Video Coding; Video coding; Visualization
• ISSN: 2471-8963
• DOI: 10.1109/EUVIP58404.2023.10323046

Multimodal medical imaging combine data obtained from multiple techniques simultaneously, yielding more detailed information about the content, which is a clear advantage over independent acquisition techniques. As these images are acquired using different imaging modalities and sometimes even in different dimensions, they commonly require a geometrical registration process. However, when they are encoded using standard image codecs the prediction methods do not exploit the redundancies related to the multimodal acquisition. In this paper, a novel lossless multimodal prediction module is introduced. The proposed method employs a deep learning-based approach with Image-to-Image translation for the purpose of joint coding of Positron Emission Tomography (PET) and Computed Tomography (CT) image pairs. Prior to the coding stage, a Generative Adversarial Network (GAN) is used for multimodal image translation. Then, a weighted estimated image is utilised as the I-frame, while the weighted sum of the original and synthesised image from the same modality serves as the P-frame for inter prediction. By employing weighted frames, the predictive frame approximates the reference frame more accurately, enhancing the overall performance of the prediction process. The experimental results, on a publicly available PET-CT dataset, demonstrate that the proposed prediction scheme outperformed the previously proposed method, and attains coding gains up to 13.20% when compared with the single modality intra coding of the Versatile Video Coding (VVC) lossless standard.