Soft Decision Quantization for H.264 With Main Profile Compatibility

• Published in: IEEE transactions on circuits and systems for video technology 2009-01, Vol.19 (1), p.122-127
• Main Authors: Yang, En-hui, Yu, Xiang
• Format: Article
• Language: English
• Subjects: Algorithm design and analysis; Algorithms; Applied sciences; Arithmetic; Circuit properties; Coding; Coding, codes; Computer programs; Cost function; Design engineering; Design methodology; Design optimization; Electric, optical and optoelectronic circuits; Electronic circuits; Electronics; Encoding; Exact sciences and technology; H.264 hybrid coding; Information, signal and communications theory; Optimization; PSNR; Quantization; Rate-distortion; rate-distortion optimization (RDO); Reduction; Sampling, quantization; Signal and communications theory; Signal convertors; soft decision quantization; Software; Software algorithms; Studies; Telecommunications and information theory
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2008.2009260

In this paper, we study the rate-distortion (RD) optimization of the H.264 main profile encoding. Specifically, a soft decision quantization (SDQ) algorithm is developed based on the context adaptive binary arithmetic coding (CABAC) method in the H.264 main profile. Given motion prediction and quantization step sizes, the proposed SDQ algorithm is proved to achieve near-optimal SDQ for residual coding in the sense of minimizing the true RD cost when the weak adjacent block dependency utilized in CABAC is ignored for optimization. The SDQ algorithm is then used in conjunction with a general RD optimization framework to jointly design motion prediction and residual coding for H.264 main profile coding given previously coded reference frames. Experiments have been conducted based on the reference encoder JM82 of H.264 main profile. Comparative studies show that the joint design method achieves on average 10% rate reduction at the same PSNR when compared with the RD method in the H.264 main-profile reference software, with half of the reduction coming from the proposed SDQ algorithm, and 20% rate reduction at the same PSNR when compared with the RD method in the H.264 baseline-profile reference software.