Hybrid ASIC/FPGA System for Fully Automatic Stereo-to-Multiview Conversion Using IDW

• Published in: IEEE transactions on circuits and systems for video technology 2016-11, Vol.26 (11), p.2093-2108
• Main Authors: Schaffner, Michael, Gurkaynak, Frank K., Greisen, Pierre, Kaeslin, Hubert, Benini, Luca, Smolic, Aljosa
• Format: Article
• Language: English
• Subjects: Algorithm design and analysis; Algorithms; Application specific integrated circuits; Application-specific integrated circuit (ASIC); Computer architecture; Displays; Field programmable gate arrays; FPGA; Hardware; image domain warping (IDW); Lattices; Mathematical models; multiview synthesis (MVS); Real time; Real-time systems; stereoscopic 3D (S3D); Streaming media; Three-dimensional displays; very large scale integration (VLSI); Video; video processing
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2015.2501640

Recently, multiview autostereoscopic dis-plays (MADs), which enable a limited glasses-free 3D experience, have become commercially available. The main problem of MADs is that they require several (typically eight or nine) views, while most of the 3D video content is in stereoscopic 3D today. In order to bridge this gap, the research community started to devise automatic multiview synthesis (MVS) methods. These algorithms require real-time processing and should be portable to end-user devices to develop their full potential. To this end, we revisit an algorithmic solution based on image domain warping (IDW) and devise a hardware architecture of a complete synthesis pipeline, provide insights into where the computationally challenging parts are, and present implementation results of a hybrid field programmable gate array/application-specific integrated circuit prototype, which is the first hardware implementation of a complete IDW-based MVS system. Based on these results, we also estimate the complexity and energy efficiency of a fully integrated solution in 65- and 28-nm CMOS technology and show that a full-high-definition real-time solution on a single chip is within reach. The proposed architecture could be used as a coprocessor in a system-on-chip targeting 3D TV sets, thereby enabling efficient content generation with limited user interaction (e.g., depth range adjustment) in real time.