Loading…

Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network

This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on computational imaging 2020, Vol.6, p.682-696
Main Authors: Peng, Jiayong, Xiong, Zhiwei, Wang, Yicheng, Zhang, Yueyi, Liu, Dong
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183
cites cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183
container_end_page 696
container_issue
container_start_page 682
container_title IEEE transactions on computational imaging
container_volume 6
creator Peng, Jiayong
Xiong, Zhiwei
Wang, Yicheng
Zhang, Yueyi
Liu, Dong
description This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.
doi_str_mv 10.1109/TCI.2020.2967148
format article
fullrecord <record><control><sourceid>proquest_ieee_</sourceid><recordid>TN_cdi_ieee_primary_8961135</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>8961135</ieee_id><sourcerecordid>2357205225</sourcerecordid><originalsourceid>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</originalsourceid><addsrcrecordid>eNo9kEFPAjEQhRujiUS5m3hp4nlx2u5ut0eygpIQNREuXpruMguLQLHtavz3FiFe5s3he5M3j5AbBgPGQN3PysmAA4cBV7lkaXFGelwIkagUxHncMykSSEV-SfrerwGApYqLIu-R13d0Nnlb2UAfcB9WdORDuzWhtTs6dnZLp-1yFei4xc2Czn27W9IhLe3uy266A2Q29Bk79yfh27qPa3LRmI3H_kmvyHw8mpVPyfTlcVIOp0nNCxkSVkmsjDTQVCqPE4TBCoVRiPUiraRkUqYN1CnWDJWK2RcNGFZHPFPACnFF7o53985-duiDXtvOxTxec5FJDhnnWaTgSNXOeu-w0XsX33M_moE-VKdjdfpQnT5VFy23R0uLiP94oXLGRCZ-ASA0adM</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2357205225</pqid></control><display><type>article</type><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><source>IEEE Electronic Library (IEL) Journals</source><creator>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</creator><creatorcontrib>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</creatorcontrib><description>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</description><identifier>ISSN: 2573-0436</identifier><identifier>EISSN: 2333-9403</identifier><identifier>DOI: 10.1109/TCI.2020.2967148</identifier><identifier>CODEN: ITCIAJ</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Artificial neural networks ; Cameras ; Chlorophyll ; convolutional neural network ; Correlation ; Datasets ; depth estimation ; Electrons ; Estimation ; Geometry ; Learning systems ; Light ; Light field ; Mapping ; Training ; Training data ; Videos ; Zero-shot learning</subject><ispartof>IEEE transactions on computational imaging, 2020, Vol.6, p.682-696</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</citedby><cites>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</cites><orcidid>0000-0001-9100-2906 ; 0000-0002-9787-7460</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/8961135$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,776,780,4010,27900,27901,27902,54771</link.rule.ids></links><search><creatorcontrib>Peng, Jiayong</creatorcontrib><creatorcontrib>Xiong, Zhiwei</creatorcontrib><creatorcontrib>Wang, Yicheng</creatorcontrib><creatorcontrib>Zhang, Yueyi</creatorcontrib><creatorcontrib>Liu, Dong</creatorcontrib><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><title>IEEE transactions on computational imaging</title><addtitle>TCI</addtitle><description>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</description><subject>Artificial neural networks</subject><subject>Cameras</subject><subject>Chlorophyll</subject><subject>convolutional neural network</subject><subject>Correlation</subject><subject>Datasets</subject><subject>depth estimation</subject><subject>Electrons</subject><subject>Estimation</subject><subject>Geometry</subject><subject>Learning systems</subject><subject>Light</subject><subject>Light field</subject><subject>Mapping</subject><subject>Training</subject><subject>Training data</subject><subject>Videos</subject><subject>Zero-shot learning</subject><issn>2573-0436</issn><issn>2333-9403</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2020</creationdate><recordtype>article</recordtype><recordid>eNo9kEFPAjEQhRujiUS5m3hp4nlx2u5ut0eygpIQNREuXpruMguLQLHtavz3FiFe5s3he5M3j5AbBgPGQN3PysmAA4cBV7lkaXFGelwIkagUxHncMykSSEV-SfrerwGApYqLIu-R13d0Nnlb2UAfcB9WdORDuzWhtTs6dnZLp-1yFei4xc2Czn27W9IhLe3uy266A2Q29Bk79yfh27qPa3LRmI3H_kmvyHw8mpVPyfTlcVIOp0nNCxkSVkmsjDTQVCqPE4TBCoVRiPUiraRkUqYN1CnWDJWK2RcNGFZHPFPACnFF7o53985-duiDXtvOxTxec5FJDhnnWaTgSNXOeu-w0XsX33M_moE-VKdjdfpQnT5VFy23R0uLiP94oXLGRCZ-ASA0adM</recordid><startdate>2020</startdate><enddate>2020</enddate><creator>Peng, Jiayong</creator><creator>Xiong, Zhiwei</creator><creator>Wang, Yicheng</creator><creator>Zhang, Yueyi</creator><creator>Liu, Dong</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0001-9100-2906</orcidid><orcidid>https://orcid.org/0000-0002-9787-7460</orcidid></search><sort><creationdate>2020</creationdate><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><author>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Artificial neural networks</topic><topic>Cameras</topic><topic>Chlorophyll</topic><topic>convolutional neural network</topic><topic>Correlation</topic><topic>Datasets</topic><topic>depth estimation</topic><topic>Electrons</topic><topic>Estimation</topic><topic>Geometry</topic><topic>Learning systems</topic><topic>Light</topic><topic>Light field</topic><topic>Mapping</topic><topic>Training</topic><topic>Training data</topic><topic>Videos</topic><topic>Zero-shot learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Peng, Jiayong</creatorcontrib><creatorcontrib>Xiong, Zhiwei</creatorcontrib><creatorcontrib>Wang, Yicheng</creatorcontrib><creatorcontrib>Zhang, Yueyi</creatorcontrib><creatorcontrib>Liu, Dong</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on computational imaging</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Peng, Jiayong</au><au>Xiong, Zhiwei</au><au>Wang, Yicheng</au><au>Zhang, Yueyi</au><au>Liu, Dong</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</atitle><jtitle>IEEE transactions on computational imaging</jtitle><stitle>TCI</stitle><date>2020</date><risdate>2020</risdate><volume>6</volume><spage>682</spage><epage>696</epage><pages>682-696</pages><issn>2573-0436</issn><eissn>2333-9403</eissn><coden>ITCIAJ</coden><abstract>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/TCI.2020.2967148</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0001-9100-2906</orcidid><orcidid>https://orcid.org/0000-0002-9787-7460</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 2573-0436
ispartof IEEE transactions on computational imaging, 2020, Vol.6, p.682-696
issn 2573-0436
2333-9403
language eng
recordid cdi_ieee_primary_8961135
source IEEE Electronic Library (IEL) Journals
subjects Artificial neural networks
Cameras
Chlorophyll
convolutional neural network
Correlation
Datasets
depth estimation
Electrons
Estimation
Geometry
Learning systems
Light
Light field
Mapping
Training
Training data
Videos
Zero-shot learning
title Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-28T21%3A21%3A07IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_ieee_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Zero-Shot%20Depth%20Estimation%20From%20Light%20Field%20Using%20A%20Convolutional%20Neural%20Network&rft.jtitle=IEEE%20transactions%20on%20computational%20imaging&rft.au=Peng,%20Jiayong&rft.date=2020&rft.volume=6&rft.spage=682&rft.epage=696&rft.pages=682-696&rft.issn=2573-0436&rft.eissn=2333-9403&rft.coden=ITCIAJ&rft_id=info:doi/10.1109/TCI.2020.2967148&rft_dat=%3Cproquest_ieee_%3E2357205225%3C/proquest_ieee_%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2357205225&rft_id=info:pmid/&rft_ieee_id=8961135&rfr_iscdi=true