Loading…
Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network
This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two...
Saved in:
Published in: | IEEE transactions on computational imaging 2020, Vol.6, p.682-696 |
---|---|
Main Authors: | , , , , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that this one cites Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
cited_by | cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183 |
---|---|
cites | cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183 |
container_end_page | 696 |
container_issue | |
container_start_page | 682 |
container_title | IEEE transactions on computational imaging |
container_volume | 6 |
creator | Peng, Jiayong Xiong, Zhiwei Wang, Yicheng Zhang, Yueyi Liu, Dong |
description | This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes. |
doi_str_mv | 10.1109/TCI.2020.2967148 |
format | article |
fullrecord | <record><control><sourceid>proquest_ieee_</sourceid><recordid>TN_cdi_ieee_primary_8961135</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>8961135</ieee_id><sourcerecordid>2357205225</sourcerecordid><originalsourceid>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</originalsourceid><addsrcrecordid>eNo9kEFPAjEQhRujiUS5m3hp4nlx2u5ut0eygpIQNREuXpruMguLQLHtavz3FiFe5s3he5M3j5AbBgPGQN3PysmAA4cBV7lkaXFGelwIkagUxHncMykSSEV-SfrerwGApYqLIu-R13d0Nnlb2UAfcB9WdORDuzWhtTs6dnZLp-1yFei4xc2Czn27W9IhLe3uy266A2Q29Bk79yfh27qPa3LRmI3H_kmvyHw8mpVPyfTlcVIOp0nNCxkSVkmsjDTQVCqPE4TBCoVRiPUiraRkUqYN1CnWDJWK2RcNGFZHPFPACnFF7o53985-duiDXtvOxTxec5FJDhnnWaTgSNXOeu-w0XsX33M_moE-VKdjdfpQnT5VFy23R0uLiP94oXLGRCZ-ASA0adM</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2357205225</pqid></control><display><type>article</type><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><source>IEEE Electronic Library (IEL) Journals</source><creator>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</creator><creatorcontrib>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</creatorcontrib><description>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</description><identifier>ISSN: 2573-0436</identifier><identifier>EISSN: 2333-9403</identifier><identifier>DOI: 10.1109/TCI.2020.2967148</identifier><identifier>CODEN: ITCIAJ</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Artificial neural networks ; Cameras ; Chlorophyll ; convolutional neural network ; Correlation ; Datasets ; depth estimation ; Electrons ; Estimation ; Geometry ; Learning systems ; Light ; Light field ; Mapping ; Training ; Training data ; Videos ; Zero-shot learning</subject><ispartof>IEEE transactions on computational imaging, 2020, Vol.6, p.682-696</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</citedby><cites>FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</cites><orcidid>0000-0001-9100-2906 ; 0000-0002-9787-7460</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/8961135$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,776,780,4010,27900,27901,27902,54771</link.rule.ids></links><search><creatorcontrib>Peng, Jiayong</creatorcontrib><creatorcontrib>Xiong, Zhiwei</creatorcontrib><creatorcontrib>Wang, Yicheng</creatorcontrib><creatorcontrib>Zhang, Yueyi</creatorcontrib><creatorcontrib>Liu, Dong</creatorcontrib><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><title>IEEE transactions on computational imaging</title><addtitle>TCI</addtitle><description>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</description><subject>Artificial neural networks</subject><subject>Cameras</subject><subject>Chlorophyll</subject><subject>convolutional neural network</subject><subject>Correlation</subject><subject>Datasets</subject><subject>depth estimation</subject><subject>Electrons</subject><subject>Estimation</subject><subject>Geometry</subject><subject>Learning systems</subject><subject>Light</subject><subject>Light field</subject><subject>Mapping</subject><subject>Training</subject><subject>Training data</subject><subject>Videos</subject><subject>Zero-shot learning</subject><issn>2573-0436</issn><issn>2333-9403</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2020</creationdate><recordtype>article</recordtype><recordid>eNo9kEFPAjEQhRujiUS5m3hp4nlx2u5ut0eygpIQNREuXpruMguLQLHtavz3FiFe5s3he5M3j5AbBgPGQN3PysmAA4cBV7lkaXFGelwIkagUxHncMykSSEV-SfrerwGApYqLIu-R13d0Nnlb2UAfcB9WdORDuzWhtTs6dnZLp-1yFei4xc2Czn27W9IhLe3uy266A2Q29Bk79yfh27qPa3LRmI3H_kmvyHw8mpVPyfTlcVIOp0nNCxkSVkmsjDTQVCqPE4TBCoVRiPUiraRkUqYN1CnWDJWK2RcNGFZHPFPACnFF7o53985-duiDXtvOxTxec5FJDhnnWaTgSNXOeu-w0XsX33M_moE-VKdjdfpQnT5VFy23R0uLiP94oXLGRCZ-ASA0adM</recordid><startdate>2020</startdate><enddate>2020</enddate><creator>Peng, Jiayong</creator><creator>Xiong, Zhiwei</creator><creator>Wang, Yicheng</creator><creator>Zhang, Yueyi</creator><creator>Liu, Dong</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0001-9100-2906</orcidid><orcidid>https://orcid.org/0000-0002-9787-7460</orcidid></search><sort><creationdate>2020</creationdate><title>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</title><author>Peng, Jiayong ; Xiong, Zhiwei ; Wang, Yicheng ; Zhang, Yueyi ; Liu, Dong</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Artificial neural networks</topic><topic>Cameras</topic><topic>Chlorophyll</topic><topic>convolutional neural network</topic><topic>Correlation</topic><topic>Datasets</topic><topic>depth estimation</topic><topic>Electrons</topic><topic>Estimation</topic><topic>Geometry</topic><topic>Learning systems</topic><topic>Light</topic><topic>Light field</topic><topic>Mapping</topic><topic>Training</topic><topic>Training data</topic><topic>Videos</topic><topic>Zero-shot learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Peng, Jiayong</creatorcontrib><creatorcontrib>Xiong, Zhiwei</creatorcontrib><creatorcontrib>Wang, Yicheng</creatorcontrib><creatorcontrib>Zhang, Yueyi</creatorcontrib><creatorcontrib>Liu, Dong</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on computational imaging</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Peng, Jiayong</au><au>Xiong, Zhiwei</au><au>Wang, Yicheng</au><au>Zhang, Yueyi</au><au>Liu, Dong</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network</atitle><jtitle>IEEE transactions on computational imaging</jtitle><stitle>TCI</stitle><date>2020</date><risdate>2020</risdate><volume>6</volume><spage>682</spage><epage>696</epage><pages>682-696</pages><issn>2573-0436</issn><eissn>2333-9403</eissn><coden>ITCIAJ</coden><abstract>This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/TCI.2020.2967148</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0001-9100-2906</orcidid><orcidid>https://orcid.org/0000-0002-9787-7460</orcidid></addata></record> |
fulltext | fulltext |
identifier | ISSN: 2573-0436 |
ispartof | IEEE transactions on computational imaging, 2020, Vol.6, p.682-696 |
issn | 2573-0436 2333-9403 |
language | eng |
recordid | cdi_ieee_primary_8961135 |
source | IEEE Electronic Library (IEL) Journals |
subjects | Artificial neural networks Cameras Chlorophyll convolutional neural network Correlation Datasets depth estimation Electrons Estimation Geometry Learning systems Light Light field Mapping Training Training data Videos Zero-shot learning |
title | Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network |
url | http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-28T21%3A21%3A07IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_ieee_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Zero-Shot%20Depth%20Estimation%20From%20Light%20Field%20Using%20A%20Convolutional%20Neural%20Network&rft.jtitle=IEEE%20transactions%20on%20computational%20imaging&rft.au=Peng,%20Jiayong&rft.date=2020&rft.volume=6&rft.spage=682&rft.epage=696&rft.pages=682-696&rft.issn=2573-0436&rft.eissn=2333-9403&rft.coden=ITCIAJ&rft_id=info:doi/10.1109/TCI.2020.2967148&rft_dat=%3Cproquest_ieee_%3E2357205225%3C/proquest_ieee_%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c287t-1b7eba7a0fb96a0f03aebe3a9eecd4b771774f0c4ec1e99436df0a1cfb9590183%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2357205225&rft_id=info:pmid/&rft_ieee_id=8961135&rfr_iscdi=true |