Deep motion estimation for parallel inter-frame prediction in video compression

Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames. We propose to learn binary motion codes that are encoded based on an input video sequence. These codes are not limited to 2D translati...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2019-12
Main Authors: Nortje, André, Engelbrecht, Herman A, Kamper, Herman
Format: Article
Language:English
Subjects:
Algorithms
Binary codes
Codec
Decoding
Motion simulation
Neural networks
Occlusion
Optical flow (image analysis)
Translations
Video compression
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Nortje, André
Engelbrecht, Herman A
Kamper, Herman
description Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames. We propose to learn binary motion codes that are encoded based on an input video sequence. These codes are not limited to 2D translations, but can capture complex motion (warping, rotation and occlusion). Our motion codes are learned as part of a single neural network which also learns to compress and decode them. This approach supports parallel video frame decoding instead of the sequential motion estimation and compensation of flow-based methods. We also introduce 3D dynamic bit assignment to adapt to object displacements caused by motion, yielding additional bit savings. By replacing the optical flow-based block-motion algorithms found in an existing video codec with our learned inter-frame prediction model, our approach outperforms the standard H.264 and H.265 video codecs across at low bitrates.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2325124612</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2325124612</sourcerecordid><originalsourceid>FETCH-proquest_journals_23251246123</originalsourceid><addsrcrecordid>eNqNjc0KgkAUhYcgSMp3uNBa0Dtq7fuhXZv2MugVRuavO9rzN0gP0OocvvPB2YgMpayKc424E3mMU1mW2J6waWQmnleiANbP2jugOGur1jp6hqBYGUMGtJuJi5GVJQhMg-5XRzv46IE89N4mHGOCB7EdlYmU_3Ivjvfb6_IoAvv3kg66yS_s0tShxKbCuq1Q_md9AXKYPzw</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2325124612</pqid></control><display><type>article</type><title>Deep motion estimation for parallel inter-frame prediction in video compression</title><source>Publicly Available Content (ProQuest)</source><creator>Nortje, André ; Engelbrecht, Herman A ; Kamper, Herman</creator><creatorcontrib>Nortje, André ; Engelbrecht, Herman A ; Kamper, Herman</creatorcontrib><description>Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames. We propose to learn binary motion codes that are encoded based on an input video sequence. These codes are not limited to 2D translations, but can capture complex motion (warping, rotation and occlusion). Our motion codes are learned as part of a single neural network which also learns to compress and decode them. This approach supports parallel video frame decoding instead of the sequential motion estimation and compensation of flow-based methods. We also introduce 3D dynamic bit assignment to adapt to object displacements caused by motion, yielding additional bit savings. By replacing the optical flow-based block-motion algorithms found in an existing video codec with our learned inter-frame prediction model, our approach outperforms the standard H.264 and H.265 video codecs across at low bitrates.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Algorithms ; Binary codes ; Codec ; Decoding ; Motion simulation ; Neural networks ; Occlusion ; Optical flow (image analysis) ; Translations ; Video compression</subject><ispartof>arXiv.org, 2019-12</ispartof><rights>2019. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2325124612?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25751,37010,44588</link.rule.ids></links><search><creatorcontrib>Nortje, André</creatorcontrib><creatorcontrib>Engelbrecht, Herman A</creatorcontrib><creatorcontrib>Kamper, Herman</creatorcontrib><title>Deep motion estimation for parallel inter-frame prediction in video compression</title><title>arXiv.org</title><description>Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames. We propose to learn binary motion codes that are encoded based on an input video sequence. These codes are not limited to 2D translations, but can capture complex motion (warping, rotation and occlusion). Our motion codes are learned as part of a single neural network which also learns to compress and decode them. This approach supports parallel video frame decoding instead of the sequential motion estimation and compensation of flow-based methods. We also introduce 3D dynamic bit assignment to adapt to object displacements caused by motion, yielding additional bit savings. By replacing the optical flow-based block-motion algorithms found in an existing video codec with our learned inter-frame prediction model, our approach outperforms the standard H.264 and H.265 video codecs across at low bitrates.</description><subject>Algorithms</subject><subject>Binary codes</subject><subject>Codec</subject><subject>Decoding</subject><subject>Motion simulation</subject><subject>Neural networks</subject><subject>Occlusion</subject><subject>Optical flow (image analysis)</subject><subject>Translations</subject><subject>Video compression</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2019</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNqNjc0KgkAUhYcgSMp3uNBa0Dtq7fuhXZv2MugVRuavO9rzN0gP0OocvvPB2YgMpayKc424E3mMU1mW2J6waWQmnleiANbP2jugOGur1jp6hqBYGUMGtJuJi5GVJQhMg-5XRzv46IE89N4mHGOCB7EdlYmU_3Ivjvfb6_IoAvv3kg66yS_s0tShxKbCuq1Q_md9AXKYPzw</recordid><startdate>20191211</startdate><enddate>20191211</enddate><creator>Nortje, André</creator><creator>Engelbrecht, Herman A</creator><creator>Kamper, Herman</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20191211</creationdate><title>Deep motion estimation for parallel inter-frame prediction in video compression</title><author>Nortje, André ; Engelbrecht, Herman A ; Kamper, Herman</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_23251246123</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Algorithms</topic><topic>Binary codes</topic><topic>Codec</topic><topic>Decoding</topic><topic>Motion simulation</topic><topic>Neural networks</topic><topic>Occlusion</topic><topic>Optical flow (image analysis)</topic><topic>Translations</topic><topic>Video compression</topic><toplevel>online_resources</toplevel><creatorcontrib>Nortje, André</creatorcontrib><creatorcontrib>Engelbrecht, Herman A</creatorcontrib><creatorcontrib>Kamper, Herman</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>ProQuest Engineering Database</collection><collection>Publicly Available Content (ProQuest)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Nortje, André</au><au>Engelbrecht, Herman A</au><au>Kamper, Herman</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Deep motion estimation for parallel inter-frame prediction in video compression</atitle><jtitle>arXiv.org</jtitle><date>2019-12-11</date><risdate>2019</risdate><eissn>2331-8422</eissn><abstract>Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames. We propose to learn binary motion codes that are encoded based on an input video sequence. These codes are not limited to 2D translations, but can capture complex motion (warping, rotation and occlusion). Our motion codes are learned as part of a single neural network which also learns to compress and decode them. This approach supports parallel video frame decoding instead of the sequential motion estimation and compensation of flow-based methods. We also introduce 3D dynamic bit assignment to adapt to object displacements caused by motion, yielding additional bit savings. By replacing the optical flow-based block-motion algorithms found in an existing video codec with our learned inter-frame prediction model, our approach outperforms the standard H.264 and H.265 video codecs across at low bitrates.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2019-12
issn 2331-8422
language eng
recordid cdi_proquest_journals_2325124612
source Publicly Available Content (ProQuest)
subjects Algorithms
Binary codes
Codec
Decoding
Motion simulation
Neural networks
Occlusion
Optical flow (image analysis)
Translations
Video compression
title Deep motion estimation for parallel inter-frame prediction in video compression
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-14T15%3A10%3A06IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Deep%20motion%20estimation%20for%20parallel%20inter-frame%20prediction%20in%20video%20compression&rft.jtitle=arXiv.org&rft.au=Nortje,%20Andr%C3%A9&rft.date=2019-12-11&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2325124612%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_23251246123%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2325124612&rft_id=info:pmid/&rfr_iscdi=true