The Importance of the Current Input in Sequence Modeling

The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some ca...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2021-12
Main Authors: Oliva, Christian, Lago-Fernández, Luis F
Format: Article
Language:English
Subjects:
Model accuracy
Modelling
Natural language processing
Neural networks
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Oliva, Christian
Lago-Fernández, Luis F
description The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2613014048</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2613014048</sourcerecordid><originalsourceid>FETCH-proquest_journals_26130140483</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwCMlIVfDMLcgvKknMS05VyE9TKAGKOJcWFaXmlSh45hWUlihk5ikEpxaWpoIU-OanpOZk5qXzMLCmJeYUp_JCaW4GZTfXEGcP3YKifKDS4pL4rPzSojygVLyRmaGxgaGJgYmFMXGqAP5TNXI</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2613014048</pqid></control><display><type>article</type><title>The Importance of the Current Input in Sequence Modeling</title><source>Publicly Available Content Database</source><creator>Oliva, Christian ; Lago-Fernández, Luis F</creator><creatorcontrib>Oliva, Christian ; Lago-Fernández, Luis F</creatorcontrib><description>The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Model accuracy ; Modelling ; Natural language processing ; Neural networks</subject><ispartof>arXiv.org, 2021-12</ispartof><rights>2021. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2613014048?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25752,37011,44589</link.rule.ids></links><search><creatorcontrib>Oliva, Christian</creatorcontrib><creatorcontrib>Lago-Fernández, Luis F</creatorcontrib><title>The Importance of the Current Input in Sequence Modeling</title><title>arXiv.org</title><description>The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.</description><subject>Model accuracy</subject><subject>Modelling</subject><subject>Natural language processing</subject><subject>Neural networks</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwCMlIVfDMLcgvKknMS05VyE9TKAGKOJcWFaXmlSh45hWUlihk5ikEpxaWpoIU-OanpOZk5qXzMLCmJeYUp_JCaW4GZTfXEGcP3YKifKDS4pL4rPzSojygVLyRmaGxgaGJgYmFMXGqAP5TNXI</recordid><startdate>20211222</startdate><enddate>20211222</enddate><creator>Oliva, Christian</creator><creator>Lago-Fernández, Luis F</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20211222</creationdate><title>The Importance of the Current Input in Sequence Modeling</title><author>Oliva, Christian ; Lago-Fernández, Luis F</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_26130140483</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Model accuracy</topic><topic>Modelling</topic><topic>Natural language processing</topic><topic>Neural networks</topic><toplevel>online_resources</toplevel><creatorcontrib>Oliva, Christian</creatorcontrib><creatorcontrib>Lago-Fernández, Luis F</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Oliva, Christian</au><au>Lago-Fernández, Luis F</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>The Importance of the Current Input in Sequence Modeling</atitle><jtitle>arXiv.org</jtitle><date>2021-12-22</date><risdate>2021</risdate><eissn>2331-8422</eissn><abstract>The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2021-12
issn 2331-8422
language eng
recordid cdi_proquest_journals_2613014048
source Publicly Available Content Database
subjects Model accuracy
Modelling
Natural language processing
Neural networks
title The Importance of the Current Input in Sequence Modeling
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-11T19%3A32%3A11IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=The%20Importance%20of%20the%20Current%20Input%20in%20Sequence%20Modeling&rft.jtitle=arXiv.org&rft.au=Oliva,%20Christian&rft.date=2021-12-22&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2613014048%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_26130140483%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2613014048&rft_id=info:pmid/&rfr_iscdi=true