Reverse Engineering the Neural Tangent Kernel

The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles desi...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-08
Main Authors: Simon, James B, Sajant Anand, DeWeese, Michael R
Format: Article
Language:English
Subjects:
Computer architecture
Kernels
Neural networks
Questions
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Simon, James B
Sajant Anand
DeWeese, Michael R
description The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles design, into equivalent network architectures, which have superior efficiency, flexibility, and feature learning. To this end, we constructively prove that, with just an appropriate choice of activation function, any positive-semidefinite dot-product kernel can be realized as either the NNGP or neural tangent kernel of a fully-connected neural network with only one hidden layer. We verify our construction numerically and demonstrate its utility as a design tool for finite fully-connected networks in several experiments.
format article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2538876461</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2538876461</sourcerecordid><originalsourceid>FETCH-proquest_journals_25388764613</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mTQDUotSy0qTlVwzUvPzEtNLcrMS1coyUhV8EstLUrMUQhJzEtPzStR8E4tykvN4WFgTUvMKU7lhdLcDMpuriHOHroFRfmFpanFJfFZ-aVFeUCpeCNTYwsLczMTM0Nj4lQBAKr3Mdc</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2538876461</pqid></control><display><type>article</type><title>Reverse Engineering the Neural Tangent Kernel</title><source>Publicly Available Content (ProQuest)</source><creator>Simon, James B ; Sajant Anand ; DeWeese, Michael R</creator><creatorcontrib>Simon, James B ; Sajant Anand ; DeWeese, Michael R</creatorcontrib><description>The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles design, into equivalent network architectures, which have superior efficiency, flexibility, and feature learning. To this end, we constructively prove that, with just an appropriate choice of activation function, any positive-semidefinite dot-product kernel can be realized as either the NNGP or neural tangent kernel of a fully-connected neural network with only one hidden layer. We verify our construction numerically and demonstrate its utility as a design tool for finite fully-connected networks in several experiments.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Computer architecture ; Kernels ; Neural networks ; Questions</subject><ispartof>arXiv.org, 2022-08</ispartof><rights>2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2538876461?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25753,37012,44590</link.rule.ids></links><search><creatorcontrib>Simon, James B</creatorcontrib><creatorcontrib>Sajant Anand</creatorcontrib><creatorcontrib>DeWeese, Michael R</creatorcontrib><title>Reverse Engineering the Neural Tangent Kernel</title><title>arXiv.org</title><description>The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles design, into equivalent network architectures, which have superior efficiency, flexibility, and feature learning. To this end, we constructively prove that, with just an appropriate choice of activation function, any positive-semidefinite dot-product kernel can be realized as either the NNGP or neural tangent kernel of a fully-connected neural network with only one hidden layer. We verify our construction numerically and demonstrate its utility as a design tool for finite fully-connected networks in several experiments.</description><subject>Computer architecture</subject><subject>Kernels</subject><subject>Neural networks</subject><subject>Questions</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mTQDUotSy0qTlVwzUvPzEtNLcrMS1coyUhV8EstLUrMUQhJzEtPzStR8E4tykvN4WFgTUvMKU7lhdLcDMpuriHOHroFRfmFpanFJfFZ-aVFeUCpeCNTYwsLczMTM0Nj4lQBAKr3Mdc</recordid><startdate>20220813</startdate><enddate>20220813</enddate><creator>Simon, James B</creator><creator>Sajant Anand</creator><creator>DeWeese, Michael R</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20220813</creationdate><title>Reverse Engineering the Neural Tangent Kernel</title><author>Simon, James B ; Sajant Anand ; DeWeese, Michael R</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_25388764613</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Computer architecture</topic><topic>Kernels</topic><topic>Neural networks</topic><topic>Questions</topic><toplevel>online_resources</toplevel><creatorcontrib>Simon, James B</creatorcontrib><creatorcontrib>Sajant Anand</creatorcontrib><creatorcontrib>DeWeese, Michael R</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content (ProQuest)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Simon, James B</au><au>Sajant Anand</au><au>DeWeese, Michael R</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Reverse Engineering the Neural Tangent Kernel</atitle><jtitle>arXiv.org</jtitle><date>2022-08-13</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design, we propose the translation of high-performing kernels, which are better-understood and amenable to first-principles design, into equivalent network architectures, which have superior efficiency, flexibility, and feature learning. To this end, we constructively prove that, with just an appropriate choice of activation function, any positive-semidefinite dot-product kernel can be realized as either the NNGP or neural tangent kernel of a fully-connected neural network with only one hidden layer. We verify our construction numerically and demonstrate its utility as a design tool for finite fully-connected networks in several experiments.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2022-08
issn 2331-8422
language eng
recordid cdi_proquest_journals_2538876461
source Publicly Available Content (ProQuest)
subjects Computer architecture
Kernels
Neural networks
Questions
title Reverse Engineering the Neural Tangent Kernel
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-01T05%3A50%3A49IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Reverse%20Engineering%20the%20Neural%20Tangent%20Kernel&rft.jtitle=arXiv.org&rft.au=Simon,%20James%20B&rft.date=2022-08-13&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2538876461%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_25388764613%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2538876461&rft_id=info:pmid/&rfr_iscdi=true