Loading…
Improving generalization in deep neural network using knowledge transformation based on fisher criterion
Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying...
Saved in:
Published in: | The Journal of supercomputing 2023-12, Vol.79 (18), p.20899-20922 |
---|---|
Main Authors: | , , , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that this one cites Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
cited_by | cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223 |
---|---|
cites | cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223 |
container_end_page | 20922 |
container_issue | 18 |
container_start_page | 20899 |
container_title | The Journal of supercomputing |
container_volume | 79 |
creator | Morabbi, Sajedeh Soltanizadeh, Hadi Mozaffari, Saeed Fadaeieslam, Mohammad Javad |
description | Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs. |
doi_str_mv | 10.1007/s11227-023-05448-0 |
format | article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2879581280</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2879581280</sourcerecordid><originalsourceid>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</originalsourceid><addsrcrecordid>eNp9UMtOwzAQtBBIlMIPcLLEObB-JLaPqOJRqRIXOFtOsknTh1PslAq-HpcgceM0-5iZ1Q4h1wxuGYC6i4xxrjLgIoNcSp3BCZmwXKVWanlKJmA4ZDqX_JxcxLgCACmUmJDlfLsL_UfnW9qix-A23Zcbut7TztMacUc97tM0wXDow5ru45G79v1hg3WLdAjOx6YP21FVuog1TUXTxSUGWoVuwJA2l-SscZuIV784JW-PD6-z52zx8jSf3S-ySjAzZFg6rICjboQ2tQQstBAsF8xJKRU3DJQDWVQOSyxFpeqizPPSaCiZKQznYkpuRt_01vse42BX_T74dNJyrUyuGdeQWHxkVaGPMWBjd6HbuvBpGdhjonZM1KZE7U-i9igSoygmsm8x_Fn_o_oGcwt6uA</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2879581280</pqid></control><display><type>article</type><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><source>Springer Link</source><creator>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</creator><creatorcontrib>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</creatorcontrib><description>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</description><identifier>ISSN: 0920-8542</identifier><identifier>EISSN: 1573-0484</identifier><identifier>DOI: 10.1007/s11227-023-05448-0</identifier><language>eng</language><publisher>New York: Springer US</publisher><subject>Artificial neural networks ; Compilers ; Computer Science ; Interpreters ; Multivariate analysis ; Neural networks ; Processor Architectures ; Programming Languages ; Regularization ; Samples ; Transformations</subject><ispartof>The Journal of supercomputing, 2023-12, Vol.79 (18), p.20899-20922</ispartof><rights>The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</citedby><cites>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925</link.rule.ids></links><search><creatorcontrib>Morabbi, Sajedeh</creatorcontrib><creatorcontrib>Soltanizadeh, Hadi</creatorcontrib><creatorcontrib>Mozaffari, Saeed</creatorcontrib><creatorcontrib>Fadaeieslam, Mohammad Javad</creatorcontrib><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><title>The Journal of supercomputing</title><addtitle>J Supercomput</addtitle><description>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</description><subject>Artificial neural networks</subject><subject>Compilers</subject><subject>Computer Science</subject><subject>Interpreters</subject><subject>Multivariate analysis</subject><subject>Neural networks</subject><subject>Processor Architectures</subject><subject>Programming Languages</subject><subject>Regularization</subject><subject>Samples</subject><subject>Transformations</subject><issn>0920-8542</issn><issn>1573-0484</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNp9UMtOwzAQtBBIlMIPcLLEObB-JLaPqOJRqRIXOFtOsknTh1PslAq-HpcgceM0-5iZ1Q4h1wxuGYC6i4xxrjLgIoNcSp3BCZmwXKVWanlKJmA4ZDqX_JxcxLgCACmUmJDlfLsL_UfnW9qix-A23Zcbut7TztMacUc97tM0wXDow5ru45G79v1hg3WLdAjOx6YP21FVuog1TUXTxSUGWoVuwJA2l-SscZuIV784JW-PD6-z52zx8jSf3S-ySjAzZFg6rICjboQ2tQQstBAsF8xJKRU3DJQDWVQOSyxFpeqizPPSaCiZKQznYkpuRt_01vse42BX_T74dNJyrUyuGdeQWHxkVaGPMWBjd6HbuvBpGdhjonZM1KZE7U-i9igSoygmsm8x_Fn_o_oGcwt6uA</recordid><startdate>20231201</startdate><enddate>20231201</enddate><creator>Morabbi, Sajedeh</creator><creator>Soltanizadeh, Hadi</creator><creator>Mozaffari, Saeed</creator><creator>Fadaeieslam, Mohammad Javad</creator><general>Springer US</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20231201</creationdate><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><author>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Artificial neural networks</topic><topic>Compilers</topic><topic>Computer Science</topic><topic>Interpreters</topic><topic>Multivariate analysis</topic><topic>Neural networks</topic><topic>Processor Architectures</topic><topic>Programming Languages</topic><topic>Regularization</topic><topic>Samples</topic><topic>Transformations</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Morabbi, Sajedeh</creatorcontrib><creatorcontrib>Soltanizadeh, Hadi</creatorcontrib><creatorcontrib>Mozaffari, Saeed</creatorcontrib><creatorcontrib>Fadaeieslam, Mohammad Javad</creatorcontrib><collection>CrossRef</collection><jtitle>The Journal of supercomputing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Morabbi, Sajedeh</au><au>Soltanizadeh, Hadi</au><au>Mozaffari, Saeed</au><au>Fadaeieslam, Mohammad Javad</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</atitle><jtitle>The Journal of supercomputing</jtitle><stitle>J Supercomput</stitle><date>2023-12-01</date><risdate>2023</risdate><volume>79</volume><issue>18</issue><spage>20899</spage><epage>20922</epage><pages>20899-20922</pages><issn>0920-8542</issn><eissn>1573-0484</eissn><abstract>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</abstract><cop>New York</cop><pub>Springer US</pub><doi>10.1007/s11227-023-05448-0</doi><tpages>24</tpages></addata></record> |
fulltext | fulltext |
identifier | ISSN: 0920-8542 |
ispartof | The Journal of supercomputing, 2023-12, Vol.79 (18), p.20899-20922 |
issn | 0920-8542 1573-0484 |
language | eng |
recordid | cdi_proquest_journals_2879581280 |
source | Springer Link |
subjects | Artificial neural networks Compilers Computer Science Interpreters Multivariate analysis Neural networks Processor Architectures Programming Languages Regularization Samples Transformations |
title | Improving generalization in deep neural network using knowledge transformation based on fisher criterion |
url | http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-02T07%3A53%3A13IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Improving%20generalization%20in%20deep%20neural%20network%20using%20knowledge%20transformation%20based%20on%20fisher%20criterion&rft.jtitle=The%20Journal%20of%20supercomputing&rft.au=Morabbi,%20Sajedeh&rft.date=2023-12-01&rft.volume=79&rft.issue=18&rft.spage=20899&rft.epage=20922&rft.pages=20899-20922&rft.issn=0920-8542&rft.eissn=1573-0484&rft_id=info:doi/10.1007/s11227-023-05448-0&rft_dat=%3Cproquest_cross%3E2879581280%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2879581280&rft_id=info:pmid/&rfr_iscdi=true |