Loading…

Improving generalization in deep neural network using knowledge transformation based on fisher criterion

Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of supercomputing 2023-12, Vol.79 (18), p.20899-20922
Main Authors: Morabbi, Sajedeh, Soltanizadeh, Hadi, Mozaffari, Saeed, Fadaeieslam, Mohammad Javad
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223
cites cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223
container_end_page 20922
container_issue 18
container_start_page 20899
container_title The Journal of supercomputing
container_volume 79
creator Morabbi, Sajedeh
Soltanizadeh, Hadi
Mozaffari, Saeed
Fadaeieslam, Mohammad Javad
description Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.
doi_str_mv 10.1007/s11227-023-05448-0
format article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2879581280</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2879581280</sourcerecordid><originalsourceid>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</originalsourceid><addsrcrecordid>eNp9UMtOwzAQtBBIlMIPcLLEObB-JLaPqOJRqRIXOFtOsknTh1PslAq-HpcgceM0-5iZ1Q4h1wxuGYC6i4xxrjLgIoNcSp3BCZmwXKVWanlKJmA4ZDqX_JxcxLgCACmUmJDlfLsL_UfnW9qix-A23Zcbut7TztMacUc97tM0wXDow5ru45G79v1hg3WLdAjOx6YP21FVuog1TUXTxSUGWoVuwJA2l-SscZuIV784JW-PD6-z52zx8jSf3S-ySjAzZFg6rICjboQ2tQQstBAsF8xJKRU3DJQDWVQOSyxFpeqizPPSaCiZKQznYkpuRt_01vse42BX_T74dNJyrUyuGdeQWHxkVaGPMWBjd6HbuvBpGdhjonZM1KZE7U-i9igSoygmsm8x_Fn_o_oGcwt6uA</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2879581280</pqid></control><display><type>article</type><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><source>Springer Link</source><creator>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</creator><creatorcontrib>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</creatorcontrib><description>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</description><identifier>ISSN: 0920-8542</identifier><identifier>EISSN: 1573-0484</identifier><identifier>DOI: 10.1007/s11227-023-05448-0</identifier><language>eng</language><publisher>New York: Springer US</publisher><subject>Artificial neural networks ; Compilers ; Computer Science ; Interpreters ; Multivariate analysis ; Neural networks ; Processor Architectures ; Programming Languages ; Regularization ; Samples ; Transformations</subject><ispartof>The Journal of supercomputing, 2023-12, Vol.79 (18), p.20899-20922</ispartof><rights>The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</citedby><cites>FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925</link.rule.ids></links><search><creatorcontrib>Morabbi, Sajedeh</creatorcontrib><creatorcontrib>Soltanizadeh, Hadi</creatorcontrib><creatorcontrib>Mozaffari, Saeed</creatorcontrib><creatorcontrib>Fadaeieslam, Mohammad Javad</creatorcontrib><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><title>The Journal of supercomputing</title><addtitle>J Supercomput</addtitle><description>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</description><subject>Artificial neural networks</subject><subject>Compilers</subject><subject>Computer Science</subject><subject>Interpreters</subject><subject>Multivariate analysis</subject><subject>Neural networks</subject><subject>Processor Architectures</subject><subject>Programming Languages</subject><subject>Regularization</subject><subject>Samples</subject><subject>Transformations</subject><issn>0920-8542</issn><issn>1573-0484</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNp9UMtOwzAQtBBIlMIPcLLEObB-JLaPqOJRqRIXOFtOsknTh1PslAq-HpcgceM0-5iZ1Q4h1wxuGYC6i4xxrjLgIoNcSp3BCZmwXKVWanlKJmA4ZDqX_JxcxLgCACmUmJDlfLsL_UfnW9qix-A23Zcbut7TztMacUc97tM0wXDow5ru45G79v1hg3WLdAjOx6YP21FVuog1TUXTxSUGWoVuwJA2l-SscZuIV784JW-PD6-z52zx8jSf3S-ySjAzZFg6rICjboQ2tQQstBAsF8xJKRU3DJQDWVQOSyxFpeqizPPSaCiZKQznYkpuRt_01vse42BX_T74dNJyrUyuGdeQWHxkVaGPMWBjd6HbuvBpGdhjonZM1KZE7U-i9igSoygmsm8x_Fn_o_oGcwt6uA</recordid><startdate>20231201</startdate><enddate>20231201</enddate><creator>Morabbi, Sajedeh</creator><creator>Soltanizadeh, Hadi</creator><creator>Mozaffari, Saeed</creator><creator>Fadaeieslam, Mohammad Javad</creator><general>Springer US</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20231201</creationdate><title>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</title><author>Morabbi, Sajedeh ; Soltanizadeh, Hadi ; Mozaffari, Saeed ; Fadaeieslam, Mohammad Javad</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Artificial neural networks</topic><topic>Compilers</topic><topic>Computer Science</topic><topic>Interpreters</topic><topic>Multivariate analysis</topic><topic>Neural networks</topic><topic>Processor Architectures</topic><topic>Programming Languages</topic><topic>Regularization</topic><topic>Samples</topic><topic>Transformations</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Morabbi, Sajedeh</creatorcontrib><creatorcontrib>Soltanizadeh, Hadi</creatorcontrib><creatorcontrib>Mozaffari, Saeed</creatorcontrib><creatorcontrib>Fadaeieslam, Mohammad Javad</creatorcontrib><collection>CrossRef</collection><jtitle>The Journal of supercomputing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Morabbi, Sajedeh</au><au>Soltanizadeh, Hadi</au><au>Mozaffari, Saeed</au><au>Fadaeieslam, Mohammad Javad</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Improving generalization in deep neural network using knowledge transformation based on fisher criterion</atitle><jtitle>The Journal of supercomputing</jtitle><stitle>J Supercomput</stitle><date>2023-12-01</date><risdate>2023</risdate><volume>79</volume><issue>18</issue><spage>20899</spage><epage>20922</epage><pages>20899-20922</pages><issn>0920-8542</issn><eissn>1573-0484</eissn><abstract>Most deep neural networks (DNNs) are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. Generalization of DNNs has been improved by applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on the variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Compared to state-of-the-art methods, the transformation of multivariate statistical knowledge yields competitive results. Experimental results show that the proposed method achieved impressive generalization performance on CIFAR-10, CIFAR-100, and Tiny ImageNet with accuracy of 91.96%, 97.52%, and 99.21% respectively. Furthermore, this method enables faster convergence during the initial epochs.</abstract><cop>New York</cop><pub>Springer US</pub><doi>10.1007/s11227-023-05448-0</doi><tpages>24</tpages></addata></record>
fulltext fulltext
identifier ISSN: 0920-8542
ispartof The Journal of supercomputing, 2023-12, Vol.79 (18), p.20899-20922
issn 0920-8542
1573-0484
language eng
recordid cdi_proquest_journals_2879581280
source Springer Link
subjects Artificial neural networks
Compilers
Computer Science
Interpreters
Multivariate analysis
Neural networks
Processor Architectures
Programming Languages
Regularization
Samples
Transformations
title Improving generalization in deep neural network using knowledge transformation based on fisher criterion
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-02T07%3A53%3A13IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Improving%20generalization%20in%20deep%20neural%20network%20using%20knowledge%20transformation%20based%20on%20fisher%20criterion&rft.jtitle=The%20Journal%20of%20supercomputing&rft.au=Morabbi,%20Sajedeh&rft.date=2023-12-01&rft.volume=79&rft.issue=18&rft.spage=20899&rft.epage=20922&rft.pages=20899-20922&rft.issn=0920-8542&rft.eissn=1573-0484&rft_id=info:doi/10.1007/s11227-023-05448-0&rft_dat=%3Cproquest_cross%3E2879581280%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c319t-ebaec02e8f389d40e68331531a444729107a046caebeb3c7d6b55b980b1969223%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2879581280&rft_id=info:pmid/&rfr_iscdi=true