On Fisher Information Matrix for Simple Neural Networks with Softplus Activation

Fisher information of simple neural networks with the softplus activation function is argued. We show that, under certain conditions, FIMs of simple models have the similar interesting spectral structure as the one shown by Takeishi et al (2021) for networks with ReLU. This work helps us to understa...

Full description

Saved in:
Bibliographic Details
Main Authors: Iida, Masazumi, Takeishi, Yoshinari, Takeuchi, Jun'ichi
Format: Conference Proceeding
Language:English
Subjects:
Fisher information
Information theory
Matrix decomposition
neural network
Neural networks
ReLU
softplus
spectral decomposition
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Fisher information of simple neural networks with the softplus activation function is argued. We show that, under certain conditions, FIMs of simple models have the similar interesting spectral structure as the one shown by Takeishi et al (2021) for networks with ReLU. This work helps us to understand why the FIM has such the structure.
ISSN:2157-8117
DOI:10.1109/ISIT50566.2022.9834516