Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Information bottleneck (IB) is a paradigm to extract information in one target random variable from another relevant random variable, which has aroused great interest due to its potential to explain deep neural networks in terms of information compression and prediction. Despite its great importance...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-08
Main Authors: Chen, Lingyi, Wu, Shitong, Ye, Wenhao, Wu, Huihui, Wu, Hao, Zhang, Wenyi, Bai, Bo, Sun, Yining
Format: Article
Language:English
Subjects:
Algorithms
Artificial neural networks
Concavity
Conditional probability
Lagrange multiplier
Machine learning
Optimization
Random variables
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Information bottleneck (IB) is a paradigm to extract information in one target random variable from another relevant random variable, which has aroused great interest due to its potential to explain deep neural networks in terms of information compression and prediction. Despite its great importance, finding the optimal bottleneck variable involves a difficult nonconvex optimization problem due to the nonconvexity of mutual information constraint. The Blahut-Arimoto algorithm and its variants provide an approach by considering its Lagrangian with fixed Lagrange multiplier. However, only the strictly concave IB curve can be fully obtained by the BA algorithm, which strongly limits its application in machine learning and related fields, as strict concavity cannot be guaranteed in those problems. To overcome the above difficulty, we derive an entropy regularized optimal transport (OT) model for IB problem from a posterior probability perspective. Correspondingly, we use the alternating optimization procedure and generalize the Sinkhorn algorithm to solve the above OT model. The effectiveness and efficiency of our approach are demonstrated via numerical experiments.
ISSN:2331-8422