Information Bottleneck and Aggregated Learning

We consider the problem of learning a neural network classifier. Under the information bottleneck (IB) principle, we associate with this classification problem a representation learning problem, which we call "IB learning". We show that IB learning is, in fact, equivalent to a special clas...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2023-12, Vol.45 (12), p.14807-14820
Main Authors: Soflaei, Masoumeh, Zhang, Richong, Guo, Hongyu, Al-Bashabsheh, Ali, Mao, Yongyi
Format: Article
Language:English
Subjects:
Aggregated learning
Classification
Codes
Distortion
Electronic mail
information bottleneck
Machine learning
Neural networks
Optimization
quantization
Rate-distortion
Representations
Vector quantization
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the problem of learning a neural network classifier. Under the information bottleneck (IB) principle, we associate with this classification problem a representation learning problem, which we call "IB learning". We show that IB learning is, in fact, equivalent to a special class of the quantization problem. The classical results in rate-distortion theory then suggest that IB learning can benefit from a "vector quantization" approach, namely, simultaneously learning the representations of multiple input objects. Such an approach assisted with some variational techniques, result in a novel learning framework, "Aggregated Learning", for classification with neural network models. In this framework, several objects are jointly classified by a single neural network. The effectiveness of this framework is verified through extensive experiments on standard image recognition and text classification tasks.
ISSN:0162-8828
2160-9292
1939-3539
DOI:10.1109/TPAMI.2023.3302150