Dynamic Self-Supervised Teacher-Student Network Learning

• Published in: IEEE transactions on pattern analysis and machine intelligence 2023-05, Vol.45 (5), p.1-19
• Main Authors: Ye, Fei, Bors, Adrian G.
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Computational modeling; Data models; Generative adversarial networks; Knowledge representation; Learning; Lifelong learning; Mixture models; Modules; Probabilistic models; representation learning; Self-supervised learning; Task analysis; teacher-student framework; Teachers; Training
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2022.3220928

Lifelong learning (LLL) represents the ability of an artificial intelligence system to learn successively a sequence of different databases. In this paper we introduce the Dynamic Self-Supervised Teacher-Student Network (D-TS), representing a more general LLL framework, where the Teacher is implemented as a dynamically expanding mixture model which automatically increases its capacity to deal with a growing number of tasks. We propose the Knowledge Discrepancy Score (KDS) criterion for measuring the relevance of the incoming information characterizing a new task when compared to the existing knowledge accumulated by the Teacher module from its previous training. The KDS ensures a light Teacher architecture while also enabling to reuse the learned knowledge whenever appropriate, accelerating the learning of given tasks. The Student module is implemented as a lightweight probabilistic generative model. We introduce a novel self-supervised learning for the Student that allows to capture cross-domain latent representations from the entire knowledge accumulated by the Teacher as well as from novel data. We perform several experiments which show that D-TS can achieve the state of the art results in LLL while requiring fewer parameters than other methods.