Simple Distillation Baselines for Improving Small Self-supervised Models

While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation bas...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2021-06
Main Authors: Gu, Jindong, Liu, Wei, Tian, Yonglong
Format: Article
Language:English
Subjects:
Distillation
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation baseline, which establishes a new state-of-the-art, and an online-distillation baseline, which achieves similar performance with minimal computational overhead. We hope these baselines will provide useful experience for relevant future research. Code is available at: https://github.com/JindongGu/SimDis/
ISSN:2331-8422