Review — Reducing Label Effort: Self-Supervised Meets Active Learning

SimSiam+AL vs Train-From-Scratch+AL

Reducing Label Effort: Self-Supervised Meets Active Learning,
SimSiam+AL, by Computer Vision Center (CVC), and Universitat Autònoma de Barcelona (UAB), 2021 ICCVW (Sik-Ho Tsang @ Medium)
Self-Supervised Learning, Active Learning, SimSiam

• The combination of active learning and self-training is fruitful to reduce label effort.

Outline

1. Self-Supervised Learning (SimSiam) + Active Learning (AL)
2. Experimental Results

1. Self-Supervised Learning (SimSiam) + Active Learning (AL)

Overview of active learning framework enhanced by self supervised pre-training

1.1. Framework

• The framework consists of 3 stages:

1. Self supervised model is trained on the entire dataset.
2. Given the frozen backbone and few labeled data, a linear classifier or an SVM is fine-tuned on top of the features in supervised way.
3. Running the model as inference on the unlabeled data and sort the samples from least to highest informative/representative via acquisition function. Finally the top samples are queried to oracle for labeling and added to labeled set.

• The above stages are repeated until the total labeling budget finishes.

1.2. Active Learning

• Several acquisition functions are considered, including Informativeness [10] and Representativeness based methods [42, 40].
• (The sampling methods want to tried here is not an issue because later authors find that the best one is random sampling.)

1.3. Self-Supervised Learning Using SimSiam

Self-Supervised Learning SimSiam Architecture

• SimSiam is used for self-supervised learning stage.
• (Any self-supervised learning frameworks can be used here. But please feel free to read SimSiam if interested.)

2. Experimental Results

2.1. Setup

• 1%, 2% and 10% of the entire dataset that are uniformly selected from all classes at random. For one of the datasets we also evaluate 0.1% and 0.2% budget sizes.
• At each cycle, training is started either from scratch or, in case of self-training, the pretrained self-supervised backbone is used. The model is trained in c cycles until the total budget is exhausted.

2.2. CIFAR-10 & CIFAR-100

AL Performance on CIFAR-10

In particular at the low budget regime, self-training drastically reduces the required labeling.

• Both types of methods (Self-training and Train-from-Scratch) achieve almost the full performance after labeling 50% of data that closes the gap between the self-supervised and supervised methods.
• From the active learning perspective, random sampling outperforms AL methods when the budget is less than 1%.

AL Performance on CIFAR-100

In the low budget regime, self-training significantly reduces the required labeling.

• While AL methods w/o self-training achieve comparable performance to self-trained counterparts as approaching to 50% labeled data, meaning that the impact of self supervised pre-training diminishes when the budget increases.
• Random sampling outperforms Active learning methods under low budget regardless of using self-training.

The trends are similar to the one in CIFAR-10.

Performance of AL methods with and without Self-training at 50% labeling

For the high labeling budget, the gap between the performances of AL and AL+ Self-training is diminished.

2.3. Tiny ImageNet

AL Performance on Tiny ImageNet

As in other datasets, Self-training drastically reduces the required labeling in low budget scheme.

• As the labeling increases to 50%, AL methods approach the performance of self-trained counterparts.
• However, unlike CIFAR datasets, AL methods require more than 50% labeling to close the performance gap they have from self-trained counterparts.
• Among the methods using self training, random sampling shows superior performance. However, increasing labeled data reduces performance gap from the AL methods.

In the active learning framework, under low-budget, SimSiam helps a lot. Under high-budget, the performance gap between train-from-scratch and SimSiam narrows.

Reference

[2021 ICCVW] [SimSiam+AL]
Reducing Label Effort: Self-Supervised Meets Active Learning

1.2. Unsupervised/Self-Supervised Learning

1993 … 2021 [MoCo v3] [SimSiam] [DINO] [Exemplar-v1, Exemplar-v2] [MICLe] [Barlow Twins] [MoCo-CXR] [W-MSE] [SimSiam+AL] 2022 [BEiT]

My Other Previous Paper Readings