Review — Reducing Label Effort: Self-Supervised Meets Active Learning
Reducing Label Effort: Self-Supervised Meets Active Learning,
SimSiam+AL, by Computer Vision Center (CVC), and Universitat Autònoma de Barcelona (UAB), 2021 ICCVW (Sik-Ho Tsang @ Medium)
Self-Supervised Learning, Active Learning, SimSiam
- The combination of active learning and self-training is fruitful to reduce label effort.
1. Self-Supervised Learning (SimSiam) + Active Learning (AL)
- The framework consists of 3 stages:
- Self supervised model is trained on the entire dataset.
- Given the frozen backbone and few labeled data, a linear classifier or an SVM is fine-tuned on top of the features in supervised way.
- Running the model as inference on the unlabeled data and sort the samples from least to highest informative/representative via acquisition function. Finally the top samples are queried to oracle for labeling and added to labeled set.
- The above stages are repeated until the total labeling budget finishes.
1.2. Active Learning
- Several acquisition functions are considered, including Informativeness  and Representativeness based methods [42, 40].
- (The sampling methods want to tried here is not an issue because later authors find that the best one is random sampling.)
1.3. Self-Supervised Learning Using SimSiam
2. Experimental Results
- 1%, 2% and 10% of the entire dataset that are uniformly selected from all classes at random. For one of the datasets we also evaluate 0.1% and 0.2% budget sizes.
- At each cycle, training is started either from scratch or, in case of self-training, the pretrained self-supervised backbone is used. The model is trained in c cycles until the total budget is exhausted.
2.2. CIFAR-10 & CIFAR-100
In particular at the low budget regime, self-training drastically reduces the required labeling.
- Both types of methods (Self-training and Train-from-Scratch) achieve almost the full performance after labeling 50% of data that closes the gap between the self-supervised and supervised methods.
- From the active learning perspective, random sampling outperforms AL methods when the budget is less than 1%.
In the low budget regime, self-training significantly reduces the required labeling.
- While AL methods w/o self-training achieve comparable performance to self-trained counterparts as approaching to 50% labeled data, meaning that the impact of self supervised pre-training diminishes when the budget increases.
- Random sampling outperforms Active learning methods under low budget regardless of using self-training.
The trends are similar to the one in CIFAR-10.
For the high labeling budget, the gap between the performances of AL and AL+ Self-training is diminished.
2.3. Tiny ImageNet
As in other datasets, Self-training drastically reduces the required labeling in low budget scheme.
- As the labeling increases to 50%, AL methods approach the performance of self-trained counterparts.
- However, unlike CIFAR datasets, AL methods require more than 50% labeling to close the performance gap they have from self-trained counterparts.
- Among the methods using self training, random sampling shows superior performance. However, increasing labeled data reduces performance gap from the AL methods.
[2021 ICCVW] [SimSiam+AL]
Reducing Label Effort: Self-Supervised Meets Active Learning