Review — ciFAIR: Do We Train on Test Data? Purging CIFAR of Near-Duplicates

A Duplicate-Free Variant of the CIFAR Test Set

Do We Train on Test Data? Purging CIFAR of Near-Duplicates,
ciFAIR, by Friedrich Schiller University Jena
2020 MDPI JoE (Sik-Ho Tsang @ Medium)
Image Classification, Image Dataset, CIFAR

  • It is found that respectively 3.3% and 10% of the CIFAR-10 and CIFAR-100 images from the test sets of these datasets have duplicates in the training set. So, the model is more likely to be learnt by memorization?
  • The “fair CIFAR” (ciFAIR) dataset is constructed, where all duplicates in the test sets are replaced with new images sampled from the same domain.

Outline

  1. CIFAR Duplicates
  2. Duplicate Statistics
  3. The Duplicate-Free ciFAIR Test Dataset
  4. Experimental Results

1. CIFAR Duplicates

Examples for different types of duplicates between the CIFAR-100 test and training set
  • The above figures shows the duplicates in the test set.
  • To find out the duplcates, a GUI is develop to show the difference.
GUI for duplicate annotation
  • Using the above GUI, the annotator can inspect the test image and its duplicate, their distance in the feature space, and a pixel-wise difference image.
  • Exact Duplicate: Almost all pixels in the two images are approximately identical.
  • Near-Duplicate: The content of the images is exactly the same, i.e., both originated from the same camera shot. However, different post-processing might have been applied to this original scene, e.g., color shifts, translations, scaling, etc.
  • Very Similar: The contents of the two images are different, but highly similar, so that the difference can only be spotted at the second glance.
  • Different: The pair does not belong to any other category.

2. Duplicate Statistics

Duplicates per type between test and training set (blue) and within the test set (orange)
  • It is worth noting that there are no exact duplicates in CIFAR-10 at all, as opposed to CIFAR-100.
  • There are 891 duplicates from the CIFAR-100 test set in the training set and 104 duplicates within the test set itself. In total, 10% of the test images have duplicates.
  • The situation is slightly better for CIFAR-10, where there are 286 duplicates in the training and 39 in the test set, amounting to 3.25% of the test set.
The classes with the most duplicates

3. The Duplicate-Free ciFAIR Test Dataset

GUI for replacement candidate selection
  • Each replacement candidate was inspected manually in a graphical user interface, which displayed the candidate and the three nearest neighbors in the feature space from the existing training and test sets.
  • The candidates are approved only for inclusion in the new test set when it is not considered as duplicates of any of the three nearest neighbors.
  • This modified datasets are as ciFAIR-10 and ciFAIR-100 (“fair CIFAR”).

4. Experimental Results

Classification error rate of various CNN architectures on the original CIFAR test sets and the modified ciFAIR test sets

There is a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set using different kinds of networks such as ResNet, WRN, DenseNet, ResNeXt, and PyramidNet.

It is surprising that there are duplicates in the training and testing sets of CIFAR datasets.

--

--

--

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn: https://www.linkedin.com/in/sh-tsang/, My Paper Reading List: https://bit.ly/33TDhxG

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Reading: QE-CNN — Quality Enhancement Convolutional Neural Network (Codec Filtering)

Introduction to RNN and LSTM(Part-4)

Getting started with AWS Deep Learning by training Convolutional Neural Network to predict Images…

Finding the Next Best Songs with Machine Learning

Celebrity Recognition : Building Scalable Pipeline

Deep learning: How much can image augmentation do?

Build a Self-Driving RC Car using Raspberry Pi and Machine Learning using Google Colab

Finding Shortest Path using Q-Learning Algorithm

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sik-Ho Tsang

Sik-Ho Tsang

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn: https://www.linkedin.com/in/sh-tsang/, My Paper Reading List: https://bit.ly/33TDhxG

More from Medium

Review — WSL: Exploring the Limits of Weakly Supervised Pretraining

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

ViT architecture

Transformer’s Training Details

A Review of Contrastive Learning for Unpaired Image to Image Translation