Brief Review — Fast AutoAugment

The First Place in AutoCV Competition of NeurIPS 2019 AutoDL Challenge

Sik-Ho Tsang
4 min readOct 21, 2022
Fast AutoAugment (FAA): Much Faster Than AutoAugment (AA)

Fast AutoAugment
FAA
, by UNIST, Kakao Brain, and Université de Montréal
2019 NeurIPS, Over 300 Citations (

@ Medium)
Data Augmentation, Image Classification

  • AutoAugment (AA) automatically searches for augmentation policies but requires thousands of GPU hours, as tabulated above.
  • Fast AutoAugment (FAA) is proposed, which finds effective augmentation policies via a more efficient search strategy based on density matching.

Outline

  1. Fast AutoAugment (FAA)
  2. Results

1. Fast AutoAugment (FAA)

1.1. Search Space

An example of augmented images via a sub-policy in the search space S
  • Let O be a set of augmentation on the input image space X. Each operation O has two parameters: the calling probability p and the magnitude λ. (Some operations (e.g. invert, flip) do not use the magnitude.)
  • Let S be the set of sub-policies where a sub-policy τS consists of N consecutive operations, where each operation is applied to an input image sequentially with the probability p as follows:
  • The output of sub-policy τ(x) can be described by a composition of operations as:
  • Where:
  • The above figure shows a specific example of augmented images by τ.
  • Thus, the final policy T is a collection of NT sub-policies and T(D) indicates a set of augmented images of dataset D transformed by every sub-policies τ T:

1.2. Search Strategy

An overall procedure of augmentation search by Fast AutoAugment algorithm
Fast AutoAugment Algorithm
  • For exploration, the proposed method splits the train dataset Dtrain into K-folds, which consists of two datasets D(k)M and D(k)A.
  • Then, model parameter θ is trained in parallel on each D(k)M. After training, the algorithm evaluates B bundles of augmentation policies on DA.
  • L(θ|T(D)) denotes the loss on augmented dataset T(D). In practice, the categorical cross-entropy loss is minimized.
  • These B candidate policies {T1, …, TB} are explored via Bayesian optimization. The Expected Improvement (EI) criterion [18] is employed for acquisition function to explore candidate policies B efficiently.
  • During the exploration process, the proposed algorithm does not train model parameter θ from scratch again. The top-N policies obtained from each K-fold are appended to an augmentation list T.
  • where R() is the accuracy.
  • Finally, the whole training set is trained using T*.

2. Results

Test set error rate (%) on CIFAR-10
Test set error rate (%) on CIFAR-100
Test set error rate (%) on SVHN
Validation set Top-1 / Top-5 error rate (%) on ImageNet

FAA significantly improves the performances of the baseline and Cutout for any network while achieving comparable performances to those of AA and PBA.

  • On reduced CIFAR-10, FAA only takes 3.5 GPU-hours for the policy search.
  • By considering the worst case, PyramidNet+ShakeDrop requires 780 GPU-hours which is even less than the computation time of AA (5000 GPU-hours).

The proposed FAA contributed to win the first place in AutoCV competition of NeurIPS 2019 AutoDL challenge.

Reference

[2019 NeurIPS] [FAA]
Fast AutoAugment

1.1. Image Classification

1989 2019 [FAA] … 2022 [ConvNeXt] [PVTv2] [ViT-G] [AS-MLP]

My Other Previous Paper Readings

--

--

Sik-Ho Tsang

PhD, Researcher. I share what I learn. :) Linktree: https://linktr.ee/shtsang for Twitter, LinkedIn, etc.