Brief Review — Fast AutoAugment

The First Place in AutoCV Competition of NeurIPS 2019 AutoDL Challenge

Fast AutoAugment (FAA): Much Faster Than AutoAugment (AA)

Fast AutoAugment
FAA, by UNIST, Kakao Brain, and Université de Montréal
2019 NeurIPS, Over 300 Citations (Sik-Ho Tsang @ Medium)
Data Augmentation, Image Classification

• AutoAugment (AA) automatically searches for augmentation policies but requires thousands of GPU hours, as tabulated above.
• Fast AutoAugment (FAA) is proposed, which finds effective augmentation policies via a more efficient search strategy based on density matching.

Outline

1. Fast AutoAugment (FAA)
2. Results

1. Fast AutoAugment (FAA)

1.1. Search Space

An example of augmented images via a sub-policy in the search space S

• Let O be a set of augmentation on the input image space X. Each operation O has two parameters: the calling probability p and the magnitude λ. (Some operations (e.g. invert, flip) do not use the magnitude.)
• Let S be the set of sub-policies where a sub-policy τ ∈ S consists of N consecutive operations, where each operation is applied to an input image sequentially with the probability p as follows:

• The output of sub-policy τ(x) can be described by a composition of operations as:

• Where:

• The above figure shows a specific example of augmented images by τ.
• Thus, the final policy T is a collection of NT sub-policies and T(D) indicates a set of augmented images of dataset D transformed by every sub-policies τ ∈ T:

1.2. Search Strategy

An overall procedure of augmentation search by Fast AutoAugment algorithm

Fast AutoAugment Algorithm

• For exploration, the proposed method splits the train dataset Dtrain into K-folds, which consists of two datasets D(k)M and D(k)A.
• Then, model parameter θ is trained in parallel on each D(k)M. After training, the algorithm evaluates B bundles of augmentation policies on DA.
• L(θ|T(D)) denotes the loss on augmented dataset T(D). In practice, the categorical cross-entropy loss is minimized.
• These B candidate policies {T1, …, TB} are explored via Bayesian optimization. The Expected Improvement (EI) criterion [18] is employed for acquisition function to explore candidate policies B efficiently.

• During the exploration process, the proposed algorithm does not train model parameter θ from scratch again. The top-N policies obtained from each K-fold are appended to an augmentation list T.

• where R() is the accuracy.
• Finally, the whole training set is trained using T*.

2. Results

Test set error rate (%) on CIFAR-10

Test set error rate (%) on CIFAR-100

Test set error rate (%) on SVHN

Validation set Top-1 / Top-5 error rate (%) on ImageNet

FAA significantly improves the performances of the baseline and Cutout for any network while achieving comparable performances to those of AA and PBA.

• On reduced CIFAR-10, FAA only takes 3.5 GPU-hours for the policy search.
• By considering the worst case, PyramidNet+ShakeDrop requires 780 GPU-hours which is even less than the computation time of AA (5000 GPU-hours).

The proposed FAA contributed to win the first place in AutoCV competition of NeurIPS 2019 AutoDL challenge.

Reference

[2019 NeurIPS] [FAA]
Fast AutoAugment

1.1. Image Classification

1989 … 2019 [FAA] … 2022 [ConvNeXt] [PVTv2] [ViT-G] [AS-MLP]

My Other Previous Paper Readings