Apply Fastformer Into Vision Transformer

Sik-Ho Tsang

·

Follow

3 min read

·

Oct 6, 2024

--

1

Listen

Share

Fast Vision Transformer via Additive Attention,
Fast Vision Transformer (FViT), by Shenzhen University, and Xidian University
2024 CAI (Sik-Ho Tsang @ Medium)

Image Classification
1989 … 2023 [Vision Permutator (ViP)] [ConvMixer] [CrossFormer++] [FastViT] [EfficientFormerV2] [MobileViTv2] [ConvNeXt V2] [SwiftFormer] [OpenCLIP] 2024 [FasterViT] [CAS-ViT] [TinySaver]
==== My Other Paper Readings Are Also Over Here ====

• Last week, Fastformer is mentioned.
• This time, the additive attention module in Fastformer is applied into Vision Transformer (ViT) to become Fast Vision Transformer (FViT).

Outline

1. Fast Vision Transformer (FViT)
2. Results

1. Fast Vision Transformer (FViT)

1.1. Vision Trasformer (ViT)

• The multi-head self-attention module in ViT:

Eq.(1) and (2) lead to quadratic complexity.

1.2. Fastformer

• (Please skip this part if you know Fastformer well.)

• Instead of full self-attention for Q, K, V matrices in Vision Transformer, first, an additive attention module is applied to exploit the query matrix into a global query vector q with the attention weight α:

• Then an element-wise product between the global query vector and each key vector integrates them into a global context-aware key matrix.
• Similarly, for computational efficiency, the i-th vector’s additive attention weight and global key vector are calculated:

• An element-wise product between the global key and value vector is then calculated.
• A linear transformation layer is then applied to each key-value interaction vector to learn its hidden representation, which is further added with query matrix to form the final output of the model.

Linear complexity is achieved.

2. Results

• Fastformer (b_32 and b_16 variations) is compared to the ViT with the variants of B/16 and B/32. B/16 and B/32 have a hidden dimension size of 768, MLP dimension of size 3072.
• The number of heads is 12 and the depth is set as 12.

In B/16 variant, ViT achieves 77% Top-1 accuracy, which is better than Fastformer-B/16 with 63%. But Fastformer-B/16 with 79M has less number of parameters than ViT-B/16 with 86M parameters.

• The Fastformer-B/16 with 45.2 GFLOPs has less computational complexity than ViT-B/16 with 49.3 GFLOPs.

In the B/32 variants, ViT-B/32 achieves 73% Top-1 accuracy, while Fastformer-B/32 65%. But Fastformer-B/32 with 81M parameters has fewer parameters than ViT-B/32 with 88M parameters.

• The computational cost of Fastformer-B/32’s 11.6 GFLOPs is less complex than ViT-B/32 with 12.6 GFLOPs.
• (Though authors mentioned that the proposed method with fewer FLOPs obtains comparable performance with the ViT, it seems that the accuracy is largely dropped.)