SWA & Fast SWA, Improving Π-Model & Mean Teacher — There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average SWA & Fast SWA, by Cornell University 2019 ICLR, Over 100 Citations (Sik-Ho Tsang @ Medium) Semi-Supervised Learning, Image Classification Stochastic Weight Averaging (SWA) is proposed, which averages weight along the trajectory of SGD with a modified learning…

Pre-LN Transformer, Warm-Up Stage is Skipped — On Layer Normalization in the Transformer Architecture Pre-LN Transformer, by Microsoft Research Asia, University of Chinese Academy of Sciences, Peking University, Microsoft Research, and Nankai University 2020 ICML, Over 100 Citations (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Language Model, Machine Translation, Transformer, Layer Normalization, BERT

Video Frame Extrapolation, Generate Future Frames — Deep Multi-Scale Video Prediction Beyond Mean Square Error Mathieu ICLR’16, by New York University, and Facebook Artificial Intelligence Research 2016 ICLR, Over 1700 Citations (Sik-Ho Tsang @ Medium) Video Frame Extrapolation A CNN is proposed to generate future frames given an input sequence. To deal with the inherently blurry predictions…

Outperforms ViT, DeiT, PVT — Transformer in Transformer TNT, by ISCAS & UCAS, Huawei Technologies, and University of Macau 2021 NeurIPS, Over 200 Citations (Sik-Ho Tsang @ Medium) Image Classification, Transformer, ViT Transformer iN Transformer (TNT) is designed, where the local patches (e.g., 16×16) are regarded as “visual sentences” and present to further divide them…

List of Learning Based IQA/VQA, Camera Tampering/Blur/Soiling Detection Papers — I moved the IQA/VQA, Camera Tampering/Blur/Soiling Detection paper reading list from Summary: My Paper Reading Lists, Tutorials & Sharings to here, so that I can shorten the list for different topics. Though I’ve not updated for so long yet, I know that there are many state-of-the-art approaches appeared in these…

HaloNet, Localized Window for Self-Attention — Scaling Local Self-Attention for Parameter Efficient Visual Backbones HaloNet, by Google Research, and UC Berkeley 2021 CVPR, Over 80 Citations (Sik-Ho Tsang @ Medium) Image Classification, Self-Attention A new self-attention model family, HaloNets, is developed: A strided self-attention layer, a natural extension of strided convolutions, is developed.

Self-Supervised Learning Without the Use of Negative Sample Pairs, Large Batches, & Momentum Encoders — Exploring Simple Siamese Representation Learning SimSiam, by Facebook AI Research (FAIR) 2021 CVPR, Over 500 Citations (Sik-Ho Tsang @ Medium) Self-Supervised Learning, Unsupervised Learning, Representation Learning, Image Classification There are many different frameworks proposed for self-supervised learning, such as MoCo and SimCLR, yet with some constraints.

Pre-Norm Transformer, Layer Normalization First — Learning Deep Transformer Models for Machine Translation Pre-Norm Transformer, by Northeastern University, NiuTrans Co., Ltd., Kingsoft AI Lab, and University of Macau 2019 ACL, Over 200 Citations (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Machine Translation, Transformer Two novel techniques are proposed: A proper use of layer normalization is…

Convolutional vision Transformer (CvT) — CvT: Introducing Convolutions to Vision Transformers CvT, by McGill University, and Microsoft Cloud + AI 2021 ICCV, Over 200 Citations (Sik-Ho Tsang @ Medium) Image Classification, Transformer, ViT Convolutional vision Transformer (CvT) is proposed through two…