WSL, Ubuntu, Docker Installation, and Run a Docker — In this story, I record my steps by installing Docker in WSL of Windows. The full name of WSL is Windows Subsystem for Linux. While I can install docker in Windows directly, I want to use Linux commands such as curl. WSL is a good way to do it. Also…

Weight Standardization (WS) and Batch-Channel Normalization (BCN) are Proposed — Micro-Batch Training with Batch-Channel Normalization and Weight Standardization, WS, BCN, by Johns Hopkins University 2020 arXiv v2, Over 40 Citations (Sik-Ho Tsang @ Medium) Image Classification, Group Normalization (GN), Weight Normalization (WN), Weight Standardization (WS) standardizes the weights in convolutional layers.

TinyBERT, Outperforms MobileBERT, Much Smaller Than BERT — TinyBERT: Distilling BERT for Natural Language Understanding TinyBERT, by Huazhong University of Science and Technology, Huawei Noah’s Ark Lab, and Huawei Technologies Co., Ltd. 2020 EMNLP, Over 600 Citations (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Language Model, BERT, Transformer, Distillation A new two-stage learning framework is proposed for…

Fine-Grained Classification for Semi-Supervised Learning Realistic Evaluation — A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification, Su CVPR’21, by University of Massachusetts Amherst 2021 CVPR, over 10 Citations (Sik-Ho Tsang @ Medium) Semi-Supervised Learning, Image Classification, Fine-Grained Image Classification, Pseudo Label Class distribution can be highly unbalanced or even unknown, and the unlabeled data may contain novel…

CPVT, Conditional Position Encodings (CPE) Instead of Absolute Position Encodings — Conditional Positional Encodings for Vision Transformers CPVT, by Meituan Inc., and The University of Adelaide 2021 arXiv v2, Over 100 Citations (Sik-Ho Tsang @ Medium) Image Classification, Vision Transformer, ViT Unlike previous fixed or learnable positional encodings, which are predefined and independent of input tokens, Conditional Positional Encodings (CPE) is…

CWN, Re-parameterize Weight With Zero-Mean and Unit-Norm, Outperforms WN — Centered-Weight Normalization in Accelerating Training of Deep Neural Networks, CWN, by Beihang University, and The University of Sydney, 2017 ICCV, Over 50 Citations (Sik-Ho Tsang @ Medium) Image Classification, Batch Normalization, BN, Weight Normalization, WN This paper proposes to reparameterize the input weight of each neuron in deep neural networks…

OpenNMT Website: https://opennmt.net — The OpenNMT Neural Machine Translation Toolkit: 2020 Edition OpenNMT: Neural Machine Translation Toolkit, OpenNMT: Open-Source Toolkit for Neural Machine Translation, OpenNMT, by SYSTRAN, Ubiqus, and Harvard SEAS 2020 AMTA, 2018 AMTA, 2017 ACL, Over 20, 90, 1600 Citations Respectively (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Neural Machine Translation…

Curriculum Labeling (CL), By Restarting Model Parameters Before Each Self-Training Cycle, Outperforms Pseudo Labeling (PL) — Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning, Curriculum Labeling, by University of Virginia 2021 AAAI, Over 40 Citations (Sik-Ho Tsang @ Medium) Semi-Supervised Learning, Image Classification, Pseudo Label Pseudo Labeling (PL) works by applying pseudo-labels to samples in the unlabeled set for model training in a self-training cycle.

DINO, A Form of Self-Distillation with No Labels — Emerging Properties in Self-Supervised Vision Transformers DINO, by Facebook AI Research, Inria, and Sorbonne University 2021 ICCV, Over 400 Citations (Sik-Ho Tsang @ Medium) Self-Supervised Learning, Teacher Student, Image Classification, Vision Transformer, ViT Self-supervised method is applied onto Vision Transformer (ViT), which forms DINO, a form of self-Distillation with no…