WSL, Ubuntu, Docker Installation, and Run a Docker — In this story, I record my steps by installing Docker in WSL of Windows. The full name of WSL is Windows Subsystem for Linux. While I can install docker in Windows directly, I want to use Linux commands such as curl. WSL is a good way to do it. Also…

Weight Standardization (WS) and Batch-Channel Normalization (BCN) are Proposed — Micro-Batch Training with Batch-Channel Normalization and Weight Standardization, WS, BCN, by Johns Hopkins University 2020 arXiv v2, Over 40 Citations (Sik-Ho Tsang @ Medium) Image Classification, Group Normalization (GN), Weight Normalization (WN), Weight Standardization (WS) standardizes the weights in convolutional layers.

TinyBERT, Outperforms MobileBERT, Much Smaller Than BERT — TinyBERT: Distilling BERT for Natural Language Understanding TinyBERT, by Huazhong University of Science and Technology, Huawei Noah’s Ark Lab, and Huawei Technologies Co., Ltd. 2020 EMNLP, Over 600 Citations (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Language Model, BERT, Transformer, Distillation A new two-stage learning…

Fine-Grained Classification for Semi-Supervised Learning Realistic Evaluation — A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification, Su CVPR’21, by University of Massachusetts Amherst 2021 CVPR, over 10 Citations (Sik-Ho Tsang @ Medium) Semi-Supervised…

CPVT, Conditional Position Encodings (CPE) Instead of Absolute Position Encodings — Conditional Positional Encodings for Vision Transformers CPVT, by Meituan Inc., and The University of Adelaide 2021 arXiv v2, Over 100 Citations (Sik-Ho Tsang @ Medium) Image Classification, Vision Transformer, ViT Unlike previous fixed or learnable positional encodings, which are predefined and independent of input tokens, Conditional Positional Encodings (CPE) is…

CWN, Re-parameterize Weight With Zero-Mean and Unit-Norm, Outperforms WN — Centered-Weight Normalization in Accelerating Training of Deep Neural Networks, CWN, by Beihang University, and The University of Sydney, 2017 ICCV, Over 50 Citations (Sik-Ho Tsang @ Medium) Image Classification, Batch Normalization, BN, Weight Normalization, WN This paper proposes to reparameterize the input weight of each neuron in deep neural networks…

OpenNMT Website: https://opennmt.net — The OpenNMT Neural Machine Translation Toolkit: 2020 Edition OpenNMT: Neural Machine Translation Toolkit, OpenNMT: Open-Source Toolkit for Neural Machine Translation, OpenNMT, by SYSTRAN, Ubiqus, and Harvard SEAS 2020 AMTA, 2018 AMTA, 2017 ACL, Over 20, 90, 1600 Citations Respectively (Sik-Ho Tsang @ Medium) Natural Language Processing, NLP, Neural Machine Translation…

Curriculum Labeling (CL), By Restarting Model Parameters Before Each Self-Training Cycle, Outperforms Pseudo Labeling (PL) — Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning, Curriculum Labeling, by University of Virginia 2021 AAAI, Over 40 Citations (Sik-Ho Tsang @ Medium) Semi-Supervised Learning, Image Classification, Pseudo Label Pseudo Labeling (PL) works by applying pseudo-labels to samples in the unlabeled set for model training in a self-training cycle.

DINO, A Form of Self-Distillation with No Labels — Emerging Properties in Self-Supervised Vision Transformers DINO, by Facebook AI Research, Inria, and Sorbonne University 2021 ICCV, Over 400 Citations (Sik-Ho Tsang @ Medium) Self-Supervised Learning, Teacher Student, Image Classification, Vision Transformer, ViT Self-supervised method is applied onto Vision Transformer (ViT), which forms DINO, a form of self-Distillation with no…