Transformer-LS, Long-Range & Short-Range Attentions for Language Modelling & Image Classification — Long-Short Transformer: Efficient Transformers for Language and Vision, Transformer-LS, by University of Maryland, NVIDIA, Arizona State University, California Institute of Technology, 2021 NeurIPS, Over 40 Citations (Sik-Ho Tsang @ Medium) NLP, LM, Image Classification, Transformer, Vision Transformer, ViT

SD-UNet, Modified U-Net Using WS & GN — SD-UNet: Stripping down U-Net for Segmentation of Biomedical Images on Platforms with Low Computational Budgets, SD-UNet, by Beijing University of Technology, and Guilin University of Electronic Technology, 2020 MDPI J. Diagnostics, Over 60 Citations (Sik-Ho Tsang @ Medium) Medical Imaging, Medical Image Analysis, Image Segmentation, U-Net An extremely fast, small…

PersonLab, for Human Pose Estimation & Person Instance Segmentation — PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model, PersonLab, by Google Research, 2018 ECCV, Over 500 Citations (Sik-Ho Tsang @ Medium) Human Pose Estimation, Keypoint Detection, Instance Segmentation First, PersonLab learns to detect individual keypoints and group keypoints into person pose instances.

DeLighT, Parameter Reduction for Transformers — DeLighT: Deep and Light-weight Transformer, DeLighT, by University of Washington, Facebook AI Research, and Allen Institute for AI, 2021 ICLR, Over 60 Citations (Sik-Ho Tsang @ Medium) NLP, LM, NMT, Transformer Within each Transformer block, a deep and lightweight transformation is used using DeLighT block. Across blocks, block-wise scaling is…

ACMix, a MIXed model that enjoys the benefit of both self-Attention and Convolution (ACmix), while — On the Integration of Self-Attention and Convolution, ACMix, by Tsinghua University, Huawei Technologies Ltd., and Beijing Academy of Artificial Intelligence, 2022 CVPR, Over 40 Citations (Sik-Ho Tsang @ Medium) Image Classification, Convolution, Self-Attention ACmix, A mixed model that enjoys the benefit of both self-Attention and Convolution, is proposed.

DALS, Using Fully Convolutional Network (FCN) and Active Contour Model (ACM) — Deep Active Lesion Segmentation, DALS, by University of California, and Stanford University, 2019 MLMI, Over 40 Citations (Sik-Ho Tsang @ Medium) Medical Imaging, Medical Image Analysis, Image Segmentation Deep Active Lesion Segmentation (DALS), a fully automated segmentation framework is introduced. DALS leverages the powerful nonlinear feature extraction abilities of fully…

SAM, Simultaneously Minimizing Training Loss and Sharpness Loss, Using Loss Landscape — Sharpness-Aware Minimization for Efficiently Improving Generalization, Sharpness-Aware Minimization (SAM), by Google Research, and Blueshift, Alphabet, 2021 ICLR, Over 400 Citations (Sik-Ho Tsang @ Medium) Image Classification A novel, effective procedure for instead simultaneously minimizing training loss value and loss for sharpness.

RA-UNet, first Work To Use Attention Residual Mechanism for Tumor Segmentation from 3D Medical Volumetric Images. — RA-UNet: A Hybrid Deep Attention-Aware Network to Extract Liver and Tumor in CT Scans, RA-UNet, by Tianjin University, CSIRO Data61, Tianjin University of Traditional Chinese Medicine, and La Trobe…

VAN, Large Kernel Convolution by Decomposition — Visual Attention Network, VAN, by Tsinghua University, Nankai University University, and Fitten Tech, 2022 arXiv v5, Over 80 Citations (Sik-Ho Tsang @ Medium) Image Classification Applying self-attention in computer vision is challenging: (1) Treating images as 1D sequences neglects their 2D structures…