Sequence Tagging Using Bidirectional LSTM and Pretrained Language Model

In this story, Semi-Supervised Sequence Tagging with Bidirectional Language Models, (TagLM), by Allen Institute for Artificial Intelligence, is briefly reviewed. It is found that in many NLP tasks, models are trained on relatively little labeled data. In this paper:

  • Sequence tagging aims to predict a linguistic tag for each word

Using Bidirectional LSTM Instead of Averaging in Word2Vec

In this story, context2vec: Learning Generic Context Embedding with Bidirectional LSTM, (context2vec), by Bar-Ilan University, is briefly reviewed. In this paper:

  • A bidirectional LSTM is proposed for efficiently learning a generic context embedding function from large corpora.

This is a paper in 2016 CoNLL with over 400 citations. …

Using global corpus statistics for learning word representation, outperforms CBOW in Word2Vec

In this story, GloVe: Global Vectors for Word Representation, (GloVe), by Stanford University, is briefly reviewed. In this paper:

  • Instead of local information, global corpus statistics is utilized for learning word representation.

This is a paper in 2014 EMNLP with over 24000 citations. (Sik-Ho Tsang @ Medium)


  1. The Statistics of…

GAN Using Transformer as Self-Attention in the form of Non-Local Neural Network

In this story, Self-Attention Generative Adversarial Networks, (SAGAN), by Rutgers University, and Google Brain, is reviewed. This is a paper from Ian Goodfellow (The inventor of GAN). In this paper:

  • SAGAN is proposed, which allows attention-driven, long-range dependency modeling for image generation tasks.
  • The discriminator can check that highly detailed…

Image Generation and Super Resolution Using Transformer

In this story, Image Transformer, by Google Brain, University of California, and Google AI, is briefly reviewed. In this paper:

  • Self-Attention Transformer is used for image generation.
  • By restricting the self-attention mechanism to attend to local neighborhoods, the size of images the model can process is significantly increased.

This is…

Using Transformer, Attention is Drawn, Long-Range Dependencies are Considered, Outperforms ByteNet, Deep-Att, GNMT, and ConvS2S

In this story, Attention Is All You Need, (Transformer), by Google Brain, Google Research, and University of Toronto, is reviewed. In this paper:

  • A new simple network architecture, the Transformer, based solely on attention mechanisms, is proposed, which dispensing with recurrence and convolutions entirely.

This is a paper in 2017…

Gated Convolutional Networks (GCNN) Using Gated Linear Unit (GLU)

In this story, Language Modeling with Gated Convolutional Networks, (GCNN/GLU), by Facebook AI Research, is briefly reviewed. In this paper:

  • A finite context approach through stacked convolutions is proposed, which can be more efficient since they allow parallelization over sequential tokens.
  • A novel simplified gating mechanism, Gated Linear Unit (GLU)

Using Fast-Forward Connections, Help Gradient Propagation

In this story, Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation, (Deep-ED & Deep-Att), by Baidu Research, is reviewed. In this paper:

  • A new type of linear connections, named fast-forward (F-F) connections, based on deep Long Short-Term Memory (LSTM), is introduced.
  • These F-F connections help propagating the gradients

With Attention, Show, Attend and Tell Outperforms Show and Tell

In this story, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, (Show, Attend and Tell), by Université de Montréal, University of Toronto, is briefly reviewed. This is a paper from Prof. Bengio’s group. In this paper:

  • An attention based model is introduced that automatically learns to describe…

Sik-Ho Tsang

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn:, My Paper Reading List:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store