3 days ago

Review: Semi-Supervised Sequence Tagging with Bidirectional Language Models (TagLM)

Sequence Tagging Using Bidirectional LSTM and Pretrained Language Model

**BIO tag representing the Beginning, Inner, and Outside of entities** (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)

In this story, Semi-Supervised Sequence Tagging with Bidirectional Language Models, (TagLM), by Allen Institute for Artificial Intelligence, is briefly reviewed. It is found that in many NLP tasks, models are trained on relatively little labeled data. In this paper:

Sequence tagging aims to predict a linguistic tag for each word…

Read more · 5 min read

4 days ago

Review — context2vec: Learning Generic Context Embedding with Bidirectional LSTM

Using Bidirectional LSTM Instead of Averaging in Word2Vec

**A 2D illustration of context2vec’s embedded space and similarity metrics. Triangles and circles denote sentential context embeddings and target word embeddings, respectively**

In this story, context2vec: Learning Generic Context Embedding with Bidirectional LSTM, (context2vec), by Bar-Ilan University, is briefly reviewed. In this paper:

A bidirectional LSTM is proposed for efficiently learning a generic context embedding function from large corpora.

This is a paper in 2016 CoNLL with over 400 citations. …

Review — GloVe: Global Vectors for Word Representation

Using global corpus statistics for learning word representation, outperforms CBOW in Word2Vec

(Image from https://unsplash.com/photos/bqzLehtF8XE)

In this story, GloVe: Global Vectors for Word Representation, (GloVe), by Stanford University, is briefly reviewed. In this paper:

Instead of local information, global corpus statistics is utilized for learning word representation.

This is a paper in 2014 EMNLP with over 24000 citations. (Sik-Ho Tsang @ Medium)

Outline

The Statistics of…

Review: Self-Attention Generative Adversarial Networks (SAGAN)

GAN Using Transformer as Self-Attention in the form of Non-Local Neural Network

**The proposed SAGAN generates images by leveraging complementary features in distant portions of the image rather than local regions of fixed shape to generate consistent objects/scenarios**

In this story, Self-Attention Generative Adversarial Networks, (SAGAN), by Rutgers University, and Google Brain, is reviewed. This is a paper from Ian Goodfellow (The inventor of GAN). In this paper:

SAGAN is proposed, which allows attention-driven, long-range dependency modeling for image generation tasks.
The discriminator can check that highly detailed…

Review: Image Transformer

Image Generation and Super Resolution Using Transformer

**Left: Super Resolution, Right: Image Generation** (Input / Image Transformer Results / Ground Truth)

In this story, Image Transformer, by Google Brain, University of California, and Google AI, is briefly reviewed. In this paper:

Self-Attention Transformer is used for image generation.
By restricting the self-attention mechanism to attend to local neighborhoods, the size of images the model can process is significantly increased.

This is…

Review — Attention Is All You Need (Transformer)

Using Transformer, Attention is Drawn, Long-Range Dependencies are Considered, Outperforms ByteNet, Deep-Att, GNMT, and ConvS2S

**Attention Is All You Need** (Figure from https://knowyourmeme.com/memes/attention-is-all-you-need)

In this story, Attention Is All You Need, (Transformer), by Google Brain, Google Research, and University of Toronto, is reviewed. In this paper:

A new simple network architecture, the Transformer, based solely on attention mechanisms, is proposed, which dispensing with recurrence and convolutions entirely.

This is a paper in 2017…

Review — Convolutional Sequence to Sequence Learning (ConvS2S)

ConvS2S as Convolutional Network, Outperforms GNMT

In this story, Convolutional Sequence to Sequence Learning, (ConvS2S), by Facebook AI Research, is briefly reviewed. In this story:

An architecture is proposed which is entirely based on convolutional neural networks (CNN).
Computations can be fully parallelized during training.
Gated linear units (GLU) eases gradient propagation and
Each decoder layer…

Read more · 5 min read

Nov 26

Review: Language Modeling with Gated Convolutional Networks (GCNN/GLU)

Gated Convolutional Networks (GCNN) Using Gated Linear Unit (GLU)

In this story, Language Modeling with Gated Convolutional Networks, (GCNN/GLU), by Facebook AI Research, is briefly reviewed. In this paper:

A finite context approach through stacked convolutions is proposed, which can be more efficient since they allow parallelization over sequential tokens.
A novel simplified gating mechanism, Gated Linear Unit (GLU)…

Read more · 5 min read

Published in Geek Culture

·Nov 23

Review — Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (Deep-ED & Deep-Att)

Using Fast-Forward Connections, Help Gradient Propagation

In this story, Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation, (Deep-ED & Deep-Att), by Baidu Research, is reviewed. In this paper:

A new type of linear connections, named fast-forward (F-F) connections, based on deep Long Short-Term Memory (LSTM), is introduced.
These F-F connections help propagating the gradients…

Review — Show, Attend and Tell: Neural Image Caption Generation

With Attention, Show, Attend and Tell Outperforms Show and Tell

**Show, Attend and Tell** (Figure from https://zhuanlan.zhihu.com/p/32333802)

In this story, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, (Show, Attend and Tell), by Université de Montréal, University of Toronto, is briefly reviewed. This is a paper from Prof. Bengio’s group. In this paper:

An attention based model is introduced that automatically learns to describe…

Read more · 5 min read