Review — Seq2Seq: Sequence to Sequence Learning with Neural Networks

Using LSTM for Encoder and Decoder for Machine Translation

Long Short-Term Memory (LSTM) (Figure from https://marssu.coderbridge.io/2020/11/21/sequence-to-sequence-model/)

Outline

1. The Use of LSTM

Vanilla RNN (Figure from https://marssu.coderbridge.io/2020/11/21/sequence-to-sequence-model/)

2. Sequence to Sequence (Seq2Seq) Model

2.1. Framework

Sequence to Sequence (Seq2Seq) Model. The Seq2Seq model reads an input sentence “ABC” and produces “WXYZ” as the output sentence.

2.2. Training

2.3. Testing

3. Experimental Results

3.1. WMT’14 English to French test set (ntst14)

The performance of the LSTM on WMT’14 English to French test set (ntst14)
Methods that use neural networks together with an SMT system on the WMT’14 English to French test set (ntst14)

3.2. Long Sentences

Left: Performance of our system as a function of sentence length, Right: the LSTM’s performance on sentences with progressively more rare words
A few examples of long translations produced by the LSTM alongside the ground truth translations

3.3. Model Analysis

The figure shows a 2-dimensional PCA projection of the LSTM hidden states that are obtained after processing the phrases in the figures

Reference

Natural Language Processing (NLP)

My Other Previous Paper Readings

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn: https://www.linkedin.com/in/sh-tsang/, My Paper Reading List: https://bit.ly/33TDhxG