Sik-Ho Tsang

Dec 5, 2021

5 min read

Review: Semi-Supervised Sequence Tagging with Bidirectional Language Models (TagLM)

Sequence Tagging Using Bidirectional LSTM and Pretrained Language Model

BIO tag representing the Beginning, Inner, and Outside of entities (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)

Outline

1. TagLM Overview

TagLM Overview

The unsupervised pre-trained word embedding model (grey) is to boost the performance of the supervised recurrent language model (orange).

2. TagLM: Network Architecture

Overview of TagLM
CRF (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)
CRF (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)

3. Experimental Results

Test set F1 comparison on CoNLL 2003 NER task, using only CoNLL 2003 data and unlabeled text
Test set F1 comparison on CoNLL 2000 Chunking task using only CoNLL 2000 data and unlabeled text
Improvements in test set F1 in CoNLL 2003 NER when including additional labeled data or task specific gazetteers
Improvements in test set F1 in CoNLL 2000 Chunking when including additional labeled data

By using pretrained language model, performance is improved for sequence model.