Review: Semi-Supervised Sequence Tagging with Bidirectional Language Models (TagLM)

Sequence Tagging Using Bidirectional LSTM and Pretrained Language Model

BIO tag representing the Beginning, Inner, and Outside of entities (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)

Outline

1. TagLM Overview

TagLM Overview

2. TagLM: Network Architecture

Overview of TagLM

2.1. Token Representation (Bottom-Left)

2.2. Sequence Representation (Top-Left)

CRF (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)
CRF (Image from https://medium.com/mosaix/deep-text-representation-for-sequence-labeling-2f2e605ed9d)

2.3. Bidirectional LM (Right)

2.4. Combining LM with Sequence Model (Middle)

3. Experimental Results

3.1. SOTA Comparison Without Additional Data

Test set F1 comparison on CoNLL 2003 NER task, using only CoNLL 2003 data and unlabeled text
Test set F1 comparison on CoNLL 2000 Chunking task using only CoNLL 2000 data and unlabeled text

3.2. SOTA Comparison With Additional Data

Improvements in test set F1 in CoNLL 2003 NER when including additional labeled data or task specific gazetteers
Improvements in test set F1 in CoNLL 2000 Chunking when including additional labeled data

Reference

Natural Language Processing (NLP)

My Other Previous Paper Readings

PhD, Researcher. I share what I've learnt and done. :) My LinkedIn: https://www.linkedin.com/in/sh-tsang/, My Paper Reading List: https://bit.ly/33TDhxG