Improving Language Understanding with Bidirectional Transformers

Summary & Methodology Analysis

Standard language models are often unidirectional, meaning they restrict architectures by preventing tokens from attending to both left and right context in all layers. This limitation hinders performance on sentence-level and token-level tasks. The paper introduces BERT to address this problem by leveraging a self-attention mechanism, which models both single text and text pairs end-to-end without architectural modification.

To pre-train the model, the authors use two unsupervised tasks. The first is Masked Language Model, which randomly masks 15 percent of tokens in a sequence and has the model predict them based on bidirectional context. The second is Next Sentence Prediction, where the model is trained on 50 percent IsNext and 50 percent NotNext sentence pairs to understand relationships between sentences. Once pre-trained, downstream tasks are handled by fine-tuning, which initializes with pre-trained parameters and adds a minimal task-specific output layer. The paper evaluates these models using several datasets including BooksCorpus, English Wikipedia, GLUE, SQuAD v1.1, SQuAD v2.0, SWAG, CoNLL-2003, TriviaQA, and Billion Word Benchmark, alongside baseline models like OpenAI GPT and ELMo. The results include an 80.5 percent GLUE score, 86.7 percent accuracy on MultiNLI, 93.2 Test F1 on SQuAD v1.1, and 83.1 Test F1 on SQuAD v2.0.

Despite its strong performance, the architecture has notable limitations. The use of the [MASK] token creates a mismatch between pre-training and fine-tuning since the token is not present during fine-tuning. Additionally, Masked LM converges more slowly than standard left-to-right language models because only a subset of tokens are predicted in each batch. The paper does not specify precise hardware requirements, dollar costs, or training latency figures beyond convergence behavior.