15. Natural Language Processing: Applications

We have seen how to represent tokens in text sequences and train their representations in Section 14. Such pretrained text representations can be fed to various models for different downstream natural language processing tasks.

In fact, earlier chapters have already discussed some natural language processing applications without pretraining, just for explaining deep learning architectures. For instance, in Section 8, we have relied on RNNs to design language models to generate novella-like text. In Section 9 and Section 10, we have also designed models based on RNNs and attention mechanisms for machine translation.

However, this book does not intend to cover all such applications in a comprehensive manner. Instead, our focus is on how to apply (deep) representation learning of languages to addressing natural language processing problems. Given pretrained text representations, this chapter will explore two popular and representative downstream natural language processing tasks: sentiment analysis and natural language inference, which analyze single text and relationships of text pairs, respectively.

Fig. 15.1 Pretrained text representations can be fed to various deep learning architectures for different downstream natural language processing applications. This chapter focuses on how to design models for different downstream natural language processing applications.

As depicted in Fig. 15.1, this chapter focuses on describing the basic ideas of designing natural language processing models using different types of deep learning architectures, such as MLPs, CNNs, RNNs, and attention. Though it is possible to combine any pretrained text representations with any architecture for either application in Fig. 15.1, we select a few representative combinations. Specifically, we will explore popular architectures based on RNNs and CNNs for sentiment analysis. For natural language inference, we choose attention and MLPs to demonstrate how to analyze text pairs. In the end, we introduce how to fine-tune a pretrained BERT model for a wide range of natural language processing applications, such as on a sequence level (single text classification and text pair classification) and a token level (text tagging and question answering). As a concrete empirical case, we will fine-tune BERT for natural language inference.

As we have introduced in Section 14.8, BERT requires minimal architecture changes for a wide range of natural language processing applications. However, this benefit comes at the cost of fine-tuning a huge number of BERT parameters for the downstream applications. When space or time is limited, those crafted models based on MLPs, CNNs, RNNs, and attention are more feasible. In the following, we start by the sentiment analysis application and illustrate the model design based on RNNs and CNNs, respectively.

• 15.1. Sentiment Analysis and the Dataset
  • 15.1.1. Reading the Dataset
  • 15.1.2. Preprocessing the Dataset
  • 15.1.3. Creating Data Iterators
  • 15.1.4. Putting All Things Together
  • 15.1.5. Summary
  • 15.1.6. Exercises
• 15.2. Sentiment Analysis: Using Recurrent Neural Networks
  • 15.2.1. Representing Single Text with RNNs
  • 15.2.2. Loading Pretrained Word Vectors
  • 15.2.3. Training and Evaluating the Model
  • 15.2.4. Summary
  • 15.2.5. Exercises
• 15.3. Sentiment Analysis: Using Convolutional Neural Networks
  • 15.3.1. One-Dimensional Convolutions
  • 15.3.2. Max-Over-Time Pooling
  • 15.3.3. The textCNN Model
  • 15.3.4. Summary
  • 15.3.5. Exercises
• 15.4. Natural Language Inference and the Dataset
  • 15.4.1. Natural Language Inference
  • 15.4.2. The Stanford Natural Language Inference (SNLI) Dataset
  • 15.4.3. Summary
  • 15.4.4. Exercises
• 15.5. Natural Language Inference: Using Attention
  • 15.5.1. The Model
  • 15.5.2. Training and Evaluating the Model
  • 15.5.3. Summary
  • 15.5.4. Exercises
• 15.6. Fine-Tuning BERT for Sequence-Level and Token-Level Applications
  • 15.6.1. Single Text Classification
  • 15.6.2. Text Pair Classification or Regression
  • 15.6.3. Text Tagging
  • 15.6.4. Question Answering
  • 15.6.5. Summary
  • 15.6.6. Exercises
• 15.7. Natural Language Inference: Fine-Tuning BERT
  • 15.7.1. Loading Pretrained BERT
  • 15.7.2. The Dataset for Fine-Tuning BERT
  • 15.7.3. Fine-Tuning BERT
  • 15.7.4. Summary
  • 15.7.5. Exercises