4. Multilayer Perceptrons

In this chapter, we will introduce your first truly deep network. The simplest deep networks are called multilayer perceptrons, and they consist of multiple layers of neurons each fully connected to those in the layer below (from which they receive input) and those above (which they, in turn, influence). When we train high-capacity models we run the risk of overfitting. Thus, we will need to provide your first rigorous introduction to the notions of overfitting, underfitting, and model selection. To help you combat these problems, we will introduce regularization techniques such as weight decay and dropout. We will also discuss issues relating to numerical stability and parameter initialization that are key to successfully training deep networks. Throughout, we aim to give you a firm grasp not just of the concepts but also of the practice of using deep networks. At the end of this chapter, we apply what we have introduced so far to a real case: house price prediction. We punt matters relating to the computational performance, scalability, and efficiency of our models to subsequent chapters.

• 4.1. Multilayer Perceptrons
  • 4.1.1. Hidden Layers
  • 4.1.2. Activation Functions
  • 4.1.3. Summary
  • 4.1.4. Exercises
• 4.2. Implementation of Multilayer Perceptrons from Scratch
  • 4.2.1. Initializing Model Parameters
  • 4.2.2. Activation Function
  • 4.2.3. Model
  • 4.2.4. Loss Function
  • 4.2.5. Training
  • 4.2.6. Summary
  • 4.2.7. Exercises
• 4.3. Concise Implementation of Multilayer Perceptrons
  • 4.3.1. Model
  • 4.3.2. Summary
  • 4.3.3. Exercises
• 4.4. Model Selection, Underfitting, and Overfitting
  • 4.4.1. Training Error and Generalization Error
  • 4.4.2. Model Selection
  • 4.4.3. Underfitting or Overfitting?
  • 4.4.4. Polynomial Regression
  • 4.4.5. Summary
  • 4.4.6. Exercises
• 4.5. Weight Decay
  • 4.5.1. Norms and Weight Decay
  • 4.5.2. High-Dimensional Linear Regression
  • 4.5.3. Implementation from Scratch
  • 4.5.4. Concise Implementation
  • 4.5.5. Summary
  • 4.5.6. Exercises
• 4.6. Dropout
  • 4.6.1. Overfitting Revisited
  • 4.6.2. Robustness through Perturbations
  • 4.6.3. Dropout in Practice
  • 4.6.4. Implementation from Scratch
  • 4.6.5. Concise Implementation
  • 4.6.6. Summary
  • 4.6.7. Exercises
• 4.7. Forward Propagation, Backward Propagation, and Computational Graphs
  • 4.7.1. Forward Propagation
  • 4.7.2. Computational Graph of Forward Propagation
  • 4.7.3. Backpropagation
  • 4.7.4. Training Neural Networks
  • 4.7.5. Summary
  • 4.7.6. Exercises
• 4.8. Numerical Stability and Initialization
  • 4.8.1. Vanishing and Exploding Gradients
  • 4.8.2. Parameter Initialization
  • 4.8.3. Summary
  • 4.8.4. Exercises
• 4.9. Environment and Distribution Shift
  • 4.9.1. Types of Distribution Shift
  • 4.9.2. Examples of Distribution Shift
  • 4.9.3. Correction of Distribution Shift
  • 4.9.4. A Taxonomy of Learning Problems
  • 4.9.5. Fairness, Accountability, and Transparency in Machine Learning
  • 4.9.6. Summary
  • 4.9.7. Exercises
• 4.10. Predicting House Prices on Kaggle
  • 4.10.1. Downloading and Caching Datasets
  • 4.10.2. Kaggle
  • 4.10.3. Accessing and Reading the Dataset
  • 4.10.4. Data Preprocessing
  • 4.10.5. Training
  • 4.10.6. \(K\)-Fold Cross-Validation
  • 4.10.7. Model Selection
  • 4.10.8. Submitting Predictions on Kaggle
  • 4.10.9. Summary
  • 4.10.10. Exercises