3. Linear Neural Networks

Before we get into the details of deep neural networks, we need to cover the basics of neural network training. In this chapter, we will cover the entire training process, including defining simple neural network architectures, handling data, specifying a loss function, and training the model. In order to make things easier to grasp, we begin with the simplest concepts. Fortunately, classic statistical learning techniques such as linear and softmax regression can be cast as linear neural networks. Starting from these classic algorithms, we will introduce you to the basics, providing the basis for more complex techniques in the rest of the book.

• 3.1. Linear Regression
  • 3.1.1. Basic Elements of Linear Regression
  • 3.1.2. Vectorization for Speed
  • 3.1.3. The Normal Distribution and Squared Loss
  • 3.1.4. From Linear Regression to Deep Networks
  • 3.1.5. Summary
  • 3.1.6. Exercises
• 3.2. Linear Regression Implementation from Scratch
  • 3.2.1. Generating the Dataset
  • 3.2.2. Reading the Dataset
  • 3.2.3. Initializing Model Parameters
  • 3.2.4. Defining the Model
  • 3.2.5. Defining the Loss Function
  • 3.2.6. Defining the Optimization Algorithm
  • 3.2.7. Training
  • 3.2.8. Summary
  • 3.2.9. Exercises
• 3.3. Concise Implementation of Linear Regression
  • 3.3.1. Generating the Dataset
  • 3.3.2. Reading the Dataset
  • 3.3.3. Defining the Model
  • 3.3.4. Initializing Model Parameters
  • 3.3.5. Defining the Loss Function
  • 3.3.6. Defining the Optimization Algorithm
  • 3.3.7. Training
  • 3.3.8. Summary
  • 3.3.9. Exercises
• 3.4. Softmax Regression
  • 3.4.1. Classification Problem
  • 3.4.2. Network Architecture
  • 3.4.3. Parameterization Cost of Fully-Connected Layers
  • 3.4.4. Softmax Operation
  • 3.4.5. Vectorization for Minibatches
  • 3.4.6. Loss Function
  • 3.4.7. Information Theory Basics
  • 3.4.8. Model Prediction and Evaluation
  • 3.4.9. Summary
  • 3.4.10. Exercises
• 3.5. The Image Classification Dataset
  • 3.5.1. Reading the Dataset
  • 3.5.2. Reading a Minibatch
  • 3.5.3. Putting All Things Together
  • 3.5.4. Summary
  • 3.5.5. Exercises
• 3.6. Implementation of Softmax Regression from Scratch
  • 3.6.1. Initializing Model Parameters
  • 3.6.2. Defining the Softmax Operation
  • 3.6.3. Defining the Model
  • 3.6.4. Defining the Loss Function
  • 3.6.5. Classification Accuracy
  • 3.6.6. Training
  • 3.6.7. Prediction
  • 3.6.8. Summary
  • 3.6.9. Exercises
• 3.7. Concise Implementation of Softmax Regression
  • 3.7.1. Initializing Model Parameters
  • 3.7.2. Softmax Implementation Revisited
  • 3.7.3. Optimization Algorithm
  • 3.7.4. Training
  • 3.7.5. Summary
  • 3.7.6. Exercises