18. Appendix: Mathematics for Deep Learning

Brent Werness (Amazon), Rachel Hu (Amazon), and authors of this book

One of the wonderful parts of modern deep learning is the fact that much of it can be understood and used without a full understanding of the mathematics below it. This is a sign that the field is maturing. Just as most software developers no longer need to worry about the theory of computable functions, neither should deep learning practitioners need to worry about the theoretical foundations of maximum likelihood learning.

But, we are not quite there yet.

In practice, you will sometimes need to understand how architectural choices influence gradient flow, or the implicit assumptions you make by training with a certain loss function. You might need to know what in the world entropy measures, and how it can help you understand exactly what bits-per-character means in your model. These all require deeper mathematical understanding.

This appendix aims to provide you the mathematical background you need to understand the core theory of modern deep learning, but it is not exhaustive. We will begin with examining linear algebra in greater depth. We develop a geometric understanding of all the common linear algebraic objects and operations that will enable us to visualize the effects of various transformations on our data. A key element is the development of the basics of eigen-decompositions.

We next develop the theory of differential calculus to the point that we can fully understand why the gradient is the direction of steepest descent, and why back-propagation takes the form it does. Integral calculus is then discussed to the degree needed to support our next topic, probability theory.

Problems encountered in practice frequently are not certain, and thus we need a language to speak about uncertain things. We review the theory of random variables and the most commonly encountered distributions so we may discuss models probabilistically. This provides the foundation for the naive Bayes classifier, a probabilistic classification technique.

Closely related to probability theory is the study of statistics. While statistics is far too large a field to do justice in a short section, we will introduce fundamental concepts that all machine learning practitioners should be aware of, in particular: evaluating and comparing estimators, conducting hypothesis tests, and constructing confidence intervals.

Last, we turn to the topic of information theory, which is the mathematical study of information storage and transmission. This provides the core language by which we may discuss quantitatively how much information a model holds on a domain of discourse.

Taken together, these form the core of the mathematical concepts needed to begin down the path towards a deep understanding of deep learning.

• 18.1. Geometry and Linear Algebraic Operations
  • 18.1.1. Geometry of Vectors
  • 18.1.2. Dot Products and Angles
  • 18.1.3. Hyperplanes
  • 18.1.4. Geometry of Linear Transformations
  • 18.1.5. Linear Dependence
  • 18.1.6. Rank
  • 18.1.7. Invertibility
  • 18.1.8. Determinant
  • 18.1.9. Tensors and Common Linear Algebra Operations
  • 18.1.10. Summary
  • 18.1.11. Exercises
• 18.2. Eigendecompositions
  • 18.2.1. Finding Eigenvalues
  • 18.2.2. Decomposing Matrices
  • 18.2.3. Operations on Eigendecompositions
  • 18.2.4. Eigendecompositions of Symmetric Matrices
  • 18.2.5. Gershgorin Circle Theorem
  • 18.2.6. A Useful Application: The Growth of Iterated Maps
  • 18.2.7. Conclusions
  • 18.2.8. Summary
  • 18.2.9. Exercises
• 18.3. Single Variable Calculus
  • 18.3.1. Differential Calculus
  • 18.3.2. Rules of Calculus
  • 18.3.3. Summary
  • 18.3.4. Exercises
• 18.4. Multivariable Calculus
  • 18.4.1. Higher-Dimensional Differentiation
  • 18.4.2. Geometry of Gradients and Gradient Descent
  • 18.4.3. A Note on Mathematical Optimization
  • 18.4.4. Multivariate Chain Rule
  • 18.4.5. The Backpropagation Algorithm
  • 18.4.6. Hessians
  • 18.4.7. A Little Matrix Calculus
  • 18.4.8. Summary
  • 18.4.9. Exercises
• 18.5. Integral Calculus
  • 18.5.1. Geometric Interpretation
  • 18.5.2. The Fundamental Theorem of Calculus
  • 18.5.3. Change of Variables
  • 18.5.4. A Comment on Sign Conventions
  • 18.5.5. Multiple Integrals
  • 18.5.6. Change of Variables in Multiple Integrals
  • 18.5.7. Summary
  • 18.5.8. Exercises
• 18.6. Random Variables
  • 18.6.1. Continuous Random Variables
  • 18.6.2. Summary
  • 18.6.3. Exercises
• 18.7. Maximum Likelihood
  • 18.7.1. The Maximum Likelihood Principle
  • 18.7.2. Numerical Optimization and the Negative Log-Likelihood
  • 18.7.3. Maximum Likelihood for Continuous Variables
  • 18.7.4. Summary
  • 18.7.5. Exercises
• 18.8. Distributions
  • 18.8.1. Bernoulli
  • 18.8.2. Discrete Uniform
  • 18.8.3. Continuous Uniform
  • 18.8.4. Binomial
  • 18.8.5. Poisson
  • 18.8.6. Gaussian
  • 18.8.7. Exponential Family
  • 18.8.8. Summary
  • 18.8.9. Exercises
• 18.9. Naive Bayes
  • 18.9.1. Optical Character Recognition
  • 18.9.2. The Probabilistic Model for Classification
  • 18.9.3. The Naive Bayes Classifier
  • 18.9.4. Training
  • 18.9.5. Summary
  • 18.9.6. Exercises
• 18.10. Statistics
  • 18.10.1. Evaluating and Comparing Estimators
  • 18.10.2. Conducting Hypothesis Tests
  • 18.10.3. Constructing Confidence Intervals
  • 18.10.4. Summary
  • 18.10.5. Exercises
• 18.11. Information Theory
  • 18.11.1. Information
  • 18.11.2. Entropy
  • 18.11.3. Mutual Information
  • 18.11.4. Kullback–Leibler Divergence
  • 18.11.5. Cross-Entropy
  • 18.11.6. Summary
  • 18.11.7. Exercises