12. Computational Performance

In deep learning, datasets and models are usually large, which involves heavy computation. Therefore, computational performance matters a lot. This chapter will focus on the major factors that affect computational performance: imperative programming, symbolic programming, asynchronous computing, automatic parallelism, and multi-GPU computation. By studying this chapter, you may further improve computational performance of those models implemented in the previous chapters, for example, by reducing training time without affecting accuracy.

• 12.1. Compilers and Interpreters
  • 12.1.1. Symbolic Programming
  • 12.1.2. Hybrid Programming
  • 12.1.3. Hybridizing the Sequential Class
  • 12.1.4. Summary
  • 12.1.5. Exercises
• 12.2. Asynchronous Computation
  • 12.2.1. Asynchrony via Backend
  • 12.2.2. Barriers and Blockers
  • 12.2.3. Improving Computation
  • 12.2.4. Summary
  • 12.2.5. Exercises
• 12.3. Automatic Parallelism
  • 12.3.1. Parallel Computation on GPUs
  • 12.3.2. Parallel Computation and Communication
  • 12.3.3. Summary
  • 12.3.4. Exercises
• 12.4. Hardware
  • 12.4.1. Computers
  • 12.4.2. Memory
  • 12.4.3. Storage
  • 12.4.4. CPUs
  • 12.4.5. GPUs and other Accelerators
  • 12.4.6. Networks and Buses
  • 12.4.7. More Latency Numbers
  • 12.4.8. Summary
  • 12.4.9. Exercises
• 12.5. Training on Multiple GPUs
  • 12.5.1. Splitting the Problem
  • 12.5.2. Data Parallelism
  • 12.5.3. A Toy Network
  • 12.5.4. Data Synchronization
  • 12.5.5. Distributing Data
  • 12.5.6. Training
  • 12.5.7. Summary
  • 12.5.8. Exercises
• 12.6. Concise Implementation for Multiple GPUs
  • 12.6.1. A Toy Network
  • 12.6.2. Network Initialization
  • 12.6.3. Training
  • 12.6.4. Summary
  • 12.6.5. Exercises
• 12.7. Parameter Servers
  • 12.7.1. Data-Parallel Training
  • 12.7.2. Ring Synchronization
  • 12.7.3. Multi-Machine Training
  • 12.7.4. Key–Value Stores
  • 12.7.5. Summary
  • 12.7.6. Exercises