Neuro-Dynamic Programming

Table of Contents:

1. Introduction

   1. Cost-to-go Approximations in Dynamic Programming
   2. Approximation Architectures
   3. Simulation and Training
   4. Neuro-Dynamic Programming
   5. Notes and Sources
   
   

2. Dynamic Programming

   1. Introduction
   2. Stochastic Shortest Path Problems
   3. Discounted Problems
   4. Problem Formulation and Examples
   5. Notes and Sources
   
   

3. Neural Network Architectures and Training

   1. Architectures for Approximation
   2. Neural Network Training
   3. Notes and Sources
   
   

4. Stochastic Iterative Algorithms

   1. The Basic Model
   2. Convergence Based on a Smooth Potential Function
   3. Convergence under Contraction or Monotonicity Assumptions
   4. The ODE Approach
   5. Notes and Sources
   
   

5. Simulation Methods for a Lookup Table Representation

   1. Some Aspects of Monte Carlo Simulation
   2. Policy Evaluation by Monte Carlo Simulation
   3. Temporal Difference Methods
   4. Optimistic Policy Iteration
   5. Simulation-Based Value Iteration
   6. Q-Learning
   7. Notes and Sources
   
   

6. Approximate DP with Cost-to-Go Function Approximation

   1. Generic Issues - From Parameters to Policies
   2. Approximate Policy Iteration
   3. Approximate Policy Evaluation Using TD(lambda)
   4. Optimistic Policy Iteration
   5. Approximate Value Iteration
   6. Q-Learning and Advantage Updating
   7. Value Iteration with State Aggregation
   8. Euclidean Contractions and Optimal Stopping
   9. Value Iteration with Representative States
   10. Bellman Error Methods
   11. Continuous States and the Slope of the Cost-to-Go
   12. Approximate Linear Programming
   13. Overview
   14. Notes and Sources
   
   

7. Extensions

   1. Average Cost per Stage Problems
   2. Dynamic Games
   3. Parallel Computation Issues
   4. Notes and Sources
   
   

8. Case Studies

   1. Parking
   2. Football
   3. Tetris
   4. Combinatorial Optimization - Maintenance and Repair
   5. Dynamic Channel Allocation
   6. Backgammon
   7. Notes and Sources
   
   

9. Appendix A: Mathematical Review

10. Appendix B: On Probability Theory and Markov Chains

11. References

12. Index