Deep Neural Networks Algorithms for Stochastic Control Problems on Finite Horizon: Convergence Analysis
Abstract
This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the value function by Monte Carlo regression. This is achieved in the dynamic programming recursion by performance or hybrid iteration and regress-now methods from numerical probabilities. We provide a theoretical justification of these algorithms. Consistency and rate of convergence for the control and value function estimates are analyzed and expressed in terms of the universal approximation error of the neural networks, and of the statistical error when estimating network function, leaving aside the optimization error. Numerical results on various applications are presented in a companion paper [Deep neural networks algorithms for stochastic control problems on finite horizon: Numerical applications, Methodol. Comput. Appl. Probab., to appear] and illustrate the performance of the proposed algorithms.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
K. Asadi and M. Littman, An alternative Softmax operator for reinforcement learning, in Proceedings of the 34th International Conference on Machine Learning, Vol. 70, 2017, pp. 243--252.
2.
F. Bach, Breaking the curse of dimensionality with convex neural networks, J. Mach. Learn. Res., 18 (2017), pp. 1--53.
3.
A. Bachouch, C. Huré, N. Langrené, and H. Pham, Deep neural networks algorithms for stochastic control problems on finite horizon: Numerical applications, Methodol. Comput. Appl. Probab., to appear.
4.
C. Beck, A. Jentzen, and B. Kuckuck, Full Error Analysis for the Training of Deep Neural Networks, arXiv:1910.00121v2, 2020.
5.
B. Bercu and J. Fort, Generic Stochastic Gradient Methods, in Wiley Encyclopedia of Operations Research and Management Science, Wiley, New York, 2011, pp. 1--8.
6.
D. P. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
7.
G. Cybenko, Approximations by superpositions of sigmoidal functions, Math. Control Signals Systems, 2 (1989), pp. 303--314.
8.
W. E, J. Han, and A. Jentzen, Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations, Commun. Math. Stat., 5 (2017), pp. 349--380.
9.
A. Géron, Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd ed., O'Reilly Media, Newton, MA, 2019.
10.
L. Györfi, M. Kohler, A. Krzyzak, and H. Walk, A Distribution-Free Theory of Nonparametric Regression, Springer Ser. Statist., Springer, New York, 2002.
11.
J. Han and W. E, Deep learning approximation for stochastic control problems, in Proceedings of NIPS Deep Reinforcement Learning Workshop, 2016.
12.
J. Han and J. Long, Convergence of the deep BSDE method for coupled FBSDEs, Probab. Uncertainty Quant. Risk, 5 (2020), pp. 1--33.
13.
P. Henry-Labordère, Deep Primal-Dual Algorithm for BSDEs: Applications of Machine Learning to CVA and IM, SSRN:3071506, 2017.
14.
K. Hornick, Approximation capabilities of multilayer feedforward networks, Neural Networks, 4 (1991), pp. 251--257.
15.
A. Jentzen, B. Kuckuck, A. Neufeld, and P. von Wurstemberger, Strong Error Analysis for Stochastic Gradient Descent Optimization Algorithms, arXiv:1801.09324v1, 2018.
16.
M. Kohler, Nonparametric regression with additional measurement errors in the dependent variable, J. Statist. Plann. Inference, 136 (2006), pp. 3339--3361.
17.
M. Kohler, A. Krzyżak, and N. Todorovic, Pricing of high-dimensional American options by neural networks, Math. Finance, 20 (2010), pp. 383--410.
18.
A. N. Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables, Math. Appl. (Soviet Ser.), 25 (1991).
19.
S. Kou, X. Peng, and X. Xu, EM Algorithm and Stochastic Control in Economics, SSRN:2865124, 2016.
20.
Y. Li, Deep Reinforcement Learning: An Overview, arXiv:1701.07274v3, 2017.
21.
F. A. Longstaff and E. S. Schwartz, Valuing American options by simulation: A simple least-squares approach, Rev. Financial Stud., 14 (2001), pp. 113--147.
22.
V. Mnih, K. Kavukcuoglu, D. Silver, and A. A. Rusu, Human-level control through deep reinforcement learning, Nature, 518 (2015), pp. 529--533.
23.
W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality, Wiley, New York, 2011.
24.
R. S. Sutton and A. G. Barto, Reinforcement Learning, MIT Press, Cambridge, UK, 1998.
Information & Authors
Information
Published In
Copyright
© 2021, Society for Industrial and Applied Mathematics.
History
Submitted: 3 February 2020
Accepted: 23 November 2020
Published online: 22 February 2021
Keywords
MSC codes
Authors
Funding Information
FiME
Funding Information
Norwegian Research Council : 250768/F20
Funding Information
Agence Nationale de la Recherche https://doi.org/10.13039/501100001665 : ANR-15-CE05-0024
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Beating a Benchmark: Dynamic Programming May Not Be the Right Numerical ApproachSIAM Journal on Financial Mathematics, Vol. 14, No. 2 | 10 May 2023
- Value-Gradient Based Formulation of Optimal Control Problem and Machine Learning AlgorithmSIAM Journal on Numerical Analysis, Vol. 61, No. 2 | 26 April 2023
- Finite Horizon Impulse control of Stochastic Functional Differential EquationsSIAM Journal on Control and Optimization, Vol. 61, No. 2 | 28 April 2023
- Statistical Machine Learning for Quantitative FinanceAnnual Review of Statistics and Its Application, Vol. 10, No. 1 | 10 Mar 2023
- Optimal Liquidation Through a Limit Order Book: A Neural Network and Simulation ApproachMethodology and Computing in Applied Probability, Vol. 25, No. 1 | 27 January 2023
- Automatic model training under restrictive time constraintsStatistics and Computing, Vol. 33, No. 1 | 13 December 2022
- Deep empirical risk minimization in finance: Looking into the futureMathematical Finance, Vol. 33, No. 1 | 17 November 2022
- SympOCnet: Solving Optimal Control Problems with Applications to High-Dimensional Multiagent Path Planning ProblemsSIAM Journal on Scientific Computing, Vol. 44, No. 6 | 17 November 2022
- Numerical Resolution of McKean-Vlasov FBSDEs Using Neural NetworksMethodology and Computing in Applied Probability, Vol. 24, No. 4 | 15 March 2022
- Deep Learning-Based Predictive Control of Injection Velocity in Injection Molding MachinesAdvances in Polymer Technology, Vol. 2022 | 15 Nov 2022
- An Algorithm to Construct Subsolutions of Convex Optimal Control ProblemsSIAM Journal on Control and Optimization, Vol. 60, No. 5 | 4 October 2022
- Solving Stochastic Optimal Control Problem via Stochastic Maximum Principle with Deep Learning MethodJournal of Scientific Computing, Vol. 93, No. 1 | 7 September 2022
- Empirical Analysis of Early Childhood Enlightenment Education Using Neural NetworkComputational Intelligence and Neuroscience, Vol. 2022 | 29 Aug 2022
- The Prediction of Enterprise Stock Change Trend by Deep Neural Network ModelComputational Intelligence and Neuroscience, Vol. 2022 | 2 Aug 2022
- Deep Learning for Constrained Utility MaximisationMethodology and Computing in Applied Probability, Vol. 24, No. 2 | 26 November 2021
- Full error analysis for the training of deep neural networksInfinite Dimensional Analysis, Quantum Probability and Related Topics, Vol. 25, No. 02 | 5 April 2022
- Polynomial affine approach to HARA utility maximization with applications to OrnsteinUhlenbeck 4/2 models.Applied Mathematics and Computation, Vol. 418 | 1 Apr 2022
- Approximation Error Analysis of Some Deep Backward Schemes for Nonlinear PDEsSIAM Journal on Scientific Computing, Vol. 44, No. 1 | 5 January 2022
- Error Estimates for a Tree Structure Algorithm Solving Finite Horizon Control ProblemsESAIM: Control, Optimisation and Calculus of Variations, Vol. 28 | 9 November 2022
- Deep combinatorial optimisation for optimal stopping time problems: application to swing options pricing.MathematicS In Action, Vol. 11, No. 1 | 27 September 2022
- Stochastic near-optimal control: additive, multiplicative, non-Markovian and applicationsThe European Physical Journal Special Topics, Vol. 230, No. 14-15 | 6 June 2021
- Three ways to solve partial differential equations with neural networks — A reviewGAMM-Mitteilungen, Vol. 44, No. 2 | 28 May 2021
- A Machine Learning Approach to Adaptive Robust Utility Maximization and HedgingSIAM Journal on Financial Mathematics, Vol. 12, No. 3 | 16 September 2021