On Iteration Improvement for Averaged Expected Cost Control for One-Dimensional Ergodic Diffusions
Abstract
An ergodic Bellman's (Hamilton--Jacobi--Bellman) equation is proved for a uniformly ergodic one-dimensional controlled diffusion with variable diffusion and drift coefficients both depending on control; convergence of the values provided by Howard's reward improvement algorithm to the value which is a component of the unique solution of Bellman's equation is established.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
S. V. Anulova, H. Mai, and A. Yu. Veretennikov, On averaged expected cost control as reliability for 1D ergodic diffusions, Reliability: Theory & Applications (RT&A), 12, (2017), pp. 31--38.
2.
A. Arapostathis, V. S. Borkar, E. Ferna͂ndes-Gaucherand, M. K. Ghosh, and S. I. Markus, Discrete-time controlled Markov processes with average cost criterion: A survey, SIAM J. Control Optim., 31 (1993), pp. 282--344.
3.
A. Arapostathis, On the policy iteration algorithm for nondegenerate controlled diffusions under the ergodic criterion, In Optimization, Control, and Applications of Stochastic Systems, Systems Control Found. Appl., Springer, New York, 2012, pp. 1--12.
4.
A. Arapostathis and V. S. Borkar, A relative value iteration algorithm for non-degenerate controlled diffusions, SIAM J. Control Optim., 50 (2012), pp. 1886--1902.
5.
A. Arapostathis, V. S. Borkar, and M. K. Ghosh, Ergodic control of diffusion processes, Encyclopedia of Mathem. and its Appl. 143, Cambridge University Press, Cambridge, 2012.
6.
V. S. Borkar, Optimal Control of Diffusion Processes, John Wiley & Sons, New York, 1989.
7.
L. D. Brown and R. Purves, Measurable selections of extrema, Ann. Stat., 1 (1973), pp. 902--912.
8.
O. L. do Valle Costa and F. Dufour, Continuous Average Control of Piecewise Deterministic Markov Processes, Springer, New York, 2013.
9.
E. B. Dynkin and A. A. Yushkevich, Upravlyaemye markovskie protsessy i ikh prilozheniya, Moskva: Nauka, 1975 (in Russian).
10.
I. I. Gikhman and A. V. Skorokhod, Stochastic Differential Equations, Springer, New York, 1972.
11.
R. A. Howard, Dynamic programming and Markov processes, John Wiley & Sons, New York, 1960.
12.
R. A. Howard, Dynamic probabilistic systems. Vol. II: Semi-Markov and decision processes, Dover, Mineola, 2007, pp. 577--1108.
13.
R. Khasminskii, Stochastic stability of differential equations, 2nd ed. Springer, New York, 2012.
14.
N. V. Krylov, On the selection of a Markov process from a system of processes and the construction of quasi-diffusion processes, Math. USSR Izv., 7 (1973), pp. 691--709.
15.
N. V. Krylov, Controlled diffusion processes, 2nd ed., Springer, Berlin, 2009.
16.
N. V. Krylov, On Ito's stochastic integral equations, Theory Probab. Appl., 14 (1969), pp. 330--336.
17.
N. V. Krylov, Addendum: On Ito's Stochastic Integral Equations, Theory Probab. Appl., 17 (1973), pp. 373--374.
18.
N. V. Krylov and M. V. Safonov, A certain property of solutions of parabolic equations with measurable coefficients, Math. USSR Izv., 16 (1981), pp. 151--164.
19.
O. A. Ladyzhenskaja, V. A. Solonnikov, and N. N. Ural'ceva, Linear and Quasi-linear Equations of Parabolic Type, AMS, 1968.
20.
A. A. Ljapunow, E. A. Stschegolkow, and W. J. Arsenin, Arbeiten zur deskriptiven Mengenlehre, Mathematische Forschungsberichte. 1, VEB Deutscher Verlag der Wissenschaften, Berlin, 1955.
21.
N. Lusin, Sur les propriétés des fonctions mesurables, Comptes rendus de l'Académie des Sciences de Paris, 154 (1912), pp. 1688--1690.
22.
P. Mandl, Analytical treatment of one-dimensional Markov processes, Springer, New York, 1968.
23.
H. P. McKean, Stochastic integrals, AMS Chelsea Publishing, 2005.
24.
S. Meyn, The Policy Iteration Algorithm for Average Reward Markov Decision Processes with General State Space, IEEE Trans. Automat. Control, 42 (1997), pp. 1663--1680.
25.
R. Morton, On the optimal control of stationary diffusion processes with inaccessible boundaries and no discounting, J. Appl. Probab., 8 (1971), pp. 551--560.
26.
É. Pardoux and A. Yu. Veretennikov, On the Poisson equation and diffusion approximation. I., Ann. Probab., 29 (2001), pp. 1061--1085.
27.
M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley Series in Probability and Statistics. John Wiley & Sons, Hoboken, 2005.
28.
V. V. Rykov, Controllable Queueing Systems: From the Very Beginning up to Nowadays, Reliability: Theory & Applications (RT&A), vol. 12, 2 (2017), pp. 39--61.
29.
E. A. Shchegol'kov, Über die Uniformisierung gewisser B-Mengen, Dokl. Akad. Nauk SSSR, n. Ser., 59 (1948), pp. 1065--1068.
30.
A. Yu. Veretennikov, On Polynomial Mixing and Convergence Rate for Stochastic Difference and Differential Equations, Theory Probab. Appl., 44 (2000), pp. 361--374.
31.
A. Yu. Veretennikov, On Sobolev Solutions of Poisson Equations in $\mathbb R^d$ with a Parameter, J. Math. Sci., 179 (2011), pp. 48--79.
Information & Authors
Information
Published In
SIAM Journal on Control and Optimization
Pages: 2312 - 2331
DOI: 10.1137/19M1271944
ISSN (online): 1095-7138
Copyright
© 2020, Society for Industrial and Applied Mathematics.
History
Submitted: 2 July 2019
Accepted: 17 May 2020
Published online: 11 August 2020
Keywords
MSC codes
Authors
Funding Information
Russian Academic Excellence Project 5-100
Funding Information
Oberwolfach Research Institute for Mathematics
Funding Information
Russian Foundation for Basic Research https://doi.org/10.13039/501100002261 : 17-01-00633_a
Funding Information
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : CRC 1283
Funding Information
Institut Louis Bachelier https://doi.org/10.13039/501100008419
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Optimizing Noisy Complex Systems Liable to FailureSIAM Journal on Applied Mathematics, Vol. 82, No. 1 | 5 January 2022