On the Koopman Operator of Algorithms
Abstract
A systematic mathematical framework for the study of numerical algorithms would allow comparisons, facilitate conjugacy arguments, and enable the discovery of improved, accelerated, data-driven algorithms. Over the course of the past century, the Koopman operator has provided a mathematical framework for the study of dynamical systems which facilitates conjugacy arguments and can provide efficient reduced descriptions. More recently, numerical approximations of the operator have enabled the analysis of a large number of deterministic and stochastic dynamical systems in a completely data-driven, essentially equation-free pipeline. Discrete- or continuous-time numerical algorithms (integrators, nonlinear equation solvers, optimization algorithms are themselves dynamical systems. In this paper, we use this insight to leverage the Koopman operator framework in the data-driven study of such algorithms and discuss benefits for analysis and acceleration of numerical computation. For algorithms acting on high-dimensional spaces by quickly contracting them toward low-dimensional manifolds, we demonstrate how basis functions adapted to the data help to construct efficient reduced representations of the operator. Our illustrative examples include the gradient descent and Nesterov optimization algorithms as well as the Newton--Raphson algorithm.
1. , Nonparametric uncertainty quantification for stochastic gradient flows , SIAM/ASA J. Uncertain. Quantif. , 3 ( 2015 ), pp. 484 -- 508 , https://doi.org/10.1137/14097940x.
2. , Comparative study of the CG and HBF ODEs used in the global minimization of nonconvex functions , in
Artificial Neural Networks---ICANN 2009 ,Springer, Berlin , 2009 , pp. 668 -- 677 , https://doi.org/10.1007/978-3-642-04274-4_69.3. , On matching, and even rectifying, dynamical systems through Koopman operator eigenfunctions , SIAM J. Appl. Dyn. Syst. , 17 ( 2018 ), pp. 1925 -- 1960 , https://doi.org/10.1137/17m116207x.
4. , Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems , Linear Algebra Appl. , 146 ( 1991 ), pp. 79 -- 91 , https://doi.org/10.1016/0024-3795(91)90021-n.
5. , Discovering governing equations from data by sparse identification of nonlinear dynamical systems , Proc. Natl. Acad. Sci. USA , 113 ( 2016 ), pp. 3932 -- 3937 , https://doi.org/10.1073/pnas.1517384113.
6. , An approximate parametrization of the ergodic partition using time averaged observables , in
Proceedings of the 48th IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, IEEE ,New York , 2009 , https://doi.org/10.1109/cdc.2009.5400512.7. , Applied Koopmanism , Chaos , 22 ( 2012 ), 047510 , https://doi.org/10.1063/1.4772195.
8. , Linear algebra algorithms as dynamical systems , Acta Numer. , 17 ( 2008 ), pp. 1 -- 86 , https://doi.org/10.1017/s0962492906340019.
9. , Koopman operator spectrum for random dynamical systems, J .
Nonlinear Sci. , ( 2019 , https://doi.org/10.1007/s00332-019-09582-z.10. , Splines minimizing rotation-invariant semi-norms in Sobolev spaces , in Constructive Theory of Functions of Several Variables , Springer , Berlin , 1977 , pp. 85 -- 100 , https://doi.org/10.1007/bfb0086566.
11. , Solving equations through particle dynamics , Comput. Phys. Commun. , 197 ( 2015 ), pp. 169 -- 181 , https://doi.org/10.1016/j.cpc.2015.08.028.
12. , Global linearization and fiber bundle structure of invariant manifolds , Nonlinearity , 31 ( 2018 ), pp. 4202 -- 4245 , https://doi.org/10.1088/1361-6544/aaca8d.
13. , A Short Course on Operator Semigroups , Springer , New York , 2006 , https://doi.org/10.1007/0-387-36619-9.
14. , Data-driven spectral decomposition and forecasting of ergodic dynamical systems ,
Appl. Comput. Harmon. Anal., (2017 2017 .09.001. , https://doi.org/10.1016/j.acha.2017.09.001.15. , Study of dynamical systems from the viewpoint of complexity and computational capabilities , Differential Equations , 52 ( 2016 ), pp. 897 -- 905 , https://doi.org/10.1134/s0012266116070090.
16. , Differential equations for roaming pseudospectra: Paths to extremal points and boundary tracking , SIAM J. Numer. Anal. , 49 ( 2011 ), pp. 1194 -- 1209 , https://doi.org/10.1137/100817851.
17. , Deep residual learning for image recognition , in
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE ,New York , 2016 , https://doi.org/10.1109/cvpr.2016.90.18. , Applied Nonlinear Programming , McGraw-Hill , New York , 1972 .
19. , Data-driven model reduction and transfer operator approximation , J. Nonlinear Sci. , 28 ( 2018 ), pp. 985 -- 1010 , https://doi.org/10.1007/s00332-017-9437-7.
20. . von Neumann, Hamiltonian systems and transformation in Hilbert space , Proc. Natl. Acad. Sci. USA , 17 ( 1932 ), pp. 315 -- 318 .
21. , Data-driven spectral analysis of the Koopman operator ,
Appl. Comput. Harmon. Anal. , ( 2018 , https://doi.org/10.1016/j.acha.2018.08.002.22. , Chaos, Fractals, and Noise , Springer New York , 1994 , https://doi.org/10.1007/978-1-4612-4286-4.
23. , New Algorithms for Macromolecular Simulation , Springer , Berlin , 2005 , https://www.ebook.de/de/product/3996213/new_algorithms_for_macromolecular_simulation.html.
24. , Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator , Chaos , 27 ( 2017 ), 103111 , https://doi.org/10.1063/1.4993854.
25. , A generalized eigenvalue algorithm for tridiagonal matrix pencils based on a nonautonomous discrete integrable system , J. Comput. Appl. Math. , 300 ( 2016 ), pp. 134 -- 154 , https://doi.org/10.1016/j.cam.2015.12.032.
26. , Koopman-based lifting techniques for nonlinear systems identification , IEEE Trans. Automat. Control, to appaear. https://doi.org/10.1109/tac. 2019 .2941433 , https://doi.org/10.1109/tac.2019.2941433.
27. , Isostables, isochrons, and Koopman spectrum for the action--angle representation of stable fixed point dynamics , Phys. D , 261 ( 2013 ), pp. 19 -- 30 , https://doi.org/10.1016/j.physd.2013.06.004.
28. , Brownian Motion , Oxford University Press , Oxford , 2008 , https://doi.org/10.1093/acprof:oso/9780199556441.001.0001.
29. , Spectral properties of dynamical systems, model reduction and decompositions , Nonlinear Dynam. , 41 ( 2005 ), pp. 309 -- 325 , https://doi.org/10.1007/s11071-005-2824-x.
30. , Analysis of fluid flows via spectral properties of the Koopman operator , Ann. Rev. Fluid Mech. , 45 ( 2013 ), pp. 357 -- 378 , https://doi.org/10.1146/annurev-fluid-011212-140652.
31. , Koopman Operator Spectrum and Data Analysis, preprint, arXiv, https://arxiv.org/abs/1702.07597v1 , 2017 . , https://arxiv.org/abs/1702.07597v1.
32. , The Dynamics of Newton's Method on Cubic Polynomials , Master's thesis , Marshall University , Huntington, WV , 2006 .
33. . Zhang, On the equivalence between SOR-type methods for linear systems and the discrete gradient methods for gradient systems , J. Comput. Appl. Math. , 342 ( 2018 ), pp. 58 -- 69 , https://doi.org/10.1016/j.cam.2018.04.013.
34. , Eine spektraltheorie für allgemeine operatoren eines unitären raumes ., Math. Nachr. , 4 ( 1950 ), pp. 258 -- 281 , https://doi.org/10.1002/mana.3210040124.
35. , Real-time robust and adaptive solutions to zero finding problems with uncertainty , in
Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, IEEE ,New York , 2009 , https://doi.org/10.1109/cdc.2009.5400043.36. , Design of second order neural networks as dynamical control systems that aim to minimize nonconvex scalar functions , Neurocomputing , 97 ( 2012 ), pp. 174 -- 191 , https://doi.org/10.1016/j.neucom.2012.05.007.
37. , Nonlinear system identification using neural networks: Dynamics and instabilities , in Neural Networks for Chemical Engineers , A. B. Bulsari, eds., Elsevier Science , Amsterdam , 1995 , pp. 409 -- 442 .
38. , Discrete- vs. continuous-time nonlinear signal processing of Cu electrodissolution data , Chem. Eng. Commun. , 118 ( 1992 ), pp. 25 -- 48 , https://doi.org/10.1080/00986449208936084.
39. , A chaotic dynamical system that paints and samples , IFAC-PapersOnLine , 50 ( 2017 ), pp. 10760 -- 10765 , https://doi.org/10.1016/j.ifacol.2017.08.2278.
40. , Dynamic mode decomposition of numerical and experimental data , J. Fluid Mech. , 656 ( 2010 ), pp. 5 -- 28 , https://doi.org/10.1017/s0022112010001217.
41. , Towards tensor-based methods for the numerical approximation of the Perron--Frobenius and Koopman operator , J. Comput. Dyn. , 3 ( 2016 ), pp. 139 -- 161 , https://doi.org/10.3934/jcd.2016007.
42. , Dynamical Systems and Numerical Analysis , Cambridge University Press , Cambridge , 1996 .
43. , A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights , in Advances in Neural Information Processing Systems 27 , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds., Curran Associates , Red Hook, NY , 2014 , pp. 2510 -- 2518 .
44. : An integrable numerical algorithm for computing eigenvalues of a specially structured matrix , Numer. Linear Algebra Appl. , 18 ( 2010 ), pp. 261 -- 274 , https://doi.org/10.1002/nla.754.
45. . Neumann, Zur operatorenmethode in der klassischen mechanik , Ann. Math. , 33 ( 1932 ), 587 , https://doi.org/10.2307/1968537.
46. , A data-driven approximation of the Koopman operator: Extending dynamic mode decomposition , J. Nonlinear Sci. , 25 ( 2015 ), pp. 1307 -- 1346 , https://doi.org/10.1007/s00332-015-9258-5.
47. , A kernel-based method for data-driven Koopman spectral analysis , J. Comput. Dyn. , 2 ( 2015 ), pp. 247 -- 265 , https://doi.org/10.3934/jcd.2015005.
48. , Data fusion via intrinsic dynamic variables: An application of data-driven Koopman spectral analysis , Europhysics Lett. , 109 ( 2015 10 .1209/0295-5075/109/40007. , https://doi.org/10.1209/0295-5075/109/40007.
