We consider parameter estimation of ordinary differential equation (ODE) models from noisy observations. For this problem, one conventional approach is to fit numerical solutions (e.g., Euler, Runge--Kutta) of ODEs to data. However, such a method does not account for the discretization error in numerical solutions and has limited estimation accuracy. In this study, we develop an estimation method that quantifies the discretization error based on data. The key idea is to model the discretization error as random variables and estimate their variance simultaneously with the ODE parameter. The proposed method has the form of iteratively reweighted least squares, where the discretization error variance is updated with the isotonic regression algorithm and the ODE parameter is updated by solving a weighted least squares problem using the adjoint system. Experimental results demonstrate that the proposed method attains robust estimation with at least comparable accuracy to the conventional method by successfully quantifying the reliability of the numerical solutions.


  1. discretization error
  2. isotonic regression
  3. parameter estimation
  4. probabilistic numerics

MSC codes

  1. 62F10
  2. 65L05

Get full access to this article

View all available purchase options and get full access to this article.


A. Abdulle and G. Garegnani, Random time step probabilistic methods for uncertainty quantification in chaotic and geometric numerical integration, Stat. Comput., 30 (2020), pp. 907--932, https://doi.org/10.1007/s11222-020-09926-w.
A. Arnold, D. Calvetti, and E. Somersalo, Linear multistep methods, particle filtering and sequential Monte Carlo, Inverse Problems, 29 (2013), 085007, https://doi.org/10.1088/0266-5611/29/8/085007.
R. E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. D. Brunk, Statistical Inference Under Order Restrictions. The Theory and Application of Isotonic Regression, John Wiley & Sons, London, New York, Sydney, 1972.
S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, Cambridge, UK, 2004.
J. Brynjarsdottir and A. O'Hagan, Learning about physical parameters: The importance of model discrepancy, Inverse Problems, 30 (2014), 114007, https://doi.org/10.1088/0266-5611/30/11/114007.
J. C. Butcher, Numerical Methods for Ordinary Differential Equations, 3rd ed., John Wiley & Sons, Chichester, UK, 2016.
O. A. Chkrebtii, D. A. Campbell, B. Calderhead, and M. A. Girolami, Bayesian solution uncertainty quantification for differential equations, Bayesian Anal., 11 (2016), pp. 1239--1267, https://doi.org/10.1214/16-BA1017.
J. Cockayne, C. J. Oates, T. J. Sullivan, and M. Girolami, Bayesian probabilistic numerical methods, SIAM Rev., 61 (2019), pp. 756--789, https://doi.org/10.1137/17M1139357.
P. R. Conrad, M. Girolami, S. Särkkä, A. Stuart, and K. Zygalakis, Statistical analysis of differential equations: Introducing probability measures on numerical solutions, Stat. Comput., 27 (2017), pp. 1065--1082, https://doi.org/10.1007/s11222-016-9671-0.
T. S. Ferguson, A Course in Large Sample Theory, Chapman & Hall, London, 1996, https://doi.org/10.1201/9781315136288.
R. FitzHugh, Impulses and physiological states in models of nerve membrane, Biophys. J., 1 (1961), pp. 445--466, https://doi.org/10.1016/S0006-3495(61)86902-6.
L. Grippo and M. Sciandrone, On the convergence of the block nonlinear Gauss-Seidel method under convex constraints, Oper. Res. Lett., 26 (2000), pp. 127--136, https://doi.org/10.1016/S0167-6377(99)00074-7.
E. Hairer, C. Lubich, and G. Wanner, Geometric numerical integration illustrated by the Störmer--Verlet method, Acta Numer., 12 (2003), pp. 399--450, https://doi.org/10.1017/S0962492902000144.
E. Hairer, C. Lubich, and G. Wanner, Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, 2nd ed., Springer-Verlag, Berlin, 2006.
E. Hairer, R. I. McLachlan, and A. Razakarivony, Achieving Brouwer's law with implicit Runge--Kutta methods, BIT, 48 (2008), pp. 231--243, https://doi.org/10.1007/s10543-008-0170-3.
E. Hairer, S. P. Nørsett, and G. Wanner, Solving Ordinary Differential Equations I. Nonstiff Problems, 2nd ed., Springer-Verlag, Berlin, 1993.
P. Hennig and S. Hauberg, Probabilistic solutions to differential equations and their application to Riemannian statistics, in Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, 2014, pp. 347--355.
P. Hennig, M. A. Osborne, and M. Girolami, Probabilistic numerics and uncertainty in computations, Proc. A, 471 (2015), 20150142, https://doi.org/10.1098/rspa.2015.0142.
S. Ito, T. Matsuda, and Y. Miyatake, Adjoint-based exact Hessian computation, BIT, to appear.
H. Kersting and P. Hennig, Active uncertainty calibration in Bayesian ODE solvers, in Uncertainty in Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, 2016, pp. 309--318, https://dl.acm.org/doi/10.5555/3020948.3020981.
B. Leimkuhler and S. Reich, Simulating Hamiltonian Dynamics, Cambridge University Press, Cambridge, UK, 2004.
H. C. Lie, T. J. Sullivan, and A. Stuart, Strong convergence rates of probabilistic integrators for ordinary differential equations, Stat. Comput., 29 (2019), pp. 1265--1283, https://doi.org/10.1007/s11222-019-09898-6.
E. N. Lorenz, Deterministic nonperiodic flow, J. Atmospheric Sci., 20 (1963), pp. 130--141.
T. Matsuda and Y. Miyatake, Generalization of partitioned Runge--Kutta methods for adjoint systems, J. Comput. Appl. Math., 388 (2021), 113308.
J. S. Nagumo, S. Arimoto, and S. Yoshizawa, An active pulse transmission line simulating a nerve axon, in Proceedings of the Institute of Radio Engineers, Vol. 50, 1962, pp. 2061--2070, https://doi.org/10.1109/JRPROC.1962.288235.
C. J. Oates, J. Cockayne, R. G. Aykroyd, and M. Girolami, Bayesian probabilistic numerical methods in time-dependent state estimation for industrial hydrocyclone equipment, J. Amer. Statist. Assoc., 114 (2019), pp. 1518--1531, https://doi.org/10.1080/01621459.2019.1574583.
M. J. D. Powell, On search directions for minimization algorithms, Math. Programming, 4 (1973), pp. 193--201, https://doi.org/10.1007/BF01584660.
T. Robertson, F. T. Wright, and R. L. Dykstra, Order Restricted Statistical Inference, Wiley Ser. Probab. Math. Statist. Probab. Math. Statist., John Wiley & Sons, Chichester, UK, 1988.
J. M. Sanz-Serna, Symplectic integrators for Hamiltonian problems: An overview, Acta Numer., 1 (1992), pp. 243--286, https://doi.org/10.1017/s0962492900002282.
J. M. Sanz-Serna, Symplectic Runge--Kutta schemes for adjoint equations, automatic differentiation, optimal control, and more, SIAM Rev., 58 (2016), pp. 3--33, https://doi.org/10.1137/151002769.
J. M. Sanz-Serna and M. P. Calvo, Numerical Hamiltonian Problems, Chapman & Hall, London, 1994.
M. Schober, D. K. Duvenaud, and P. Hennig, Probabilistic ODE solvers with Runge--Kutta means, in Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 2014, pp. 739--747.
M. Schober, S. Särkkä, and P. Hennig, A probabilistic model for the numerical solution of initial value problems, Stat. Comput., 29 (2019), pp. 99--122, https://doi.org/10.1007/s11222-017-9798-7.
O. Teymur, K. Zygalakis, and B. Calderhead, Probabilistic linear multistep methods, in Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 2016, pp. 4321--4328.
F. Tronarp, H. Kersting, S. Särkkä, and P. Hennig, Probabilistic solutions to ordinary differential equations as non-linear Bayesian filtering: A new perspective, Stat. Comput., 29 (2019), pp. 1297--1315, https://doi.org/10.1007/s11222-019-09900-1.
C. van Eeden, Restricted Parameter Space Estimation Problems, Springer, New York, 2006, https://doi.org/10.1007/978-0-387-48809-7.
D. J. Venzon and S. H. Moolgavkar, A method for computing profile-likelihood-based confidence intervals, J. R. Statist. Soc. C, 37 (1988), pp. 87--94, https://doi.org/10.2307/2347496.
H. Xue, H. Miao, and H. Wu, Sieve estimation of constant and time-varying coefficients in nonlinear ordinary differential equation models by considering both numerical error and measurement error, Ann. Statist., 38 (2010), pp. 2351--2387, https://doi.org/10.1214/09-AOS784.

Information & Authors


Published In

cover image SIAM/ASA Journal on Uncertainty Quantification
SIAM/ASA Journal on Uncertainty Quantification
Pages: 302 - 331
ISSN (online): 2166-2525


Submitted: 31 July 2019
Accepted: 30 December 2020
Published online: 24 March 2021


  1. discretization error
  2. isotonic regression
  3. parameter estimation
  4. probabilistic numerics

MSC codes

  1. 62F10
  2. 65L05



Metrics & Citations



If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

There are no citations for this item







Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.