Abstract

We consider filtering in high-dimensional non-Gaussian state-space models with intractable transition kernels, nonlinear and possibly chaotic dynamics, and sparse observations in space and time. We propose a novel filtering methodology that harnesses transportation of measures, convex optimization, and ideas from probabilistic graphical models to yield robust ensemble approximations of the filtering distribution in high dimensions. Our approach can be understood as the natural generalization of the ensemble Kalman filter (EnKF) to nonlinear updates, using stochastic or deterministic couplings. The use of nonlinear updates can reduce the intrinsic bias of the EnKF at a marginal increase in computational cost. We avoid any form of importance sampling and introduce non-Gaussian localization approaches for dimension scalability. Our framework achieves state-of-the-art tracking performance on challenging configurations of the Lorenz-96 model in the chaotic regime.

Keywords

  1. nonlinear filtering
  2. state-space models
  3. couplings
  4. transport maps
  5. ensemble Kalman filter
  6. graphical models
  7. localization
  8. approximate Bayesian computation

MSC codes

  1. 93E11
  2. 62M20
  3. 60G35
  4. 62L12

Get full access to this article

View all available purchase options and get full access to this article.

Supplementary Material


PLEASE NOTE: These supplementary files have not been peer-reviewed.


Index of Supplementary Materials

Title of paper: Coupling Techniques for Nonlinear Ensemble Filtering

Authors: Alessio Spantini, Ricardo Baptista, and Youssef Marzouk

File: M131220_supplementary.pdf

Type: PDF

Contents: In the supplementary text, we provide more details on the numerical computation of triangular maps in Section SM1. We also describe a variant of the deterministic map filtering algorithm from Section 10 for the setting of conditionally independent and local observations in Section SM2. Lastly, we provide additional numerical results using the stochastic map algorithm in Sections SM3-SM6.

References

1.
W. Acevedo, J. de Wiljes, and S. Reich, Second-order accurate ensemble transform particle filters, SIAM J. Sci. Comput., 39 (2017), pp. A1834--A1850, https://doi.org/10.1137/16M1095184.
2.
J. Amezcua and P. J. Van Leeuwen, Gaussian anamorphosis in the analysis step of the EnKF: A joint state-variable/observation approach, Tellus Ser. A, 66 (2014), art. 23493.
3.
E. Anderes and M. Coram, A General Spline Representation for Nonparametric and Semiparametric Density Estimates Using Diffeomorphisms, preprint, https://arxiv.org/abs/1205.5314, 2012.
4.
R. Baptista, Y. Marzouk, and O. Zahm, On the Representation and Learning of Monotone Triangular Transport Maps, preprint, https://arxiv.org/abs/2009.10303, 2022.
5.
T. Bengtsson, C. Snyder, and D. Nychka, Toward a nonlinear ensemble filter for high-dimensional systems, J. Geophys. Res. Atmos., 108 (D24) (2003), art. 8775.
6.
P. J. Bickel and E. Levina, Regularized estimation of large covariance matrices, Ann. Statist., 36 (2008), pp. 199--227.
7.
V. I. Bogachev, A. V. Kolesnikov, and K. V. Medvedev, Triangular transformations of measures, Sb. Math., 196 (2005), pp. 309--335.
8.
G. E. Box and D. R. Cox, An analysis of transformations, J. Roy. Statist. Soc. Ser. B, 26 (1964), pp. 211--252.
9.
M. Brennan, D. Bigoni, O. Zahm, A. Spantini, and Y. Marzouk, Greedy inference with structure-exploiting lazy maps, in Advances in Neural Information Processing Systems, 2020, pp. 8330--8342.
10.
G. Burgers, P. Jan van Leeuwen, and G. Evensen, Analysis scheme in the ensemble Kalman filter, Monthly Weather Rev., 126 (1998), pp. 1719--1724.
11.
A. J. Chorin and X. Tu, Implicit sampling for particle filters, Proc. Natl. Acad. Sci. USA, 106 (2009), pp. 17249--17254.
12.
K. Cranmer, J. Brehmer, and G. Louppe, The frontier of simulation-based inference, Proc. Natl. Acad. Sci. USA, 117 (2020), pp. 30055--30062.
13.
D. Crisan and A. Doucet, A survey of convergence results on particle filtering methods for practitioners, IEEE Trans. Signal Process., 50 (2002), pp. 736--746.
14.
T. Cui, J. Martin, Y. Marzouk, A. Solonen, and A. Spantini, Likelihood-informed dimension reduction for nonlinear inverse problems, Inverse Problems, 30 (2014), art. 114015.
15.
F. Daum and J. Huang, Particle flow for nonlinear filters with log-homotopy, in SPIE Defense and Security Symposium, International Society for Optics and Photonics, SPIE, 2008, art. 696918.
16.
P. Del Moral, A. Kurtzmann, and J. Tugaut, On the stability and the uniform propagation of chaos of a class of extended ensemble Kalman--Bucy filters, SIAM J. Control Optim., 55 (2017), pp. 119--155, https://doi.org/10.1137/16M1087497.
17.
G. Detommaso, T. Cui, Y. Marzouk, A. Spantini, and R. Scheichl, A Stein variational Newton method, in Advances in Neural Information Processing Systems, 2018, pp. 9169--9179.
18.
A. Doucet and A. M. Johansen, A tutorial on particle filtering and smoothing: Fifteen years later, in The Oxford Handbook of Nonlinear Filtering, Oxford University Press, 2011, pp. 656--704.
19.
R. J. Douglas, Applications of the Monge-Ampere equation and Monge transport problem to meteorology and oceanography, in Monge Ampère Equation: Applications to Geometry and Optimization, Contemp. Math. 226, American Mathematical Society, 1999, pp. 33--53.
20.
J. Durbin and S. J. Koopman, Time Series Analysis by State Space Methods, 2nd ed., Oxford Statist. Sci. Ser. 38, Oxford University Press, 2012.
21.
G. Evensen, Data Assimilation, Springer, 2007.
22.
M. Frei and H. R. Künsch, Bridging the ensemble Kalman and particle filters, Biometrika, 100 (2013), pp. 781--800.
23.
T. Gneiting, F. Balabdaoui, and A. E. Raftery, Probabilistic forecasts, calibration and sharpness, J. R. Stat. Soc. Ser. B Stat. Methodol., 69 (2007), pp. 243--268.
24.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, Generative adversarial nets, in Advances in Neural Information Processing Systems, 2014, pp. 2672--2680.
25.
N. J. Gordon, D. J. Salmond, and A. F. Smith, Novel approach to nonlinear/non-Gaussian Bayesian state estimation, IEE Proc. F, 140 (1993), pp. 107--113.
26.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer, 2009.
27.
J. Heng, A. Doucet, and Y. Pokern, Gibbs flow for approximate transport with applications to Bayesian computation, J. R. Stat. Soc. Ser. B. Stat. Methodol., 83 (2021), pp. 156--187.
28.
J. Houssineau, A. Jasra, and S. S. Singh, Multilevel Monte Carlo for smoothing via transport methods, SIAM J. Sci. Comput., 40 (2018), pp. A2315--A2335, https://doi.org/10.1137/17M1156071.
29.
P. L. Houtekamer and H. L. Mitchell, A sequential ensemble Kalman filter for atmospheric data assimilation, Monthly Weather Rev., 129 (2001), pp. 123--137.
30.
L. V. Kantorovich, The Best Use of Economic Resources, Pergamon Press, 1965.
31.
M. Katzfuss, J. R. Stroud, and C. K. Wikle, Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models, J. Amer. Statist. Assoc., 115 (2020), pp. 866--885.
32.
S. Kim, R. Ma, D. Mesa, and T. Coleman, Efficient Bayesian inference methods via convex optimization and optimal transport, in the 2013 IEEE International Symposium on Information Theory, 2013, pp. 2259--2263.
33.
D. Koller and N. Friedman, Probabilistic Graphical Models: Principles and Techniques, MIT Press, 2009.
34.
S. L. Lauritzen, Graphical Models, Oxford University Press, 1996.
35.
K. J. Law and A. M. Stuart, Evaluating data assimilation algorithms, Monthly Weather Rev., 140 (2012), pp. 3757--3782.
36.
M. Le Provost, R. Baptista, Y. Marzouk, and J. Eldredge, A low-rank nonlinear ensemble filter for vortex models of aerodynamic flows, in AIAA Scitech 2021 Forum, 2021, art. 1937.
37.
J. Lei and P. Bickel, A moment matching ensemble filter for nonlinear non-Gaussian data assimilation, Monthly Weather Rev., 139 (2011), pp. 3964--3973.
38.
F. Lindgren, H. Rue, and J. Lindström, An explicit link between Gaussian fields and Gaussian Markov random fields: The stochastic partial differential equation approach, J. Roy. Statist. Soc. Ser. B, 73 (2011), pp. 423--498.
39.
Q. Liu and D. Wang, Stein variational gradient descent: A general purpose Bayesian inference algorithm, in Advances in Neural Information Processing Systems, 2016, pp. 2370--2378.
40.
E. N. Lorenz, Deterministic nonperiodic flow, J. Atmospher. Sci., 20 (1963), pp. 130--141.
41.
E. N. Lorenz, Predictability: A problem partly solved, in Proc. Seminar on Predictability, Vol. 1, 1996, pp. 1--18.
42.
A. J. Majda and J. Harlim, Filtering Complex Turbulent Systems, Cambridge University Press, 2012.
43.
J. Mandel, L. Cobb, and J. D. Beezley, On the convergence of the ensemble Kalman filter, Appl. Math., 56 (2011), pp. 533--541.
44.
J. M. Marin, P. Pudlo, C. P. Robert, and R. J. Ryder, Approximate Bayesian computational methods, Statist. Comput., 22 (2012), pp. 1167--1180.
45.
Y. Marzouk, T. Moselhy, M. Parno, and A. Spantini, Sampling via measure transport: An introduction, in Handbook of Uncertainty Quantification, Springer, 2016, pp. 1--41.
46.
S. Metref, E. Cosme, C. Snyder, and P. Brasseur, A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation, Nonlinear Process. Geophys., 21 (2014), pp. 869--885.
47.
G. Monge, Mémoire sur la théorie des déblais et des remblais, De l'Imprimerie Royale, 1781.
48.
M. Morzfeld, X. Tu, E. Atkins, and A. J. Chorin, A random map implementation of implicit filters, J. Comput. Phys., 231 (2012), pp. 2049--2066.
49.
T. Moselhy and Y. Marzouk, Bayesian inference with optimal maps, J. Comput. Phys., 231 (2012), pp. 7815--7850.
50.
E. D. Nino-Ruiz, A. Sandu, and X. Deng, An ensemble Kalman filter implementation based on modified Cholesky decomposition for inverse covariance matrix estimation, SIAM J. Sci. Comput., 40 (2018), pp. A867--A886, https://doi.org/10.1137/16M1097031.
51.
M. D. Parno and Y. M. Marzouk, Transport map accelerated Markov chain Monte Carlo, SIAM/ASA J. Uncertain. Quantif., 6 (2018), pp. 645--682, https://doi.org/10.1137/17M1134640.
52.
J. Poterjoy, A localized particle filter for high-dimensional nonlinear systems, Monthly Weather Rev., 144 (2016), pp. 59--76.
53.
W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes: The Art of Scientific Computing, 3rd ed., Cambridge University Press, 2007.
54.
M. Pulido and P. J. van Leeuwen, Sequential Monte Carlo with kernel embedded mappings: The mapping particle filter, J. Comput. Phys., 396 (2019), pp. 400--415, https://doi.org/10.1016/j.jcp.2019.06.060.
55.
S. Reich, A nonparametric ensemble transform method for Bayesian inference, SIAM J. Sci. Comput., 35 (2013), pp. A2013--A2024, https://doi.org/10.1137/130907367.
56.
S. Reich and C. Cotter, Probabilistic Forecasting and Bayesian Data Assimilation, Cambridge University Press, 2015.
57.
D. J. Rezende and S. Mohamed, Variational inference with normalizing flows, in Proceedings of the 32nd International Conference on Machine Learning, PMLR, 2015, pp. 1530--1538.
58.
M. Rosenblatt, Remarks on a multivariate transformation, Ann. Math. Statist., 23 (1952), pp. 470--472.
59.
J. S\aetrom and H. Omre, Ensemble Kalman filtering with shrinkage regression techniques, Comput. Geosci., 15 (2011), pp. 271--292.
60.
F. Santambrogio, Optimal Transport for Applied Mathematicians, Progr. Nonlinear Differential Equations Appl. 87, Springer, 2015.
61.
C. Snyder, T. Bengtsson, P. Bickel, and J. Anderson, Obstacles to high-dimensional particle filtering, Monthly Weather Rev., 136 (2008), pp. 4629--4640.
62.
A. Spantini, On the Low-Dimensional Structure of Bayesian Inference, Ph.D. thesis, Massachusetts Institute of Technology, 2017.
63.
A. Spantini, D. Bigoni, and Y. Marzouk, Inference via low-dimensional couplings, J. Mach. Learn. Res., 19 (2018), pp. 2639--2709.
64.
A. Spantini, A. Solonen, T. Cui, J. Martin, L. Tenorio, and Y. Marzouk, Optimal low-rank approximations of Bayesian linear inverse problems, SIAM J. Sci. Comput., 37 (2015), pp. A2451--A2487, https://doi.org/10.1137/140977308.
65.
M. K. Tippett, J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, Ensemble square root filters, Monthly Weather Rev., 131 (2003), pp. 1485--1490.
66.
G. Ueno and T. Tsuchiya, Covariance regularization in inverse space, Quart. J. Roy. Meteor. Soc., 135 (2009), pp. 1133--1156.
67.
P. J. van Leeuwen, H. R. Künsch, L. Nerger, R. Potthast, and S. Reich, Particle filters for high-dimensional geoscience applications: A review, Quart. J. Roy. Meteorol. Soc., 145 (2019), pp. 2335--2365, https://doi.org/10.1002/qj.3551.
68.
C. Villani, Optimal Transport: Old and New, Grundlehren Math. Wiss. 338, Springer Science & Business Media, 2008.
69.
H. Wackernagel, Multivariate Geostatistics: An Introduction with Applications, Springer, 2003.
70.
S. J. Wright and J. Nocedal, Numerical Optimization, Vol. 2, Springer, New York, 1999.
71.
W. B. Wu and M. Pourahmadi, Nonparametric estimation of large covariance matrices of longitudinal data, Biometrika, 90 (2003), pp. 831--844.
72.
T. Yang, P. G. Mehta, and S. P. Meyn, Feedback particle filter, IEEE Trans. Automat. Control, 58 (2013), pp. 2465--2480.

Information & Authors

Information

Published In

cover image SIAM Review
SIAM Review
Pages: 921 - 953
ISSN (online): 1095-7200

History

Submitted: 10 January 2020
Accepted: 16 November 2021
Published online: 3 November 2022

Keywords

  1. nonlinear filtering
  2. state-space models
  3. couplings
  4. transport maps
  5. ensemble Kalman filter
  6. graphical models
  7. localization
  8. approximate Bayesian computation

MSC codes

  1. 93E11
  2. 62M20
  3. 60G35
  4. 62L12

Authors

Affiliations

Funding Information

Natural Sciences and Engineering Research Council of Canada : PGSD-D Fellowship
Air Force Office of Scientific Research https://doi.org/10.13039/100000181 : FA9550-15-1-0038
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : CRC 1294
U.S. Department of Energy https://doi.org/10.13039/100000015 : AEOLUS

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

View Options

View options

PDF

View PDF

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media