Bayesian Quadrature, Energy Minimization, and Space-Filling Design
Abstract
A standard objective in computer experiments is to approximate the behavior of an unknown function on a compact domain from a few evaluations inside the domain. When little is known about the function, space-filling design is advisable: typically, points of evaluation spread out across the available space are obtained by minimizing a geometrical (for instance, covering radius) or a discrepancy criterion measuring distance to uniformity. The paper investigates connections between design for integration (quadrature design), construction of the (continuous) best linear unbiased estimator (BLUE) for the location model, space-filling design, and minimization of energy (kernel discrepancy) for signed measures. Integrally strictly positive definite kernels define strictly convex energy functionals, with an equivalence between the notions of potential and directional derivative, showing the strong relation between discrepancy minimization and more traditional design of optimal experiments. In particular, kernel herding algorithms, which are special instances of vertex-direction methods used in optimal design, can be applied to the construction of point sequences with suitable space-filling properties.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
H. Aikawa and M. Essén, Potential Theory---Selected Topics, Springer, Berlin, 1996.
2.
A. Antoniadis, Analysis of variance on function spaces, Math. Operationsforsch. Statist. Ser. Statist., 15 (1984), pp. 59--71.
3.
P. Audze and V. Eglais, New approach for planning out of experiments, Probl. Dynam. Strengths, 35 (1977), pp. 104--107.
4.
Y. Auffray, P. Barbillon, and J.-M. Marin, Maximin design on non hypercube domains and kernel interpolation, Stat. Comput., 22 (2012), pp. 703--712.
5.
F. Bach, S. Lacoste-Julien, and G. Obozinski, On the Equivalence between Herding and Conditional Gradient Algorithms, preprint, https://arxiv.org/abs/1203.4523, 2012.
6.
A. Beck and M. Teboulle, A conditional gradient method with linear rate of convergence for solving convex linear systems, Math. Methods Oper. Res., 59 (2004), pp. 235--247.
7.
J. Bect, F. Bachoc, and D. Ginsbourger, A supermartingale approach to Gaussian process based sequential design of experiments, Bernoulli, 25 (2019), pp. 2883--2919.
8.
J. Bect, D. Ginsbourger, L. Li, V. Picheny, and E. Vazquez, Sequential design of computer experiments for the estimation of a probability of failure, Stat. Comput., 22 (2012), pp. 773--793.
9.
A. Berlinet and C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics, Kluwer, Boston, MA, 2004.
10.
S. Biedermann and H. Dette, Minimax optimal designs for nonparametric regression---a further optimality property of the uniform distribution, in Proceedings of the 6th International Workshop on Advances in Model-Oriented Design and Analysis (MODA) (Puchberg/Schneeberg, Austria), P. Hackl, A.C. Atkinson, and W.G. Müller, eds., Physica Verlag, Heidelberg, 2001, pp. 13--20.
11.
G. Björck, Distributions of positive mass, which maximize a certain generalized energy integral, Ark. Mat., 3 (1956), pp. 255--269.
12.
D. Böhning, Numerical estimation of a probability measure, J. Statist. Plann. Inference, 11 (1985), pp. 57--69.
13.
D. Böhning, A vertex-exchange-method in $D$-optimal design theory, Metrika, 33 (1986), pp. 337--347.
14.
A. Breger, M. Ether, and M. Gräf, Points on manifolds with asymptotically optimal covering radius, J. Complexity, 48 (2018), pp. 1--14.
15.
F.-X. Briol, C. Oates, M. Girolami, and M. A. Osborne, Frank-Wolfe Bayesian quadrature: Probabilistic integration with theoretical guarantees, in Advances in Neural Information Processing Systems (NIPS 2015), NeurIPS, San Diego, CA, 2015, pp. 1162--1170.
16.
F.-X. Briol, C. J. Oates, M. Girolami, M.A. Osborne, and D. Sejdinovic, Probabilistic integration: A role in statistical computation?, Statist. Sci., 34 (2019), pp. 1--22.
17.
H. Cardot, P. Cénac, and J.-M. Monnez, A fast and recursive algorithm for clustering large datasets, Comput. Statist. Data Anal., 56 (2012), pp. 1434--1449.
18.
W. Y. Chen, L. Mackey, J. Gorham, F.-X. Briol, and C. J. Oates, Stein Points, preprint, https://arxiv.org/abs/1803.10161, 2018.
19.
Y. Chen, M. Welling, and A. Smola, Super-samples from kernel herding, in Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI'10), Catalina Island, CA, 2010, AUAI Press, Arlington, VA, pp. 109--116.
20.
K. M. Clarkson, Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm, ACM Trans. Algorithms, 6 (2010), 63.
21.
S. B. Damelin, F. J. Hickernell, D. L. Ragozin, and X. Zeng, On energy, discrepancy and group invariant measures on measurable subsets of Euclidean space, J. Fourier Anal. Appl., 16 (2010), pp. 813--839.
22.
G. Detommaso, T. Cui, Y. Marzouk, A. Spantini, and R. Scheichl, A Stein variational Newton method, in Advances in Neural Information Processing Systems (NIPS 2018), NeurIPS, San Diego, CA, 2018, pp. 9187--9197.
23.
H. Dette, A. Pepelyshev, and A. Zhigljavsky, Best linear unbiased estimators in continuous time regression models, Ann. Statist., 47 (2019), pp. 1928--1959.
24.
P. Diaconis, Bayesian numerical analysis, in Statistical Decision Theory and Related Topics IV, Springer, New York, 1988, pp. 163--175.
25.
J. Dick and F. Pillichshammer, Digital Nets and Sequences. Discrepancy Theory and Quasi-Monte Carlo Integration, Cambridge University Press, Cambridge, UK, 2010.
26.
Q. Du, V. Faber, and M. Gunzburger, Centroidal Voronoi tessellations: Applications and algorithms, SIAM Rev., 41 (1999), pp. 637--676, https://doi.org/10.1137/S0036144599352836.
27.
J. C. Dunn, Convergence rates for conditional gradient sequences generated by implicit step length rules, SIAM J. Control Optim., 18 (1980), pp. 473--487, https://doi.org/10.1137/0318035.
28.
J. C. Dunn and S. Harshbarger. Conditional gradient algorithms with open loop step size rules, J. Math. Anal. Appl., 62 (1978), pp. 432--444.
29.
N. Durrande, D. Ginsbourger, O. Roustant, and L. Carraro, ANOVA kernels and RKHS of zero mean functions for model-based sensitivity analysis, J. Multivariate Anal., 115 (2013), pp. 57--67.
30.
N. Durrande, J. Hensman, M. Rattray, and N. D. Lawrence, Detecting periodicities with Gaussian processes, Peer J. Comput. Sci., 2 (2016), e50.
31.
K.-T. Fang, R. Li, and A. Sudjianto, Design and Modeling for Computer Experiments, Chapman & Hall/CRC, Boca Raton, FL, 2006.
32.
V.V. Fedorov, Theory of Optimal Experiments, Academic Press, New York, 1972.
33.
M. Frank and P. Wolfe, An algorithm for quadratic programming, Naval Res. Logist. Quart., 3 (1956), pp. 95--110.
34.
B. Fuglede, On the theory of potentials in locally compact spaces, Acta Math., 103 (1960), pp. 139--215.
35.
B. Fuglede and N. Zorii, Green kernels associated with Riesz kernels, Ann. Acad. Sci. Fenn. Math., 43 (2018), pp. 121--145.
36.
B. Gauthier and L. Pronzato, Spectral approximation of the IMSE criterion for optimal designs in kernel-based interpolation models, SIAM/ASA J. Uncertain. Quantif., 2 (2014), pp. 805--825, https://doi.org/10.1137/130928534.
37.
B. Gauthier and L. Pronzato, Convex relaxation for IMSE optimal design in random-field models, Comput. Statist. Data Anal., 113 (2017), pp. 375--394.
38.
D. Ginsbourger, Sequential Design of Computer Experiments, Wiley StatsRef 99, John Wiley, Hoboken, NJ, 2017, pp. 1--11.
39.
D. Ginsbourger, O. Roustant, D. Schuhmacher, N. Durrande, and N. Lenz, On ANOVA decompositions of kernels and Gaussian random field paths, in Monte Carlo and Quasi-Monte Carlo Methods, Springer, Berlin, 2016, pp. 315--330.
40.
T. Gneiting, Strictly and non-strictly positive define functions on spheres, Bernoulli, 19 (2013), pp. 1327--1349.
41.
A. Gorodetsky and Y. Marzouk, Mercer kernels and integrated variance experimental design: Connections between Gaussian process regression and polynomial approximation, SIAM/ASA J. Uncertain. Quantif., 4 (2016), pp. 796--828, https://doi.org/10.1137/15M1017119.
42.
S. Graf and H. Luschgy, Foundations of Quantization for Probability Distributions, Springer, Berlin, 2000.
43.
U. Grenander, Stochastic processes and statistical inference, Ark. Mat., 1 (1950), pp. 195--277.
44.
J. Hájek, Linear estimation of the mean value of a stationary random process with convex correlation function, Czechoslovak Math. J., 6 (1956), pp. 94--117 (in Russian).
45.
D. P. Hardin and E. B. Saff, Discretizing manifolds via minimum energy points, Notices Amer. Math. Soc., 51 (2004), pp. 1186--1194.
46.
P. Hennig, M. A. Osborne, and M. Girolami, Probabilistic numerics and uncertainty in computations, Proc. A, 471 (2015), 20150142.
47.
J. Hensman, N. Durrande, and A. Solin, Variational Fourier features for Gaussian processes, J. Mach. Learn. Res., 18 (2018), pp. 1--52.
48.
F. J. Hickernell, A generalized discrepancy and quadrature error bound, Math. Comp., 67 (1998), pp. 299--322.
49.
F. Huszár and D. Duvenaud, Optimally-weighted herding is Bayesian quadrature, in Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI'12) (Catalina Island, CA), AUAI Press, Arlington, VA, 2012, pp. 377--385.
50.
M. E. Johnson, L. M. Moore, and D. Ylvisaker, Minimax and maximin distance designs, J. Statist. Plann. Inference, 26 (1990), pp. 131--148.
51.
V. R. Joseph, T. Dasgupta, R. Tuo, and C. F. J. Wu, Sequential exploration of complex surfaces using minimum energy designs, Technometrics, 57 (2015), pp. 64--74.
52.
V. R. Joseph, E. Gul, and S. Ba, Maximum projection designs for computer experiments, Biometrika, 102 (2015), pp. 371--380.
53.
T. Karvonen, C. J. Oates, and S. Särkkä, A Bayes-Sard cubature method, in Advances in Neural Information Processing Systems (NIPS 2018), NeurIPS, San Diego, CA, 2018, pp. 5886--5897.
54.
T. Karvonen and S. Särkkä, Classical quadrature rules via Gaussian processes, in Proceedings of the 27th IEEE International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, Washington, DC, 2017, pp. 1--6.
55.
J. Kiefer and J. Wolfowitz, Optimum designs in regression problems, Ann. Math. Statist., 30 (1959), pp. 271--294.
56.
N. M. Korobov, Properties and calculation of optimal coefficients, Dokl. Akad. Nauk SSSR, 132 (1960), pp. 1009--1012 (in Russian).
57.
N. S. Landkof, Foundations of Modern Potential Theory, Springer, Berlin, 1972.
58.
F. M. Larkin, Gaussian measure in Hilbert space and applications in numerical analysis, Rocky Mountain J. Math., 2 (1972), pp. 379--421.
59.
F. M. Larkin, Probabilistic error estimates in spline interpolation and quadrature, in Information Processing 74: Proceedings of the IFIP Congress (Stockholm, Sweden, 1974), North-Holland, Amsterdam, 1974, pp. 605--609.
60.
R. Lekivetz and B. Jones, Fast flexible space-filling designs for nonrectangular regions, Qual. Reliab. Engrg. Internat., 31 (2015), pp. 829--837.
61.
Q. Liu and D. Wang, Stein variational gradient descent: A general purpose Bayesian inference algorithm, in Advances in Neural Information Processing Systems (NIPS 2016), NeurIPS, San Diego, CA, 2016, pp. 2378--2386.
62.
H. Luschgy and G. Pagès, Greedy vector quantization, J. Approx. Theory, 198 (2015), pp. 111--131.
63.
S. Mak and V. R. Joseph, Projected Support Points, with Application to Optimal MCMC Reduction, preprint, https://arxiv.org/abs/1708.06897v1, 2017.
64.
S. Mak and V. R. Joseph, Minimax and minimax projection designs using clustering, J. Comput. Graph. Statist., 27 (2018), pp. 166--178.
65.
S. Mak and V. R. Joseph, Support points, Ann. Statist., 46 (2018), pp. 2562--2592.
66.
J. Matousek, On the $L2$-discrepancy for anchored boxes, J. Complexity, 14 (1998), pp. 527--556.
67.
I. Molchanov and S. Zuyev, Variational calculus in the space of measures and optimal design, in Optimum Design 2000, A. Atkinson, B. Bogacka, and A. Zhigljavsky, eds., Kluwer, Dordrecht, The Netherlands, 2001, pp. 79--90.
68.
I. Molchanov and S. Zuyev, Steepest descent algorithm in a space of measures, Stat. Comput., 12 (2002), pp. 115--123.
69.
W. Näther, Effective Observation of Random Fields, Teubner-Texte zur Mathematik 72, BSB B. G. Teubner Verlagsgesellschaft, Leipzig, Germany, 1985.
70.
H. Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods, CMBS-NSF Reg. Conf. Ser. Appl. Math. 63, SIAM, Philadelphia, 1992, https://doi.org/10.1137/1.9781611970081.
71.
C. J. Oates, A. Barp, and M. Girolami, Posterior Integration on a Riemannian Manifold. preprint, https://arxiv.org/abs/1712.01793, 2018.
72.
C. J. Oates, M. Girolami, and N. Chopin, Control functionals for Monte Carlo integration, J. R. Stat. Soc. Ser. B Stat. Methodol., 79 (2017), pp. 695--718.
73.
A. O'Hagan, Bayes-Hermite quadrature, J. Statist. Plann. Inference, 29 (1991), pp. 245--260.
74.
V. I. Paulsen, An Introduction to the Theory of Reproducing Kernel Hilbert Spaces, https://www.math.uh.edu/~vern/rkhs.pdf, 2009.
75.
L. Pronzato, Minimax and maximin space-filling designs: Some properties and methods for construction, J. SFdS, 158 (2017), pp. 7--36.
76.
L. Pronzato and A. Pázman, Design of Experiments in Nonlinear Models. Asymptotic Normality, Optimality Criteria and Small-Sample Properties, Lectures Notes in Statistics 212, Springer, New York, 2013.
77.
L. Pronzato, H. P. Wynn, and A. Zhigljavsky, Extremal measures maximizing functionals based on simplicial volumes, Statist. Papers, 57 (2016), pp. 1059--1075.
78.
L. Pronzato and A. A. Zhigljavsky, Algorithmic construction of optimal designs on compact sets for concave and differentiable criteria, J. Statist. Plann. Inference, 154 (2014), pp. 141--155.
79.
L. Pronzato and A. A. Zhigljavsky, Measures minimizing regularized dispersion, J. Sci. Comput., 78 (2019), pp. 1550--1570.
80.
L. Pronzato and A. A. Zhigljavsky, Minimum-energy measures for singular kernels, J. Comput. Appl. Math., 382 (2021), 113089, to appear, https://doi.org/10.1016/j.cam.2020.113089.
81.
C. R. Rao, Diversity and dissimilarity coefficients: A unified approach, Theoret. Population Biol., 21 (1982), pp. 24--43.
82.
C. R. Rao and T. K. Nayak, Cross entropy, dissimilarity measures and characterizations of quadratic entropy, IEEE Trans. Inform. Theory, 31 (1985), pp. 589--593.
83.
K. Ritter, G. W. Wasilkowski, and H. Woźniakowski, Multivariate integration and approximation for random fields satisfying Sacks-Ylvisaker conditions, Ann. Appl. Probab., 5 (1995), pp. 518--540.
84.
J. Sacks, W. J. Welch, T. J. Mitchell, and H. P. Wynn, Design and analysis of computer experiments, Statist. Sci., 4 (1989), pp. 409--435.
85.
T. J. Santner, B. J. Williams, and W. I. Notz, The Design and Analysis of Computer Experiments, Springer, Heidelberg, 2003.
86.
R. Schaback, Error estimates and condition numbers for radial basis function interpolation, Adv. Comput. Math., 3 (1995), pp. 251--264.
87.
R. Schaback, Native Hilbert spaces for radial basis functions I, in New Developments in Approximation Theory, Springer, New York, 1999, pp. 255--282.
88.
I. J. Schoenberg, Metric spaces and positive definite functions, Trans. Amer. Math. Soc., 44 (1938), pp. 522--536.
89.
S. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu, Equivalence of distance-based and RKHS-based statistics in hypothesis testing, Ann. Statist., 41 (2013), pp. 2263--2291.
90.
B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. G. Lanckriet, Hilbert space embeddings and metrics on probability measures, J. Mach. Learn. Res., 11 (2010), pp. 1517--1561.
91.
M. L. Stein, Interpolation of Spatial Data. Some Theory for Kriging, Springer, Heidelberg, 1999.
92.
I. Steinwart, D. Hush, and C. Scovel, An explicit description of the reproducing kernel Hilbert spaces of Gaussian RBF kernels, IEEE Trans. Inform. Theory, 52 (2006), pp. 4635--4643.
93.
Z. Szabó and B. Sriperumbudur, Characteristic and universal tensor product kernels, J. Mach. Learn. Res., 18 (2018), pp. 1--29.
94.
G. J. Székely and M. L. Rizzo, Energy statistics: A class of statistics based on distances, J. Statist. Plann. Inference, 143 (2013), pp. 1249--1272.
95.
E. Vazquez and J. Bect, Sequential search based on kriging: Convergence analysis of some algorithms, in Proceedings of the 58th World Statistics Congress of the ISI, Dublin, Ireland, 2011; preprint, https://arxiv.org/abs/1111.3866, 2011.
96.
H. Wackernagel, Multivariate Geostatistics. An Introduction with Applications, Springer, Berlin, 1998.
97.
H. P. Wynn, The sequential generation of $D$-optimum experimental designs, Ann. Math. Statist., 41 (1970), pp. 1655--1664.
Information & Authors
Information
Published In
SIAM/ASA Journal on Uncertainty Quantification
Pages: 959 - 1011
DOI: 10.1137/18M1210332
ISSN (online): 2166-2525
Copyright
© 2020, Society for Industrial and Applied Mathematics.
History
Submitted: 28 August 2018
Accepted: 3 May 2020
Published online: 11 August 2020
Keywords
MSC codes
Authors
Funding Information
Agence Nationale de la Recherche https://doi.org/10.13039/501100001665 : ANR-18-CE91-0007
Funding Information
Engineering and Physical Sciences Research Council https://doi.org/10.13039/501100000266 : EP/K032208/1
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Boundary-Layer Analysis of Repelling Particles Pushed to an Impenetrable BarrierSIAM Journal on Mathematical Analysis, Vol. 54, No. 2 | 15 March 2022
- Reproducing Kernel Hilbert Spaces, Polynomials, and the Classical Moment ProblemSIAM/ASA Journal on Uncertainty Quantification, Vol. 9, No. 4 | 4 November 2021