Compressive Sensing with Cross-Validation and Stop-Sampling for Sparse Polynomial Chaos Expansions
Abstract
Compressive sensing is a powerful technique for recovering sparse solutions of underdetermined linear systems, which is often encountered in uncertainty quantification analysis of expensive and high-dimensional physical models. We perform numerical investigations employing several compressive sensing solvers that target the unconstrained LASSO formulation, with a focus on linear systems that arise in the construction of polynomial chaos expansions. With core solvers l1_ls, SpaRSA, CGIST, FPC_AS, and ADMM, we develop techniques to mitigate overfitting through an automated selection of regularization constant based on cross-validation, and a heuristic strategy to guide the stop-sampling decision. Practical recommendations on parameter settings for these techniques are provided and discussed. The overall method is applied to a series of numerical examples of increasing complexity, including large eddy simulations of supersonic turbulent jet-in-crossflow involving a 24-dimensional input. Through empirical phase-transition diagrams and convergence plots, we illustrate sparse recovery performance under structures induced by polynomial chaos, accuracy, and computational trade-offs between polynomial bases of different degrees, and practicability of conducting compressive sensing for a realistic, high-dimensional physical application. Across test cases studied in this paper, we find ADMM to have demonstrated empirical advantages through consistent lower errors and faster computational times.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
H. Akaike, A new look at the statistical model identification, IEEE Trans. Automat. Contr., 19 (1974), pp. 716--723, https://doi.org/10.1109/TAC.1974.1100705.
2.
S. Babacan, R. Molina, and A. Katsaggelos, Bayesian compressive sensing using Laplace priors, IEEE Trans. Image Process., 19 (2010), pp. 53--63, https://doi.org/10.1109/TIP.2009.2032894.
3.
V. Barthelmann, E. Novak, and K. Ritter, High dimensional polynomial interpolation on sparse grids, Adv. Comput. Math., 12 (2000), pp. 273--288, https://doi.org/10.1023/A:1018977404843.
4.
G. Blatman and B. Sudret, Adaptive sparse polynomial chaos expansion based on least angle regression, J. Comput. Phys., 230 (2011), pp. 2345--2367, https://doi.org/10.1016/j.jcp.2010.12.021.
5.
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends in Machine Learning, 3 (2010), pp. 1--122, https://doi.org/10.1561/2200000016.
6.
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Solve LASSO via ADMM, 2011, https://web.stanford.edu/~boyd/papers/admm/lasso/lasso.html (accessed 2017-07-02).
7.
E. J. Candès, J. Romberg, and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2006), pp. 489--509, https://doi.org/10.1109/TIT.2005.862083.
8.
S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM Rev., 43 (2001), pp. 129--159, https://doi.org/10.1137/S003614450037906X.
9.
G. Davis, S. Mallat, and M. Avellaneda, Adaptive greedy approximations, Constr. Approx., 13 (1997), pp. 57--98, https://doi.org/10.1007/BF02678430.
10.
D. L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), pp. 1289--1306, https://doi.org/10.1109/Tit.2006.871582.
11.
D. L. Donoho, For most large underdetermined systems of linear equations the minimal $\ell_1$-norm solution is also the sparsest solution, Comm. Pure Appl. Math., 59 (2006), pp. 797--829, https://doi.org/10.1002/cpa.20132.
12.
D. L. Donoho and J. Tanner, Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing, Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 367 (2009), pp. 4273--4293, https://doi.org/10.1098/rsta.2009.0152.
13.
D. L. Donoho and J. Tanner, Precise undersampling theorems, Proc. IEEE, 98 (2010), pp. 913--924, https://doi.org/10.1109/JPROC.2010.2045630.
14.
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Ann. Statist., 32 (2004), pp. 407--499, https://doi.org/10.1214/009053604000000067.
15.
M. S. Eldred, L. W. T. Ng, M. F. Barone, and S. P. Domino, Multifidelity uncertainty quantification using spectral stochastic discrepancy models, in Handbook of Uncertainty Quantification, Springer, Cham, 2015, pp. 1--45, https://doi.org/10.1007/978-3-319-11259-6_25-1.
16.
O. G. Ernst, A. Mugler, H.-J. Starkloff, and E. Ullmann, On the convergence of generalized polynomial chaos expansions, ESAIM Math. Model. Numer. Anal., 46 (2012), pp. 317--339, https://doi.org/10.1051/m2an/2011045.
17.
N. Fajraoui, S. Marelli, and B. Sudret, On Optimal Experimental Designs for Sparse Polynomial Chaos Expansions, preprint, https://arxiv.org/abs/1703.05312, 2017.
18.
T. Gerstner and M. Griebel, Numerical integration using sparse grids, Numer. Algorithms, 18 (1998), pp. 209--232, https://doi.org/10.1023/A:1019129717644.
19.
T. Gerstner and M. Griebel, Dimension-adaptive tensor-product quadrature, Computing, 71 (2003), pp. 65--87, https://doi.org/10.1007/s00607-003-0015-5.
20.
R. G. Ghanem and P. D. Spanos, Stochastic Finite Elements: A Spectral Approach, 1st ed., Springer-Verlag, New York, 1991.
21.
T. Goldstein and S. Setzer, High-order Methods for Basis Pursuit, Tech. Report CAM Report 10-41, University of California, Los Angeles, 2010.
22.
T. Goldstein and S. Setzer, CGIST: A Fast and Exact Solver for L1 Minimization, 2011, http://tag7.web.rice.edu/CGIST.html (accessed 2017-07-02).
23.
J. H. Halton, On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals, Numer. Math., 2 (1960), pp. 84--90, https://doi.org/10.1007/BF01386213.
24.
J. Hampton and A. Doostan, Compressive sampling of polynomial chaos expansions: Convergence analysis and sampling strategies, J. Comput. Phys., 280 (2015), pp. 363--386, https://doi.org/10.1016/j.jcp.2014.09.019.
25.
N. E. Hass, K. F. Cabell, and A. M. Storch, HIFiRE Direct-Connect Rig (HDCR) Phase I Ground Test Results from the NASA Langley Arc-Heated Scramjet Test Facility, Tech. Report CR-2010-002215, NASA, 2010.
26.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, New York, 2009.
27.
X. Huan, C. Safta, K. Sargsyan, G. Geraci, M. S. Eldred, Z. P. Vane, G. Lacaze, J. C. Oefelein, and H. N. Najm, Global sensitivity analysis and estimation of model error, toward uncertainty quantification in scramjet computations, AIAA J., 56 (2018), pp. 1170--1184, https://doi.org/10.2514/1.J056278.
28.
J. D. Jakeman, M. S. Eldred, and K. Sargsyan, Enhancing $\ell_1$-minimization estimates of polynomial chaos expansions using basis selection, J. Comput. Phys., 289 (2015), pp. 18--34, https://doi.org/10.1016/j.jcp.2015.02.025.
29.
J. D. Jakeman, A. Narayan, and T. Zhou, A generalized sampling and preconditioning scheme for sparse approximation of polynomial chaos expansions, SIAM J. Sci. Comput., 39 (2017), pp. A1114--A1144, https://doi.org/10.1137/16M1063885.
30.
S. Ji, Y. Xue, and L. Carin, Bayesian compressive sensing, IEEE Trans. Signal Process., 56 (2008), pp. 2346--2356, https://doi.org/10.1109/TSP.2007.914345.
31.
R. E. Kass and A. E. Raftery, Bayes factor, J. Amer. Statist. Assoc., 90 (1995), pp. 773--795, https://doi.org/10.2307/2291091.
32.
S.-J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, An interior point method for large-scale $\ell_1$-regularized least squares, IEEE J. Sel. Topics Signal Process., 1 (2007), pp. 606--617, https://doi.org/10.1109/JSTSP.2007.910971.
33.
K. Koh, S.-J. Kim, and S. Boyd, Simple Matlab Solver for l$1$-regularized Least Squares Problems, beta version, 2008, https://stanford.edu/~boyd/l1_ls/ (accessed 2017-07-02).
34.
O. P. Le Maître and O. M. Knio, Spectral Methods for Uncertainty Quantification: With Applications to Computational Fluid Dynamics, Springer Netherlands, Houten, Netherlands, 2010.
35.
K. Mahesh, The interaction of jets with crossflow, Annu. Rev. Fluid Mech., 45 (2013), pp. 379--407, https://doi.org/10.1146/annurev-fluid-120710-101115.
36.
D. M. Malioutov, S. R. Sanghavi, and A. S. Willsky, Sequential compressed sensing, IEEE J. Sel. Topics Signal Process., 4 (2010), pp. 435--444, https://doi.org/10.1109/JSTSP.2009.2038211.
37.
H. N. Najm, Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics, Annu. Rev. Fluid Mech., 41 (2009), pp. 35--52, https://doi.org/10.1146/annurev.fluid.010908.165248.
38.
J. C. Oefelein, Simulation and Analysis of Turbulent Multiphase Combustion Processes at High Pressures, Ph.D. thesis, The Pennsylvania State University, State College, PA 1997.
39.
J. C. Oefelein, Large eddy simulation of turbulent combustion processes in propulsion and power systems, Prog. Aerosp. Sci., 42 (2006), pp. 2--37, https://doi.org/10.1016/j.paerosci.2006.02.001.
40.
J. Peng, J. Hampton, and A. Doostan, A weighted $\ell_1$-minimization approach for sparse polynomial chaos expansions, J. Comput. Phys., 267 (2014), pp. 92--111, https://doi.org/10.1016/j.jcp.2014.02.024.
41.
H. Rauhut and R. Ward, Sparse Legendre expansions via $\ell_1$-minimization, J. Approx. Theory, 164 (2012), pp. 517--533, https://doi.org/10.1016/j.jat.2012.01.008.
42.
K. Sargsyan, C. Safta, H. N. Najm, B. J. Debusschere, D. Ricciuto, and P. Thornton, Dimensionality reduction for complex models via Bayesian compressive sensing, Int. J. Uncertain. Quantif., 4 (2014), pp. 63--93, https://doi.org/10.1615/Int.J.UncertaintyQuantification.2013006821.
43.
G. Schwarz, Estimating the dimension of a model, Ann. Statist., 6 (1978), pp. 461--464, https://doi.org/10.1214/aos/1176344136.
44.
J. Tanner, Phase Transitions of the Regular Polytopes and Cone, 2012, https://people.maths.ox.ac.uk/tanner/polytopes.shtml (accessed 2017-07-03).
45.
E. van den Berg and M. P. Friedlander, Probing the Pareto frontier for basis pursuit solutions, SIAM J. Sci. Comput., 31 (2008), pp. 890--912, https://doi.org/10.1137/080714488.
46.
L. Wasserman, Bayesian model selection and model averaging, J. Math. Psych., 44 (2000), pp. 92--107, https://doi.org/10.1006/jmps.1999.1278.
47.
Z. Wen, W. Yin, D. Goldfarb, and Y. Zhang, A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization, and continuation, SIAM J. Sci. Comput., 32 (2010), pp. 1832--1857, https://doi.org/10.1137/090747695.
48.
S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, SpaRSA: Sparse Reconstruction by Separable Approximation, version 2.0, 2009, http://www.lx.it.pt/~mtf/SpaRSA/ (accessed 2017-07-02).
49.
S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, Sparse reconstruction by separable approximation, IEEE Trans. Signal Process., 57 (2009), pp. 2479--2493, https://doi.org/10.1109/TSP.2009.2016892.
50.
D. Xiu, Fast numerical methods for stochastic computations: A review, Commun. Comput. Phys., 5 (2009), pp. 242--272.
51.
D. Xiu and G. E. Karniadakis, The Wiener--Askey polynomial chaos for stochastic differential equations, SIAM J. Sci. Comput., 24 (2002), pp. 619--644, https://doi.org/10.1137/S1064827501387826.
52.
W. Yin and Z. Wen, FPC_AS: A MATLAB Solver for L1-Regularization Problems, version 1.21, 2010, http://www.caam.rice.edu/~optimization/L1/FPC_AS/ (accessed 2017-07-02).
Information & Authors
Information
Published In
SIAM/ASA Journal on Uncertainty Quantification
Pages: 907 - 936
DOI: 10.1137/17M1141096
ISSN (online): 2166-2525
Copyright
© 2018, Society for Industrial and Applied Mathematics.
History
Submitted: 28 July 2017
Accepted: 27 March 2018
Published online: 19 June 2018
Keywords
MSC codes
Authors
Funding Information
Defense Advanced Research Projects Agency https://doi.org/10.13039/100000185
Funding Information
National Nuclear Security Administration https://doi.org/10.13039/100006168 : DE-NA-0003525
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Uncertainty Quantification Framework for Predicting Material Response with Large Number of Parameters: Application to Creep Prediction in Ferritic-Martensitic Steels Using Combined Crystal Plasticity and Grain Boundary ModelsIntegrating Materials and Manufacturing Innovation, Vol. 11, No. 4 | 28 September 2022
- An ensemble Synthetic Eddy Method for accurate treatment of inhomogeneous turbulenceComputers & Fluids, Vol. 248 | 1 Nov 2022
- Multifidelity Model Calibration in Structural Dynamics Using Stochastic Variational Inference on ManifoldsEntropy, Vol. 24, No. 9 | 13 September 2022
- Uncertainty quantification and reliability analysis by an adaptive sparse Bayesian inference based PCE modelEngineering with Computers, Vol. 38, No. S2 | 28 January 2021
- Prospects and Limitations of Predicting Fuel Ignition Properties from Low-Temperature Speciation DataEnergy & Fuels, Vol. 36, No. 6 | 8 March 2022
- Power module heat sink design optimization with ensembles of data-driven polynomial chaos surrogate modelse-Prime - Advances in Electrical Engineering, Electronics and Energy, Vol. 2 | 1 Jan 2022
- A hybrid sequential sampling strategy for sparse polynomial chaos expansion based on compressive sampling and Bayesian experimental designComputer Methods in Applied Mechanics and Engineering, Vol. 386 | 1 Dec 2021
- A generalized multi-fidelity simulation method using sparse polynomial chaos expansionJournal of Computational and Applied Mathematics, Vol. 397 | 1 Dec 2021
- Analyzing cross-validation in compressed sensing with Poisson noiseSignal Processing, Vol. 182 | 1 May 2021
- Accelerating Ultrafast Spectroscopy with Compressive SensingPhysical Review Applied, Vol. 15, No. 2 | 15 February 2021
- Sparse Polynomial Chaos Expansions: Literature Survey and BenchmarkSIAM/ASA Journal on Uncertainty Quantification, Vol. 9, No. 2 | 13 May 2021
- Sparse Polynomial Chaos expansions using variational relevance vector machinesJournal of Computational Physics, Vol. 416 | 1 Sep 2020
- A Method for Under-Sampling Modulation Pattern Recognition in Satellite CommunicationCommunications, Signal Processing, and Systems | 14 August 2019
- A new surrogate modeling method combining polynomial chaos expansion and Gaussian kernel in a sparse Bayesian learning frameworkInternational Journal for Numerical Methods in Engineering, Vol. 120, No. 4 | 14 July 2019
- Uncertainty Study of Periodic-Grating Wideband Filters With Sparse Polynomial-Chaos ExpansionsIEEE Photonics Technology Letters, Vol. 31, No. 18 | 15 Sep 2019
- Compressive sensing adaptation for polynomial chaos expansionsJournal of Computational Physics, Vol. 380 | 1 Mar 2019
- Uncertainty Propagation Using Conditional Random Fields in Large-Eddy Simulations of Scramjet ComputationsAIAA Scitech 2019 Forum | 6 January 2019
- Technique for Signal Noise Reduction based on Sparse Representation2018 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC) | 1 Nov 2018