Abstract

The modern ability to collect vast quantities of data provides a challenge for parameter estimation. When posed as a nonlinear least squares problem fitting a model to data, the cost of each iteration grows linearly with the amount of data and with large data it can easily become too expensive to perform many iterations. Here we reduce the cost of each iteration by orthogonally projecting the data onto a low-dimensional subspace preserving the quality of the resulting parameter estimates. We provide results from both an optimization and a statistical perspective that show accurate parameter estimates are recovered when the subspace angles between this subspace and the range Jacobian of the model at the current iterate remain small. However, for this approach to reduce computational complexity, both the projected model and projected Jacobian must be computed inexpensively. This places a constraint on the pairs of models and subspaces for which this approach provides a computational speedup. Here we consider the exponential fitting problem projected onto the range of a Vandermonde matrix for which both the projected model and projected Jacobian can be computed in closed form using a generalized geometric sum formula. We further provide an inexpensive heuristic for choosing this Vandermonde matrix which ensures the subspace angles with the Jacobian remain small, and use this heuristic to update the subspace during optimization. Although the asymptotic cost still depends on the data dimension, the overall cost of solving this sequence of projected problems is significantly less expensive than the original.

Keywords

  1. exponential fitting
  2. harmonic estimation
  3. modal analysis
  4. spectral analysis
  5. parameter estimation
  6. nonlinear least squares
  7. dimension reduction
  8. experimental design

MSC codes

  1. 11L03
  2. 62K99
  3. 65K10
  4. 90C55

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
H. Barkhuijsen, R. de Beer, and D. van Ormondt, Improved algorithm for noniterative time-domain model fitting to exponentially damped magnetic resonance signals, J. Magn. Reson., 73 (1987), pp. 553--557, https://doi.org/10.1016/0022-2364(87)90023-0.
2.
D. P. Bertsekas, Incremental least squares methods and the extended Kalman filter, SIAM J. Optim., 6 (1996), pp. 807--822, https://doi.org/10.1137/S1052623494268522.
3.
D. P. Bertsekas, A new class of incremental gradient methods for least squares problems, SIAM J. Optim., 7 (1997), pp. 913--926, https://doi.org/10.1137/S1052623495287022.
4.
R. S. Dembo, S. C. Eisenstat, and T. Steihaug, Inexact Newton methods, SIAM J. Numer. Anal., 19 (1982), pp. 400--408, https://doi.org/10.1137/0719025.
5.
D. J. Ewins, Modal Testing: Theory and Practice, Research Studies Press, Letchworth, Hertfordshire, England, 1984.
6.
V. V. Fedorov, Theory of Optimal Experiments, Academic Press, New York, 1972.
7.
R. A. Fisher, On the mathematical foundations of theoretical statistics, Philos. Trans. R. Soc. Lond. Ser. A, 222 (1922), pp. 309--368.
8.
M. P. Friedlander and M. Schmidt, Hybrid deterministic-stochastic methods for data fitting, SIAM J. Sci. Comput., 34 (2012), pp. A1380--A1405, https://doi.org/10.1137/110830629.
9.
G. Golub and V. Pereyra, Separable nonlinear least squares: The variable projection method and its applications, Inverse Problems, 19 (2003), pp. R1--R26, https://doi.org/10.1088/0266-5611/19/2/201.
10.
G. H. Golub and V. Pereyra, The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate, SIAM J. Numer. Anal., 10 (1973), pp. 413--432, https://doi.org/10.1137/0710036.
11.
G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, Baltimore, 2013.
12.
P. C. Hansen, V. Pereyra, and G. Scherer, Least Squares Data Fitting with Applications, Johns Hopkins University Press, Baltimore, 2013.
13.
B. L. Ho and R. E. Kalman, Effective construction of linear state-variable models from input/output functions, Regelungstechnik, 14 (1966), pp. 545--592, https://doi.org/10.1524/auto.1966.14.112.545.
14.
R. A. Horn and C. R. Johnson, Matrix Analysis, Cambridge University Press, Cambridge, UK, 1985.
15.
R. A. Horn and C. R. Johnson, Topics in Matrix Analysis, Cambridge University Press, Cambridge, UK, 1991.
16.
Y. Hua and T. K. Sarkar, Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise, IEEE Trans. Acoust. Speech Signal Process., 38 (1990), pp. 814--824, https://doi.org/10.1109/29.56027.
17.
A. A. Istratov and O. F. Vyvenko, Exponential analysis in physical phenomena, Rev. Sci. Instrum., 70 (1999), pp. 1233--1257, https://doi.org/10.1063/1.1149581.
18.
S. K. Jain and S. N. Singh, Harmonics estimation in emerging power system: Key issues and challenges, Electr. Power Syst. Res., 81 (2011), pp. 1754--1766, https://doi.org/10.1016/j.epsr.2011.05.004.
19.
F. Johansson et al., mpmath: A Python library for arbitrary-precision floating-point arithmetic (version 0.19), 2014, http://mpmath.org.
20.
D. W. Kammler and R. J. McGlinn, A bibliography for approximation with exponential sums, J. Comput. Appl. Math., 4 (1978), pp. 167--173, https://doi.org/10.1016/0771-050X(78)90042-6.
21.
C. T. Kelley, Iterative Methods for Optimization, SIAM, Philadelphia, 1999, https://doi.org/10.1137/1.9781611970920.
22.
T. Laudadio, N. Mastronardi, L. Vanhamme, P. Van Hecke, and S. Van Huffel, Improved Lanczos algorithms for blackbox MRS data quantitation, J. Magn. Reson., 157 (2002), pp. 292--297, https://doi.org/10.1006/jmre.2002.2593.
23.
M. D. Macleod, Fast nearly ML estimation of the parameters of real or complex single tones or resolved multiple tones, IEEE Trans. Signal Process., 46 (1998), pp. 141--148, https://doi.org/10.1109/78.651200.
24.
V. B. Melas, Functional Approach to Optimal Experimental Design, Springer, New York, 2006.
25.
National Institute of Standards and Technology, Digital Library of Mathematical Functions, version 1.0.16, http://dlmf.nist.gov/, 2017.
26.
J. Nocedal and S. J. Wright, Numerical Optimization, Springer, New York, 2006.
27.
V. Pereyra and G. Scherer, Exponential Data Fitting and Its Applications, Bentham Science Publishers, Emirate of Sharjah, United Arab Emirates, 2010.
28.
R. Prony, Essai expérimental et analytique sur les lois de la dilatabilité et sur celles de la force expansive de la vapeur de l'eau et de la vapeur de l'alkool, à différentes températures, J. de l'Ecole Polytechnique, 1 (1795), pp. 24--76; English translation in [37, App. A].
29.
B. D. Rao, Perturbation analysis of an SVD-based linear prediction method for estimating the frequencies of multiple sinusoids, IEEE Trans. Acoust. Speech Signal Process., 36 (1988), pp. 1026--1035, https://doi.org/10.1109/29.1626.
30.
P. J. Schreier and L. L. Scharf, Statistical Signal Processing of Complex-Valued Data: Theory of Improper and Noncircular Signals, Cambridge University Press, Cambridge, UK, 2010.
31.
H. D. Scolnik, On the Solution of Nonlinear Least Squares Problems, Ph.D. thesis, University of Zurich, Zurich, 1970.
32.
G. A. F. Seber and C. J. Wild, Nonlinear Regression, John Wiley & Sons, New York, 1989.
33.
S. D. Silvey, Optimal Design, Chapman & Hall, London, 1980.
34.
P. Stoica and R. Moses, Introduction to Spectral Analysis, Prentice Hall, Upper Saddle River, NJ, 1997.
35.
G. Tang, B. N. Bhaskar, P. Shah, and B. Recht, Compressed sensing off the grid, IEEE Trans. Inform. Theory, 59 (2013), pp. 7465--7490, https://doi.org/10.1109/TIT.2013.2277451.
36.
D. Vandevoorde, A Fast Exponential Decomposition Algorithm and Its Applications to Structured Matrices, Ph.D. thesis, Rensselaer Polytechnic Institute, Troy, NY, 1996.
37.
L. Vanhamme, T. Sudin, P. Van Hecke, and S. Van Huffel, MR spectroscopy quantitation: A review of time-domain methods, NMR Biomed., 14 (2001), pp. 233--246, https://doi.org/10.1002/nbm.695.
38.
L. Vanhamme, A. van den Boogaart, and S. Van Huffel, Fast and accurate parameter estimation of noisy complex exponentials with use of prior knowledge, in Proceedings EUSIPCO-96, IEEE, Piscataway, NJ, 1996, pp. 1--4.
39.
L. Vanhamme, A. van den Boogaart, and S. Van Huffel, Improved method for accurate and efficient quantification of MRS data with use of prior knowledge, J. Magn. Reson., 129 (1997), pp. 35--43, https://doi.org/10.1006/jmre.1997.1244.
40.
S. Van Huffel, H. Chen, C. Decanniere, and P. Van Hecke, Algorithm for time-domain NMR data fitting based on total least squares, J. Magn. Reson. Ser. A, 110 (1994), pp. 228--237, https://doi.org/10.1006/jmra.1994.1209.
41.
S. J. Wright and J. N. Holt, An inexact Levenberg-Marquardt method for large sparse nonlinear least squares, J. Austral. Math. Soc. Ser. B, 26 (1985), pp. 387--403, https://doi.org/10.1017/S0334270000004604.

Information & Authors

Information

Published In

cover image SIAM Journal on Scientific Computing
SIAM Journal on Scientific Computing
Pages: A3107 - A3128
ISSN (online): 1095-7197

History

Submitted: 14 July 2016
Accepted: 19 September 2017
Published online: 21 December 2017

Keywords

  1. exponential fitting
  2. harmonic estimation
  3. modal analysis
  4. spectral analysis
  5. parameter estimation
  6. nonlinear least squares
  7. dimension reduction
  8. experimental design

MSC codes

  1. 11L03
  2. 62K99
  3. 65K10
  4. 90C55

Authors

Affiliations

Funding Information

Defense Advanced Research Projects Agency https://doi.org/10.13039/100000185
National Science Foundation https://doi.org/10.13039/100000001 : DMS-0240058, DMS-0739420

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

View Options

View options

PDF

View PDF

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.