Search
Search
Quick Search in Journals
Quick Search anywhere
Quick Search anywhere
Quick Search anywhere
Quick Search anywhere
Advanced Search
Skip main navigation
Previous article
Next article

A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems

Authors: Amir Beck and Marc TeboulleAuthors Info & Affiliations
https://doi.org/10.1137/080716542
Get Access
BibTeX
Tools

Abstract

We consider the class of iterative shrinkage-thresholding algorithms (ISTA) for solving linear inverse problems arising in signal/image processing. This class of methods, which can be viewed as an extension of the classical gradient algorithm, is attractive due to its simplicity and thus is adequate for solving large-scale problems even with dense matrix data. However, such methods are also known to converge quite slowly. In this paper we present a new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically. Initial promising numerical results for wavelet-based image deblurring demonstrate the capabilities of FISTA which is shown to be faster than ISTA by several orders of magnitude.

MSC codes

  1. 90C25
  2. 90C06
  3. 65F22

Keywords

  1. iterative shrinkage-thresholding algorithm
  2. deconvolution
  3. linear inverse problem
  4. least squares and $l_1$ regularization problems
  5. optimal gradient method
  6. global rate of convergence
  7. two-step iterative algorithms
  8. image deblurring

Get full access to this article

View all available purchase options and get full access to this article.

Get Access

References

1.
A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications, MPS/SIAM Ser. Optim., SIAM, Philadelphia, 2001.
Google Scholar
2.
D. P. Bertsekas, Nonlinear Programming, 2nd ed., Athena Scientific, Belmont, MA, 1999.
Google Scholar
3.
J. Bioucas-Dias and M. Figueiredo, A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration, IEEE Trans. Image Process., 16 (2007), pp. 2992–3004.
Crossref
Web of Science
Google Scholar
4.
A. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.
Google Scholar
5.
K. Bredies and D. Lorenz, Iterative Soft-Thresholding Converges Linearly, Technical report, 2008; available online at http://arxiv.org/abs/0709.1598v3.
Google Scholar
6.
R. J. Bruck, On the weak convergence of an ergodic iteration for the solution of variational inequalities for monotone operators in Hilbert space, J. Math. Anal. Appl., 61 (1977), pp. 159–164.
Crossref
Web of Science
Google Scholar
7.
A. Chambolle, R. A. DeVore, N. Y. Lee, and B. J. Lucier, Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage, IEEE Trans. Image Process., 7 (1998), pp. 319–335.
Crossref
Web of Science
Google Scholar
8.
S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM J. Sci. Comput., 20 (1998), pp. 33–61.
Crossref
Web of Science
Google Scholar
9.
P. L. Combettes and V. R. Wajs, Signal recovery by proximal forward-backward splitting, Multiscale Model. Simul., 4 (2005), pp. 1168–1200.
Crossref
Web of Science
Google Scholar
10.
I. Daubechies, M. Defrise, and C. D. Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint, Comm. Pure Appl. Math., 57 (2004), pp. 1413–1457.
Crossref
Web of Science
Google Scholar
11.
D. L. Donoho and I. M. Johnstone, Adapting to unknown smoothness via wavelet shrinkage, J. Amer. Statist. Assoc., 90 (1995), pp. 1200–1224.
Crossref
Web of Science
Google Scholar
12.
M. Elad, B. Matalon, and M. Zibulevsky, Subspace optimization methods for linear least squares with non-quadratic regularization, Appl. Comput. Harmon. Anal., 23 (2007), pp. 346–367.
Crossref
Web of Science
Google Scholar
13.
H. W. Engl, M. Hanke, and A. Neubauer, Regularization of Inverse Problems, Math. Appl. 375, Kluwer Academic Publishers Group, Dordrecht, The Netherlands, 1996.
Google Scholar
14.
F. Facchinei and J.-S. Pang, Finite-Dimensional Variational Inequalities and Complementarity Problems, Vol. II, Springer Ser. Oper. Res., Springer-Verlag, New York, 2003.
Google Scholar
15.
M. A. T. Figueiredo and R. D. Nowak, An EM algorithm for wavelet-based image restoration, IEEE Trans. Image Process., 12 (2003), pp. 906–916.
Crossref
Web of Science
Google Scholar
16.
M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems, IEEE J. Sel. Top. Signal Process., 1 (2007), pp. 586–597.
Crossref
Web of Science
Google Scholar
17.
G. H. Golub, P. C. Hansen, and D. P. O'Leary, Tikhonov regularization and total least squares, SIAM J. Matrix Anal. Appl., 21 (1999), pp. 185–194.
Crossref
Web of Science
Google Scholar
18.
E. Hale, W. Yin, and Y. Zhang, A Fixed-Point Continuation Method for $l_1$-Regularized Minimization with Applications to Compressed Sensing, CAAM Technical report TR07-07, Rice University, Houston, TX, 2007.
Google Scholar
19.
P. C. Hansen and D. P. O'Leary, The use of the L-curve in the regularization of discrete ill-posed problems, SIAM J. Sci. Comput., 14 (1993), pp. 1487–1503.
Crossref
Web of Science
Google Scholar
20.
P. C. Hansen, Regularization tools: A MATLAB package for analysis and solution of discrete ill-posed problems, Numer. Algorithms, 6 (1994), pp. 1–35.
Crossref
Google Scholar
21.
P. C. Hansen, Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion, SIAM, Philadelphia, 1997.
Google Scholar
22.
P. C. Hansen, J. G. Nagy, and D. P. O'Leary, Deblurring Images: Matrices, Spectra, and Filtering, Fundam. Algorithms 3, SIAM, Philadelphia, 2006.
Google Scholar
23.
E. S. Levitin and B. T. Polyak, Constrained minimization methods, Comput. Math. Math. Phys., 6 (1966), pp. 787–823.
Crossref
Google Scholar
24.
B. Martinet, Régularisation d'inéquations variationnelles par approximations successives, Rev. Française Informat. Recherche Opérationnelle, 4 (1970), pp. 154–158.
Google Scholar
25.
P. Moulin and J. Liu, Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors, IEEE Trans. Inform. Theory, 45 (1999), pp. 909–919.
Crossref
Web of Science
Google Scholar
26.
A. S. Nemirovsky and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Interscience Series in Discrete Mathematics, John Wiley & Sons, New York, 1983.
Google Scholar
27.
Y. E. Nesterov, A method for solving the convex programming problem with convergence rate $O(1/k\sp{2})$, Dokl. Akad. Nauk SSSR, 269 (1983), pp. 543–547 (in Russian).
Google Scholar
28.
Y. E. Nesterov, Gradient Methods for Minimizing Composite Objective Function, CORE report, 2007; available at http://www.ecore.be/DPs/dp1191313936.pdf.
Google Scholar
29.
J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Classics Appl. Math. 30, SIAM, Philadelphia, 2000.
Google Scholar
30.
G. B. Passty, Ergodic convergence to a zero of the sum of monotone operators in Hilbert space, J. Math. Anal. Appl., 72 (1979), pp. 383–390.
Crossref
Web of Science
Google Scholar
31.
B. T. Polyak, Introduction to Optimization, Translations Series in Mathematics and Engineering, Optimization Software, Publications Division, New York, 1987.
Google Scholar
32.
J. L. Starck, D. L. Donoho, and E. J. Candès, Astronomical image representation by the curevelet transform, Astron. Astrophys., 398 (2003), pp. 785–800.
Crossref
Web of Science
Google Scholar
33.
A. N. Tikhonov and V. Y. Arsenin, Solution of Ill-Posed Problems, V. H. Winston, Washington, DC, 1977.
Google Scholar
34.
C. Vonesch and M. Unser, Fast iterative thresholding algorithm for wavelet-regularized deconvolution, in Proceedings of the SPIE Optics and Photonics 2007 Conference on Mathematical Methods: Wavelet XII, Vol. 6701, San Diego, CA, 2007, pp. 1–5.
Google Scholar
35.
S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, Sparse reconstruction by separable approximation, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2008), 2008, pp. 3373–3376.
Google Scholar

Information & Authors

Information

Published In

cover image SIAM Journal on Imaging Sciences
SIAM Journal on Imaging Sciences
Volume 2 • Issue 1 • January 2009
Pages: 183 - 202
DOI: 10.1137/080716542
ISSN (online): 1936-4954

Copyright

Copyright © 2009 Society for Industrial and Applied Mathematics.

History

Submitted: 25 February 2008
Accepted: 23 October 2008
Published online: 4 March 2009

MSC codes

  1. 90C25
  2. 90C06
  3. 65F22

Keywords

  1. iterative shrinkage-thresholding algorithm
  2. deconvolution
  3. linear inverse problem
  4. least squares and $l_1$ regularization problems
  5. optimal gradient method
  6. global rate of convergence
  7. two-step iterative algorithms
  8. image deblurring

Authors

Affiliations

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By