Search
Search
Quick Search in Journals
Quick Search anywhere
Quick Search anywhere
Quick Search anywhere
Quick Search anywhere
Advanced Search
Skip main navigation
Previous article
Next article

Atomic Decomposition by Basis Pursuit

https://doi.org/10.1137/S1064827596304010

Abstract

The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), Matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).

Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation denoising, and multiscale edge denoising.

BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.

  • [1]  R. E. Bixby, Commentary: Progress in linear programming, ORSA J. Comput., 6 (1994), pp. 15–22. 9cc OJCOE3 0899-1499 ORSA J. Comput. CrossrefGoogle Scholar

  • [2]  Peter Bloomfield and , William Steiger, Least absolute deviations, Progress in Probability and Statistics, Vol. 6, Birkhäuser Boston Inc., 1983xiv+349, Theory, applications, and algorithms 86a:62005 Google Scholar

  • [3]  J. Buckheit and D. L. Donoho, WaveLab and reproducible research, in Wavelets and Statistics, A. Antoniadis, Ed., Springer, Berlin, New York, 1995. Google Scholar

  • [4]  S. S. Chen, Basis Pursuit, Ph.D. Thesis, Department of Statistics, Stanford University, Stanford, CA, 1995 (http://www‐stat.stanford.edu/∼schen/). Google Scholar

  • [5]  S. Chen, , S. Billings and , W. Luo, Orthogonal least squares methods and their application to nonlinear system identification, Internat. J. Control, 50 (1989), 1873–1896 90m:93127 CrossrefISIGoogle Scholar

  • [6]  Ronald Coifman and , Yves Meyer, Remarques sur l’analyse de Fourier à fenêtre, C. R. Acad. Sci. Paris Sér. I Math., 312 (1991), 259–261 92k:42042 Google Scholar

  • [7]  R. R. Coifman and  and M. V. Wickerhauser, Entropy‐based algorithms for best‐basis selection, IEEE Trans. Inform. Theory, 38 (1992), pp. 713–718. iet IETTAW 0018-9448 IEEE Trans. Inf. Theory CrossrefISIGoogle Scholar

  • [8]  G. B. Dantzig, Linear Programming and Extensions, Princeton University Press, Princeton, NJ, 1963. Google Scholar

  • [9]  Ingrid Daubechies, Time‐frequency localization operators: a geometric phase space approach, IEEE Trans. Inform. Theory, 34 (1988), 605–612 10.1109/18.9761 966733 CrossrefISIGoogle Scholar

  • [10]  Ingrid Daubechies, Ten lectures on wavelets, CBMS‐NSF Regional Conference Series in Applied Mathematics, Vol. 61, Society for Industrial and Applied Mathematics (SIAM), 1992xx+357 93e:42045 LinkGoogle Scholar

  • [11]  G. Davis, , S. Mallat and , and Z. Zhang, Adaptive time‐frequency decompositions, Optical Engrg., 33 (1994), pp. 2183–2191. gar OPEGAR 0091-3286 Opt. Eng. CrossrefISIGoogle Scholar

  • [12]  R. DeVore and , V. Temlyakov, Some remarks on greedy algorithms, Adv. Comput. Math., 5 (1996), 173–187 97g:41029 CrossrefISIGoogle Scholar

  • [13]  David Donoho, De‐noising by soft‐thresholding, IEEE Trans. Inform. Theory, 41 (1995), 613–627 10.1109/18.382009 96b:94002 CrossrefISIGoogle Scholar

  • [14]  David Donoho and , Iain Johnstone, Ideal denoising in an orthonormal basis chosen from a library of bases, C. R. Acad. Sci. Paris Sér. I Math., 319 (1994), 1317–1322 95i:42022 Google Scholar

  • [15]  D. L. Donoho and I. M. Johnstone, Empirical Atomic Decomposition, manuscript, 1995. Google Scholar

  • [16]  David Donoho, , Iain Johnstone, , Gérard Kerkyacharian and , Dominique Picard, Wavelet shrinkage: asymptopia?, J. Roy. Statist. Soc. Ser. B, 57 (1995), 301–369, With discussion and a reply by the authors 96g:62068 Google Scholar

  • [17]  P. E. Gill, W. Murray, D. B. Ponceleón, and M. A. Saunders, Solving Reduced KKT Systems in Barrier Methods for Linear and Quadratic Programming, Tech. report SOL 91‐7, Stanford University, Stanford, CA, July 1991. Google Scholar

  • [18]  Philip Gill, , Walter Murray and , Margaret Wright, Numerical linear algebra and optimization. Vol. 1, Addison‐Wesley Publishing Company Advanced Book Program, 1991xx+426 92b:65001 Google Scholar

  • [19]  G. Golub and C. V Loan, Matrix Computations, 2nd ed. Johns Hopkins University Press, Baltimore, MD, 1989. Google Scholar

  • [20]  N. Karmarkar, A new polynomial‐time algorithm for linear programming, Combinatorica, 4 (1984), 373–395 86i:90072 CrossrefISIGoogle Scholar

  • [21]  Masakazu Kojima, , Shinji Mizuno and , Akiko Yoshise, A primal‐dual interior point algorithm for linear programming, Springer, New York, 1989, 29–47 90k:90093 Google Scholar

  • [22]  Y. Li and  and F. Santosa, A computational algorithm for minimizing total variation in image restoration, IEEE Trans. Image Proc., 5 (1996), pp. 987–995. iei IIPRE4 1057-7149 IEEE Trans. Image Process. CrossrefISIGoogle Scholar

  • [23]  Irvin Lustig, , Roy Marsten and , David Shanno, Interior point methods for linear programming: computational state of the art, ORSA J. Comput., 6 (1994), 1–14 1261376 CrossrefGoogle Scholar

  • [24]  Stephane Mallat and , Wen Hwang, Singularity detection and processing with wavelets, IEEE Trans. Inform. Theory, 38 (1992), 617–643 10.1109/18.119727 93b:94007 CrossrefISIGoogle Scholar

  • [25]  S. Mallat and  and Z. Zhang, Matching pursuit in a time‐frequency dictionary, IEEE Trans. Signal Proc., 41 (1993), pp. 3397–3415. itl ITPRED 1053-587X IEEE Trans. Signal Process. CrossrefISIGoogle Scholar

  • [26]  Lokenath Debnath, Wavelet transforms and their applications, Proc. Indian Nat. Sci. Acad. Part A, 64 (1998), 685–713 2000g:42048 Google Scholar

  • [27]  Nimrod Megiddo, On finding primal‐ and dual‐optimal bases, ORSA J. Comput., 3 (1991), 63–65 93d:90033 CrossrefGoogle Scholar

  • [28]  Yves Meyer, Ondelettes sur l’intervalle, Rev. Mat. Iberoamericana, 7 (1991), 115–133 92m:42001 CrossrefGoogle Scholar

  • [29]  Y. Meyer, Wavelets: Algorithms and Applications, SIAM, Philadelphia, 1993. Google Scholar

  • [30]  Y. Nesterov and A. Nemirovskii, Interior‐Point Polynomial Algorithms in Convex Programming, SIAM, Philadelphia, 1994. Google Scholar

  • [31]  L. J. Rudin, , S. Osher and , and E. Fatemi, Nonlinear total‐variation‐based noise removal algorithms, Physica D, 60 (1992), pp. 259–268. phd PDNPDT 0167-2789 Physica D CrossrefISIGoogle Scholar

  • [32]  Christopher Paige and , Michael Saunders, LSQR: an algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software, 8 (1982), 43–71 10.1145/355984.355989 83f:65048 CrossrefISIGoogle Scholar

  • [33]  C. C. Paige and  and M. A. Saunders, Algorithm 583; LSQR: Sparse linear equations and least‐squares problems, ACM Trans. Math. Software, 8 (1982), pp. 195–209. att ACMSCU 0098-3500 ACM Trans. Math. Softw. CrossrefISIGoogle Scholar

  • [34]  Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, in Proc. 27th Asilomar Conference on Signals, Systems and Computers, A. Singh, ed., Los Alamitos, CA, USA IEEE Comput. Soc. Press, 1993. Google Scholar

  • [35]  Shie Qian and  and Dapang Chen, Signal representation using adaptive normalized Gaussian functions, Signal Process., 36 (1994), pp. 1–11. spq SPRODR 0165-1684 Signal Process. CrossrefISIGoogle Scholar

  • [36]  M. A. Saunders, Commentary: Major Cholesky would feel proud, ORSA J. Comput., 6 (1994), pp. 23–27. 9cc OJCOE3 0899-1499 ORSA J. Comput. CrossrefGoogle Scholar

  • [37]  Eero Simoncelli, , William Freeman, , Edward Adelson and , David Heeger, Shiftable multiscale transforms, IEEE Trans. Inform. Theory, 38 (1992), 587–607 10.1109/18.119725 1162216 CrossrefISIGoogle Scholar

  • [38]  M. J. Todd, Commentary: Theory and practice for interior point methods, ORSA J. Comput., 6 (1994), pp. 28–31. 9cc OJCOE3 0899-1499 ORSA J. Comput. CrossrefGoogle Scholar

  • [39]  R. J. Vanderbei, Commentary: Interior point methods: Algorithms and formulations, ORSA J. Comput., 6 (1994), pp. 32–34. 9cc OJCOE3 0899-1499 ORSA J. Comput. CrossrefGoogle Scholar

  • [40]  L. Villemoes, Best approximation with Walsh atoms, Constr. Approx., 13 (1997), 329–355 10.1007/s003659900046 99f:42069 CrossrefISIGoogle Scholar

Keywords

Previous article
Next article