From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images

A full-rank matrix ${\bf A}\in \mathbb{R}^{n\times m}$ with $n<m$ generates an underdetermined system of linear equations ${\bf Ax} = {\bf b}$ having infinitely many solutions. Suppose we seek the sparsest solution, i.e., the one with the fewest nonzero entries. Can it ever be unique? If so, when? As optimization of sparsity is combinatorial in nature, are there efficient methods for finding the sparsest solution? These questions have been answered positively and constructively in recent years, exposing a wide variety of surprising phenomena, in particular the existence of easily verifiable conditions under which optimally sparse solutions can be found by concrete, effective computational methods.

Such theoretical results inspire a bold perspective on some important practical problems in signal and image processing. Several well-known signal and image processing problems can be cast as demanding solutions of undetermined systems of equations. Such problems have previously seemed, to many, intractable, but there is considerable evidence that these problems often have sparse solutions. Hence, advances in finding sparse solutions to underdetermined systems have energized research on such signal and image processing problems—to striking effect.

In this paper we review the theoretical results on sparse solutions of linear systems, empirical results on sparse modeling of signals and images, and recent applications in inverse problems and compression in image processing. This work lies at the intersection of signal processing and applied mathematics, and arose initially from the wavelets and harmonic analysis research communities. The aim of this paper is to introduce a few key notions and applications connected to sparsity, targeting newcomers interested in either the mathematical aspects of this area or its applications.

  • [1]  M. Aharon, M. Elad, and A.M. Bruckstein, K-SVD and its non-negative variant for dictionary design, in Wavelet Applications in Signal and Image Processing XI, Proc. SPIE Conf. 5914, SPIE, Bellingham, WA, 2005, pp. 11.1–11.13. Google Scholar

  • [2]  M. Aharon, M. Elad and , and A.M. Bruckstein, On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them, J. Linear Algebra Appl., 416 (2006), pp. 48–67. CrossrefISIGoogle Scholar

  • [3]  M. Aharon, M. Elad and , and A.M. Bruckstein, K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation, IEEE Trans. Signal Process., 54 (2006), pp. 4311–4322. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [4]  A. Barron, A. Cohen, W. Dahmen and , and R.A. DeVore, Approximation and learning by greedy algorithms, Ann. Statist., 36 (2008), pp. 64–94. ASTSC7 0090-5364 CrossrefISIGoogle Scholar

  • [5]  J. Bioucas-Dias, Bayesian wavelet-based image deconvolution: A GEM algorithm exploiting a class of heavy-tailed priors, IEEE Trans. Image Process., 15 (2006), pp. 937–951. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [6]  A. Björner, M. Las Vergnas, B. Sturmfels, N. White, and G. M. Ziegler, Oriented Matroids, Encyclopedia Math. Appl. 46, Cambridge University Press, Cambridge, UK, 1993. Google Scholar

  • [7]  J. Bobin, Y. Moudden, J.-L. Starck and , and M. Elad, Morphological diversity and source separation, IEEE Signal Process. Lett., 13 (2006), pp. 409–412. ISPLEM 1070-9908 CrossrefISIGoogle Scholar

  • [8]  J. Bobin, J.-L. Starck, M.J. Fadili, and Y. Moudden, SZ and CMB reconstruction using generalized morphological component analysis, Statist. Method., to appear. Google Scholar

  • [9]  O. Bryt and  and M. Elad, Compression of facial images using the K-SVD algorithm, J.Vrs. Commun. Image Rep., 19 (2008), pp. 270–283. CrossrefISIGoogle Scholar

  • [10]  R.W. Buccigrossi and  and E.P. Simoncelli, Image compression via joint statistical characterization in the wavelet domain, IEEE Trans. Image Process., 8 (1999), pp. 1688–1701. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [11]  A.R. Calderbank and  and P.W. Shor, Good quantum error-correcting codes exist, Phys. Rev. A, 54 (1996), pp. 1098–1105. PLRAAN 1050-2947 CrossrefISIGoogle Scholar

  • [12]  E.J. Candès and  and D.L. Donoho, Recovering edges in ill-posed inverse problems: Optimality of curvelet frames, Ann. Statist., 30 (2000), pp. 784–842. ASTSC7 0090-5364 ISIGoogle Scholar

  • [13]  E.J. Candès and  and D.L. Donoho, New tight frames of curvelets and optimal representations of objects with piecewise-$C^2$ singularities, Comm. Pure Appl. Math., 57 (2002), pp. 219–266. CPMAMV 0010-3640 CrossrefISIGoogle Scholar

  • [14]  E.J. Candès and J. Romberg, Stable signal recovery from incomplete observations, in Wavelet Applications in Signal and Image Processing XI, Proc. SPIE Conf. 5914, SPIE, Bellingham, WA, 2005. Google Scholar

  • [15]  E.J. Candès, J. Romberg and , and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2006), pp. 489–509. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [16]  E. Candès, J. Romberg and , and T. Tao, Quantitative robust uncertainty principles and optimally sparse decompositions, Found. Comput. Math., 6 (2006), pp. 227–254. 1615-3375 CrossrefISIGoogle Scholar

  • [17]  E. Candès, J. Romberg and , and T. Tao, Stable signal recovery from incomplete and inaccurate measurements, Comm. Pure Appl. Math., 59 (2006), pp. 1207–1223. CPMAMV 0010-3640 CrossrefISIGoogle Scholar

  • [18]  E.J. Candès and  and T. Tao, Decoding by linear programming, IEEE Trans. Inform. Theory, 51 (2005), pp. 4203–4215. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [19]  S.G. Chang, B. Yu and , and M. Vetterli, Adaptive wavelet thresholding for image denoising and compression, IEEE Trans. Image Process., 9 (2000), pp. 1532–1546. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [20]  S.G. Chang, B. Yu and , and M. Vetterli, Wavelet thresholding for multiple noisy image copies, IEEE Trans. Image Process., 9 (2000), pp. 1631–1635. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [21]  S.G. Chang, B. Yu and , and M. Vetterli, Spatially adaptive wavelet thresholding with context modeling for image denoising, IEEE Trans. Image Process., 9 (2000), pp. 1522–1530. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [22]  V. Chandrasekaran, M. Wakin, D. Baron, and R. Baraniuk, Surflets: A sparse representation for multidimensional functions containing smooth discontinuities, in Proceedings of the IEEE Symposium on Information Theory, Chicago, IL, 2004, p. 563. Google Scholar

  • [23]  S. Chen, S.A. Billings and , and W. Luo, Orthogonal least squares methods and their application to non-linear system identification, Internat. J. Control, 50 (1989), pp. 1873–1896. IJCOAZ 0020-7179 CrossrefISIGoogle Scholar

  • [24]  S.S. Chen, D.L. Donoho and , and M.A. Saunders, Atomic decomposition by basis pursuit, SIAM J. Sci. Comput., 20 (1998), pp. 33–61. SJOCE3 1064-8275 LinkISIGoogle Scholar

  • [25]  S.S. Chen, D.L. Donoho and , and M.A. Saunders, Atomic decomposition by basis pursuit, SIAM Rev., 43 (2001), pp. 129–159. SIREAD 0036-1445 LinkISIGoogle Scholar

  • [26]  A. Cohen, R.A. DeVore, P. Petrushev and , and H. Xu, Nonlinear approximation and the space $BV(\mathbb{R}^2)$, Amer. J. Math., 121 (1999), pp. 587–628. AJMAAN 0002-9327 CrossrefISIGoogle Scholar

  • [27]  R. Coifman and D.L. Donoho, Translation-invariant denoising, in Wavelets and Statistics, Lecture Notes in Statist. 103, Springer-Verlag, New York, 1995, pp. 120–150. Google Scholar

  • [28]  R.R. Coifman, Y. Meyer, S. Quake, and M.V. Wickerhauser, Signal processing and compression with wavelet packets, in Progress in Wavelet Analysis and Applications (Toulouse, 1992), Frontières, 1993, pp. 77–93. Google Scholar

  • [29]  R.R. Coifman and  and M.V. Wickerhauser, Adapted waveform analysis as a tool for modeling, feature extraction, and denoising, Optical Engrg., 33 (1994), pp. 2170–2174. OPEGAR 0091-3286 CrossrefISIGoogle Scholar

  • [30]  C. Couvreur and  and Y. Bresler, On the optimality of the backward greedy algorithm for the subset selection problem, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 797–808. SJMAEL 0895-4798 LinkISIGoogle Scholar

  • [31]  C. Daniel and F.S. Wood, Fitting Equations to Data: Computer Analysis of Multifactor Data, 2nd ed., John Wiley and Sons, New York, 1980. Google Scholar

  • [32]  I. Daubechies, M. Defrise and , and C. De-Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint, Commun. Pure Appl. Math., 57 (2004), pp. 1413–1457. CPMAMV 0010-3640 CrossrefISIGoogle Scholar

  • [33]  G. Davis, S. Mallat and , and M. Avellaneda, Adaptive greedy approximations, J. Constructive Approx., 13 (1997), pp. 57–98. CrossrefISIGoogle Scholar

  • [34]  G. Davis, S. Mallat and , and Z. Zhang, Adaptive time-frequency decompositions, Optical Engrg., 33 (1994), pp. 2183–2191. OPEGAR 0091-3286 CrossrefISIGoogle Scholar

  • [35]  R.A. DeVore, B. Jawerth and , and B.J. Lucier, Image compression through wavelet transform coding, IEEE Trans. Inform. Theory, 38 (1992), pp. 719–746. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [36]  R.A. DeVore and  and V. Temlyakov, Some remarks on greedy algorithms, Adv. Comput. Math., 5 (1996), pp. 173–187. ACMHEX 1019-7168 CrossrefISIGoogle Scholar

  • [37]  M.N. Do and  and M. Vetterli, Rotation invariant texture characterization and retrieval using steerable wavelet-domain hidden Markov models, IEEE Trans. Multimedia, 4 (2002), pp. 517–527. ITMUF8 1520-9210 CrossrefISIGoogle Scholar

  • [38]  M.N. Do and  and M. Vetterli, The finite ridgelet transform for image representation, IEEE Trans. Image Process., 12 (2003), pp. 16–28. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [39]  M.N. Do and  and M. Vetterli, Framing pyramids, IEEE Trans. Signal Process., 51 (2003), pp. 2329–2342. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [40]  M.N. Do and  and M. Vetterli, The contourlet transform: An efficient directional multiresolution image representation, IEEE Trans. Image Process., 14 (2005), pp. 2091–2106. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [41]  D.C. Dobson and  and F. Santosa, Recovery of blocky images from noisy and blurred data, SIAM J. Appl. Math., 56 (1996), pp. 1181–1198. SMJMAP 0036-1399 LinkISIGoogle Scholar

  • [42]  D.L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), pp. 1289–1306. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [43]  D.L. Donoho, De-noising by soft thresholding, IEEE Trans. Inform. Theory, 41 (1995), pp. 613–627. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [44]  D.L. Donoho, For most large underdetermined systems of linear equations, the minimal $\ell_1$-norm solution is also the sparsest solution, Commun. Pure Appl. Math., 59 (2006), pp. 797–829. CPMAMV 0010-3640 CrossrefISIGoogle Scholar

  • [45]  D.L. Donoho, For most large underdetermined systems of linear equations, the minimal $\ell_1$-norm near-solution approximates the sparsest near-solution, Commun. Pure Appl. Math., 59 (2006), pp. 907–934. CPMAMV 0010-3640 CrossrefISIGoogle Scholar

  • [46]  D.L. Donoho and  and M. Elad, Optimally sparse representation in general (non-orthogonal) dictionaries via L1 minimization, Proc. Natl. Acad. Sci., 100 (2003), pp. 2197–2202. CrossrefISIGoogle Scholar

  • [47]  D.L. Donoho and  and M. Elad, On the stability of the basis pursuit in the presence of noise, Signal Process., 86 (2006), pp. 511–532. SPRODR 0165-1684 CrossrefISIGoogle Scholar

  • [48]  D.L. Donoho, M. Elad and , and V. Temlyakov, Stable recovery of sparse overcomplete representations in the presence of noise, IEEE Trans. Inform. Theory, 52 (2006), pp. 6–18. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [49]  D.L. Donoho and  and X. Huo, Uncertainty principles and ideal atomic decomposition, IEEE Trans. Inform. Theory, 47 (1999), pp. 2845–2862. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [50]  D.L. Donoho and  and I.M. Johnstone, Ideal denoising in an orthonormal basis chosen from a library of bases, C. R. Acad. Sci. Paris Sér. I Math., 319 (1994), pp. 1317–1322. CASMEI 0764-4442 Google Scholar

  • [51]  D.L. Donoho and  and I.M. Johnstone, Ideal spatial adaptation by wavelet shrinkage, Biometrika, 81 (1994), pp. 425–455. BIOKAX 0006-3444 CrossrefISIGoogle Scholar

  • [52]  D.L. Donoho and  and I.M. Johnstone, Minimax estimation via wavelet shrinkage, Ann. Statist., 26 (1998), pp. 879–921. ASTSC7 0090-5364 CrossrefISIGoogle Scholar

  • [53]  D.L. Donoho, I.M. Johnstone, G. Kerkyacharian and , and D. Picard, Wavelet shrinkage—asymptopia, J. Roy. Statist. Soc. Ser. B, 57 (1995), pp. 301–337. Google Scholar

  • [54]  D.L. Donoho and  and P.B. Starck, Uncertainty principles and signal recovery, SIAM J. Appl. Math., 49 (1989), pp. 906–931. SMJMAP 0036-1399 LinkISIGoogle Scholar

  • [55]  D.L. Donoho and V. Stodden, When does non-negative matrix factorization give a correct decomposition into parts?, in Proceedings of NIPS 2003, Adv. Neural Inform. Process. 16, MIT Press, Cambridge, MA, 2004. Google Scholar

  • [56]  D.L. Donoho and  and J. Tanner, Sparse nonnegative solutions of underdetermined linear equations by linear programming, Proc. Natl. Acad. Sci., 102 (2005), pp. 9446–9451. CrossrefISIGoogle Scholar

  • [57]  D.L. Donoho and  and J. Tanner, Neighborliness of randomly-projected simplices in high dimensions, Proc. Natl. Acad. Sci., 102 (2005), pp. 9452–9457. CrossrefISIGoogle Scholar

  • [58]  D.L. Donoho and  and Y. Tsaig, Extensions of compressed sensing, Signal Process., 86 (2006), pp. 549–571. SPRODR 0165-1684 CrossrefISIGoogle Scholar

  • [59]  D.L. Donoho, Y. Tsaig, I. Drori, and J.-L. Starck, Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit, IEEE Trans. Inform. Theory, submitted. Google Scholar

  • [60]  B. Efron, T. Hastie, I.M. Johnstone and , and R. Tibshirani, Least angle regression, Ann. Statist., 32 (2004), pp. 407–499. ASTSC7 0090-5364 CrossrefISIGoogle Scholar

  • [61]  M. Elad, Sparse representations are most likely to be the sparsest possible, EURASIP J. Appl. Signal Process., 1 (2006), article 96247. 1110-8657 Google Scholar

  • [62]  M. Elad, Why simple shrinkage is still relevant for redundant representations?, IEEE Trans. Inform. Theory, 52 (2006), pp. 5559–5569. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [63]  M. Elad and M. Aharon, Image denoising via learned dictionaries and sparse representation, in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, New York, 2006, pp. 895–900. Google Scholar

  • [64]  M. Elad and  and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Trans. Image Process., 15 (2006), pp. 3736–3745. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [65]  M. Elad and  and A.M. Bruckstein, A generalized uncertainty principle and sparse representation in pairs of bases, IEEE Trans. Inform. Theory, 48 (2002), pp. 2558–2567. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [66]  M. Elad, B. Matalon, and M. Zibulevsky, Image denoising with shrinkage and redundant representations, in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, New York, 2006, pp. 1924–1931. Google Scholar

  • [67]  M. Elad, B. Matalon and , and M. Zibulevsky, Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization, Appl. Comput. Harmon. Anal., 23 (2007), pp. 346–367. ACOHE9 1063-5203 CrossrefISIGoogle Scholar

  • [68]  M. Elad, J.-L. Starck, P. Querre and , and D.L. Donoho, Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA), Appl. Comput. Harmon. Anal., 19 (2005), pp. 340–358. ACOHE9 1063-5203 CrossrefISIGoogle Scholar

  • [69]  K. Engan, S.O. Aase and , and J.H. Husoy, Multi-frame compression: Theory and design, Signal Process., 80 (2000), pp. 2121–2140. SPRODR 0165-1684 CrossrefISIGoogle Scholar

  • [70]  R. Eslami and H. Radha, The contourlet transform for image de-noising using cycle spinning, in Proceedings of Asilomar Conference on Signals, Systems, and Computers, 2003, pp. 1982–1986. Google Scholar

  • [71]  R. Eslami and  and H. Radha, Translation-invariant contourlet transform and its application to image denoising, IEEE Trans. Image Process., 15 (2006), pp. 3362–3374. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [72]  M.J. Fadili and J.-L. Starck, Sparse representation-based image deconvolution by iterative thresholding, Astronomical Data Analysis ADA'06, Marseilles, France, 2006. Google Scholar

  • [73]  A. Feuer and  and A. Nemirovsky, On sparse representation in pairs of bases, IEEE Trans. Inform. Theory, 49 (2002), pp. 1579–1581. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [74]  M.A. Figueiredo, J.M. Bioucas-Dias and , and R.D. Nowak, Majorization-minimization algorithms for wavelet-based image restoration, IEEE Trans. Image Process., 16 (2007), pp. 2980–2991. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [75]  M.A. Figueiredo and R.D. Nowak, A bound optimization approach to wavelet-based image deconvolution, in Proceedings of the IEEE International Conference on Image Processing (ICIP 2005), Genoa, Italy, 2005, pp. 782–785. Google Scholar

  • [76]  M.A. Figueiredo and  and R.D. Nowak, An EM algorithm for wavelet-based image restoration, IEEE Trans. Image Process., 12 (2003), pp. 906–916. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [77]  A.K. Fletcher, S. Rangan, V.K. Goyal, and K. Ramchandran, Analysis of denoising by sparse approximation with random frame asymptotics, in Proceedings of the IEEE International Symposium on Information Theory, 2005, pp. 1706–1710. Google Scholar

  • [78]  A.K. Fletcher, S. Rangan, V.K. Goyal and , and K. Ramchandran, Denoising by sparse approximation: Error bounds based on rate-distortion theory, EURASIP J. Appl. Signal Process., 1 (2006), article 26318. 1110-8657 Google Scholar

  • [79]  J.J. Fuchs, On sparse representations in arbitrary redundant bases, IEEE Trans. Inform. Theory, 50 (2004), pp. 1341–1344. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [80]  J.J. Fuchs, Recovery of exact sparse representations in the presence of bounded noise, IEEE Trans. Inform. Theory, 51 (2005), pp. 3601–3608. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [81]  A. Gersho and R.M. Gray, Vector Quantization and Signal Compression, Kluwer Academic, Dordrecht, The Netherlands, 1992. Google Scholar

  • [82]  A.C. Gilbert, S. Muthukrishnan, and M.J. Strauss, Approximation of functions over redundant dictionaries using coherence, in Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York, SIAM, Philadelphia, 2003, pp. 243–252. Google Scholar

  • [83]  G.H. Golub and C.F. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, Baltimore, MD, 1996. Google Scholar

  • [84]  I.F. Gorodnitsky and  and B.D. Rao, Sparse signal reconstruction from limited data using FOCUSS: A re-weighted norm minimization algorithm, IEEE Trans. Signal Process., 45 (1997), pp. 600–616. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [85]  R. Gribonval, R. Figueras and , and P. Vandergheynst, A simple test to check the optimality of a sparse signal approximation, Signal Process., 86 (2006), pp. 496–510. SPRODR 0165-1684 CrossrefISIGoogle Scholar

  • [86]  R. Gribonval and  and M. Nielsen, Sparse decompositions in unions of bases, IEEE Trans. Inform. Theory, 49 (2003), pp. 3320–3325. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [87]  R. Gribonval and  and P. Vandergheynst, On the exponential convergence of matching pursuits in quasi-incoherent dictionaries, IEEE Trans. Inform. Theory, 52 (2006), pp. 255–261. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [88]  O.G. Guleryuz, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising—Part I: Theory, IEEE Trans. Image Process., 15 (2006), pp. 539–554. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [89]  O.G. Guleryuz, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising—Part II: Adaptive algorithms, IEEE Trans. Image Process., 15 (2006), pp. 555–571. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [90]  T. Hastie, R. Tibshirani, and J.H. Friedman, Elements of Statistical Learning, Springer-Verlag, New York, 2001. Google Scholar

  • [91]  R.A. Horn and C.R. Johnson, Matrix Analysis, Cambridge University Press, New York, 1985. Google Scholar

  • [92]  P.O. Hoyer, Non-negative matrix factorization with sparseness constraints, J. Machine Learning Res., 5 (2004), pp. 1457–1469. 1532-4435 ISIGoogle Scholar

  • [93]  X. Huo, Sparse Image representation via Combined Transforms, Ph.D. thesis, Stanford, 1999. Google Scholar

  • [94]  A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, Wiley Interscience, New York, 2001. Google Scholar

  • [95]  A.K. Jain, Fundamentals of Digital Image Processing, Prentice–Hall, Englewood Cliffs, NJ, 1989. Google Scholar

  • [96]  M. Jansen, Noise Reduction by Wavelet Thresholding, Springer-Verlag, New York, 2001. Google Scholar

  • [97]  L.A. Karlovitz, Construction of nearest points in the $\ell_p$, p even and $\ell_\infty$ norms, J. Approx. Theory, 3 (1970), pp. 123–127. JAXTAZ 0021-9045 CrossrefGoogle Scholar

  • [98]  B. Kashin, The widths of certain finite-dimensional sets and classes of smooth functions, Izv. Akad. Nauk SSSR Ser. Mat., 41 (1977), pp. 334–351. 0373-2436 Google Scholar

  • [99]  E. Kidron, Y.Y. Schechner and , and M. Elad, Cross-modality localization via sparsity, IEEE Trans. Signal Process., 55 (2007), pp. 1390–1404. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [100]  S.-J. Kim, K. Koh, M. Lustig, S. Boyd and , and D. Gorinevsky, A method for large-scale $\ell_1$-regularized least squares problems with applications in signal processing and statistics, IEEE J. Sel. Topics Signal Proc., 1 (2007), pp. 606–617. CrossrefISIGoogle Scholar

  • [101]  K. Kreutz-Delgado, J.F. Murray, B.D. Rao, K. Engan, T.-W, Lee and , and T.J. Sejnowski, Dictionary learning algorithms for sparse representation, Neural Comput., 15 (2003), pp. 349–396. NEUCEB 0899-7667 CrossrefISIGoogle Scholar

  • [102]  J.B. Kruskal, Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics, Linear Algebra Appl., 18 (1977), pp. 95–138. LAAPAW 0024-3795 CrossrefISIGoogle Scholar

  • [103]  M. Lang, H. Guo and , and J.E. Odegard, Noise reduction using undecimated discrete wavelet transform, IEEE Signal Process. Lett., 3 (1996), pp. 10–12. ISPLEM 1070-9908 CrossrefISIGoogle Scholar

  • [104]  D. Lee and  and H. Seung, Learning the parts of objects by non-negative matrix factorization, Nature, 401 (1999), pp. 788–791. NATUAS 0028-0836 CrossrefISIGoogle Scholar

  • [105]  E. Le Pennec and  and S. Mallat, Sparse geometric image representation with bandelets, IEEE Trans. Image Process., 14 (2005), pp. 423–438. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [106]  S. Lesage, R. Gribonval, F. Bimbot, and L. Benaroya, Learning unions of orthonormal bases with thresholded singular value decomposition, in Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing (ICASSP 2005), 2005, pp. 293–296. Google Scholar

  • [107]  M.S. Lewicki and  and B.A. Olshausen, A probabilistic framework for the adaptation and comparison of image codes, J. Opt. Soc. Amer. A, 16 (1999), pp. 1587–1601. JOAOD6 0740-3232 CrossrefGoogle Scholar

  • [108]  M.S. Lewicki and  and T.J. Sejnowski, Learning overcomplete representations, Neural Comput., 12 (2000), pp. 337–365. NEUCEB 0899-7667 CrossrefISIGoogle Scholar

  • [109]  Y. Li, A. Cichocki and , and S.-i. Amari, Analysis of sparse representation and blind source separation, Neural Comput., 16 (2004), pp. 1193–1234. NEUCEB 0899-7667 CrossrefISIGoogle Scholar

  • [110]  X. Liu and  and N.D. Sidiropoulos, Cramer–Rao lower bounds for low-rank decomposition of multidimensional arrays, IEEE Trans. Signal Process., 49 (2001), pp. 2074–2086. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [111]  =-1M. Lustig, D.L. Donoho, and J.M. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), pp. 1182–1195. MRMEEN 0740-3194 CrossrefISIGoogle Scholar

  • [112]  M. Lustig, J.M. Santos, D.L. Donoho, and J.M. Pauly, k-t SPARSE: High frame rate dynamic MRI exploiting spatio-temporal sparsity, in Proceedings of the 13th Annual Meeting of ISMRM, Seattle, 2006. Google Scholar

  • [113]  J. Mairal, M. Elad and , and G. Sapiro, Sparse representation for color image restoration, IEEE Trans. Image Process., 17 (2008), pp. 53–69. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [114]  D.M. Malioutov, M. Cetin, and A.S. Willsky, Optimal sparse representations in general overcomplete bases, in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, Canada, 2004, pp. 793–796. Google Scholar

  • [115]  D.M. Malioutov, M. Cetin and , and A.S. Willsky, Sparse signal reconstruction perspective for source localization with sensor arrays, IEEE Trans. Signal Process., 53 (2005), pp. 3010–3022. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [116]  S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, San Diego, CA, 1998. Google Scholar

  • [117]  S. Mallat and  and E. Le Pennec, Bandelet image approximation and compression, Multiscale Model. Simul., 4 (2005), pp. 992–1039. MMSUBT 1540-3459 LinkISIGoogle Scholar

  • [118]  S. Mallat and  and Z. Zhang, Matching pursuits with time-frequency dictionaries, IEEE Trans. Signal Process., 41 (1993), pp. 3397–3415. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [119]  S. Mendelson, A. Pajor and , and N. Tomczak-Jaegermann, Reconstruction and subGaussian processes, C. R. Math. Acad. Sci. Paris, 340 (2005), pp. 885–888. CrossrefISIGoogle Scholar

  • [120]  S. Mendelson, A. Pajor, and N. Tomczak-Jaegermann, Uniform uncertainty principle for Bernoulli and subGaussian ensembles, Constructive Approx., to appear. Google Scholar

  • [121]  F.G. Meyer, A. Averbuch and , and J.O. Stromberg, Fast adaptive wavelet packet image compression, IEEE Trans. Image Process., 9 (2000), pp. 792–800. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [122]  F.G. Meyer, A.Z. Averbuch and , and R.R. Coifman, Multilayered image representation: Application to image compression, IEEE Trans. Image Process., 11 (2002), pp. 1072–1080. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [123]  F.G. Meyer and  and R.R. Coifman, Brushlets: A tool for directional image analysis and image compression, Appl. Comput. Harmon. Anal., 4 (1997), pp. 147–187. ACOHE9 1063-5203 CrossrefISIGoogle Scholar

  • [124]  P. Moulin and  and J. Liu, Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors, IEEE Trans. Inform. Theory, 45 (1999), pp. 909–919. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [125]  B.K. Natarajan, Sparse approximate solutions to linear systems, SIAM J. Comput., 24 (1995), pp. 227–234. SMJCAT 0097-5397 LinkISIGoogle Scholar

  • [126]  B.A. Olshausen and  and B.J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature, 381 (1997), pp. 607–609. NATUAS 0028-0836 CrossrefISIGoogle Scholar

  • [127]  B.A. Olshausen and  and B.J. Field, Sparse coding with an overcomplete basis set: A strategy employed by V1?, Vision Res., 37 (1997), pp. 3311–3325. VISRAM 0042-6989 CrossrefISIGoogle Scholar

  • [128]  B.A. Olshausen and  and D.J. Field, Natural image statistics and efficient coding, Network—Computation in Neural Systems, 7 (1996), pp. 333–339. NTWREZ 0954-898X CrossrefISIGoogle Scholar

  • [129]  M.R. Osborne, B. Presnell and , and B.A. Turlach, A new approach to variable selection in least squares problems, IMA J. Numer. Anal., 20 (2000), pp. 389–403. IJNADH 0272-4979 CrossrefISIGoogle Scholar

  • [130]  Y.C. Pati, R. Rezaiifar, and P.S. Krishnaprasad, Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, in Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers, Vol. 1, 1993, pp. 40–44. Google Scholar

  • [131]  W.W. Peterson and E.J. Weldon, Jr., Error-Correcting Codes, 2nd ed., MIT Press, Cambridge, MA, 1972. Google Scholar

  • [132]  G. Pisier, The Volume of Convex Bodies and Banach Space Geometry, Cambridge University Press, Cambridge, UK, 1989. Google Scholar

  • [133]  M.D. Plumbley, Geometry and homotopy for L1 sparse representations, in Proceedings of the Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS'05), Rennes, France, 2005. Google Scholar

  • [134]  M.D. Plumbley, Polar Polytopes and Recovery of Sparse Representations, preprint, 2005; available online from http://arxiv.org/abs/cs/0510032v1. Google Scholar

  • [135]  M.D. Plumbley, Recovery of sparse representations by polytope faces pursuit, in Proceedings of the 6th International Conference on Independent Component Analysis and Blind Source Separation (ICA 2006), Charleston, SC, Lecture Notes in Comput. Sci. 3889, Springer-Verlag, Berlin, 2006, pp. 206–213. Google Scholar

  • [136]  J. Portilla, V. Strela, M.J. Wainwright and , and E.P. Simoncelli, Image denoising using scale mixtures of Gaussians in the wavelet domain, IEEE Trans. Image Process., 12 (2003), pp. 1338–1351. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [137]  M. Protter and M. Elad, Image sequence denoising via sparse and redundant representations, IEEE Trans. Image Process., to appear. Google Scholar

  • [138]  B.D. Rao, K. Engan, S.F. Cotter, J. Palmer and , and K. Kreutz-Delgado, Subset selection in noise based on diversity measure minimization, IEEE Trans. Signal Process., 51 (2003), pp. 760–770. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [139]  B.D. Rao and  and K. Kreutz-Delgado, An affine scaling methodology for best basis selection, IEEE Trans. Signal Process., 47 (1999), pp. 187–200. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [140]  M. Rudelson and R. Vershynin, Geometric Approach to Error-Correcting Codes and Reconstruction of Signals, Technical Report, Department of Mathematics, University of California, Davis, CA, 2005. Google Scholar

  • [141]  L. Rudin, S. Osher and , and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, 60 (1992), pp. 259–268. PDNPDT 0167-2789 CrossrefISIGoogle Scholar

  • [142]  F. Santosa and  and W.W. Symes, Linear inversion of band-limited reflection seismograms, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 1307–1330. SIJCD4 0196-5204 LinkISIGoogle Scholar

  • [143]  S. Sardy, A.G. Bruce and , and P. Tseng, Block coordinate relaxation methods for nonparametric signal denoising with wavelet dictionaries, J. Comput. Graphical Statist., 9 (2000), pp. 361–379. 1061-8600 CrossrefISIGoogle Scholar

  • [144]  E.P. Simoncelli and E.H. Adelson, Noise removal via Bayesian wavelet coring, in Proceedings of the International Conference on Image Processing, Lausanne, Switzerland, 1996, pp. 379–382. Google Scholar

  • [145]  E.P. Simoncelli, W.T. Freeman, E.H. Adelson and , and D.J. Heeger, Shiftable multi-scale transforms, IEEE Trans. Inform. Theory, 38 (1992), pp. 587–607. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [146]  J.-L. Starck, E.J. Candès and , and D.L. Donoho, The curvelet transform for image denoising, IEEE Trans. Image Process., 11 (2002), pp. 670–684. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [147]  J.-L. Starck, M. Elad and , and D.L. Donoho, Image decomposition via the combination of sparse representations and a variational approach, IEEE Trans. Image Process., 14 (2005), pp. 1570–1582. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [148]  J.-L. Starck, M. Elad and , and D.L. Donoho, Redundant multiscale transforms and their application for morphological component separation, Adv. Imaging Electron Phys., 132 (2004), pp. 287–348. AIEPFQ 1076-5670 CrossrefISIGoogle Scholar

  • [149]  J.-L. Starck, M.J. Fadili and , and F. Murtagh, The undecimated wavelet decomposition and its reconstruction, IEEE Trans. Image Process., 16 (2007), pp. 297–309. IIPRE4 1057-7149 CrossrefISIGoogle Scholar

  • [150]  T. Strohmer and  and R. W. Heath, Grassmannian frames with applications to coding and communication, Appl. Comput. Harmon. Anal., 14 (2004), pp. 257–275. ACOHE9 1063-5203 CrossrefISIGoogle Scholar

  • [151]  S. Szarek, Condition number of random matrices, J. Complexity, 7 (1991), pp. 131–149. JOCOEH 0885-064X CrossrefGoogle Scholar

  • [152]  S. Szarek, Spaces with large distance to $\ell^\infty$ and random matrices, Amer. J. Math., 112 (1990), pp. 899–942. AJMAAN 0002-9327 CrossrefISIGoogle Scholar

  • [153]  D.S. Taubman and M.W. Marcellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice, Kluwer Academic, Norwell, MA, 2001. Google Scholar

  • [154]  V.N. Temlyakov, Greedy algorithms and m-term approximation, J. Approx. Theory, 98 (1999), pp. 117–145. JAXTAZ 0021-9045 CrossrefISIGoogle Scholar

  • [155]  V.N. Temlyakov, Weak greedy algorithms, Adv. Comput. Math., 5 (2000), pp. 173–187. ACMHEX 1019-7168 Google Scholar

  • [156]  J.A. Tropp, Greed is good: Algorithmic results for sparse approximation, IEEE Trans. Inform. Theory, 50 (2004), pp. 2231–2242. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [157]  J.A. Tropp, Just relax: Convex programming methods for subset selection and sparse approximation, IEEE Trans. Inform. Theory, 52 (2006), pp. 1030–1051. IETTAW 0018-9448 CrossrefISIGoogle Scholar

  • [158]  J.A. Tropp and A.A. Gilbert, Signal recovery from random measurements via orthogonal matching pursuit, IEEE Trans. Inform. Theory. 53 (2007), pp. 4655–4666. Google Scholar

  • [159]  J.A. Tropp, A.C. Gilbert, S. Muthukrishnan, and M.J. Strauss, Improved sparse approximation over quasi-incoherent dictionaries, in Proceedings of the IEEE International Conference on Image Processing, Barcelona, 2003, pp. 37–40. Google Scholar

  • [160]  Y. Tsaig, Sparse Solution of Underdetermined Linear Systems: Algorithms and Applications, Ph.D. thesis, Stanford, CA, 2007. Google Scholar

  • [161]  Y. Tsaig and  and D.L. Donoho, Breakdown of equivalence between the minimal $L_1$-norm solution and the sparsest solution, Signal Process., 86 (2006), pp. 533–548. SPRODR 0165-1684 CrossrefISIGoogle Scholar

  • [162]  H. Wintney, On the abstract properties of linear dependence, Amer. J. Math., 57 (1935), pp. 509–533. AJMAAN 0002-9327 CrossrefGoogle Scholar

  • [163]  B. Wohlberg, Noise sensitivity of sparse signal representations: Reconstruction error bounds for the inverse problem, IEEE Trans. Signal Process., 51 (2003), pp. 3053–3060. ITPRED 1053-587X CrossrefISIGoogle Scholar

  • [164]  M. Zibulevsky and  and B.A. Pearlmutter, Blind source separation by sparse decomposition in a signal dictionary, Neural Comput., 13 (2001), pp. 863–882. NEUCEB 0899-7667 CrossrefISIGoogle Scholar