Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms

This paper treats the problem of minimizing a general continuously differentiable function subject to sparsity constraints. We present and analyze several different optimality criteria which are based on the notions of stationarity and coordinatewise optimality. These conditions are then used to derive three numerical algorithms aimed at finding points satisfying the resulting optimality criteria: the iterative hard thresholding method and the greedy and partial sparse-simplex methods. The first algorithm is essentially a gradient projection method, while the remaining two algorithms are of a coordinate descent type. The theoretical convergence of these techniques and their relations to the derived optimality conditions are studied. The algorithms and results are illustrated by several numerical examples.

  • 1.  S. Bahmani, B. Raj, and P. Boufounos, Greedy Sparsity-Constrained Optimization, arxiv:1203.5483v2.pdf, 2012. Google Scholar

  • 2.  A. Beck and M. Teboulle, Gradient-based algorithms with applications to signal recovery problems, in Convex Optimization in Signal Processing and Communications, Y. Eldar and D. Palomar, eds., Cambridge University Press, Cambridge, UK, 2010.12pt Google Scholar

  • 3.  E. V. D. Berg and  M. P. Friedlander , Sparse optimization with least-squares constraints , SIAM J. Optim. , 21 ( 2011 ), pp. 1201 -- 1229 . LinkISIGoogle Scholar

  • 4.  D. P. Bertsekas, Nonlinear Programming, 2nd ed., Athena Scientific, Belmont, MA, 1999. Google Scholar

  • 5.  T. Blumensath, Compressed Sensing with Nonlinear Observations, http://users.fmrib.ox.ac. uk/˜tblumens/papers/B_Nonlinear.pdf (2010). Google Scholar

  • 6.  T. Blumensath and  M. E. Davies , Iterative thresholding for sparse approximations , J. Fourier Anal. Appl. , 14 ( 2008 ), pp. 629 -- 654 . CrossrefISIGoogle Scholar

  • 7.  T. Blumensath and  M. E. Davies , Normalised iterative hard thresholding: Guaranteed stability and performance , IEEE J. Selected Topics in Signal Processing , 4 ( 2010 ), pp. 298 -- 309 . CrossrefISIGoogle Scholar

  • 8.  E. Candès J. Romberg and  T. Tao , Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information , IEEE Trans. Inform. Theory , 52 ( 2006 ), pp. 489 -- 509 . CrossrefISIGoogle Scholar

  • 9.  G. Davis S. Mallat and  M. Avellaneda , Adaptive greedy approximations , Constr. Approx. , 13 ( 1997 ), pp. 57 -- 98 . CrossrefISIGoogle Scholar

  • 10.  R. DeVore and  Nonlinear , Acta Numerica , 7 ( 1998 ), pp. 51 -- 150 . CrossrefGoogle Scholar

  • 11.  D. Donoho , Denoising by soft-thresholding , IEEE Trans. Inform. Theory , 41 ( 1995 ), pp. 613 -- 627 . CrossrefISIGoogle Scholar

  • 12.  D. L. Donoho and  Compressed , IEEE Trans. Inform. Theory , 52 ( 2006 ), pp. 1289 -- 1306 . CrossrefISIGoogle Scholar

  • 13.  D. L. Donoho and  M. Elad , Optimally sparse representation in general (non-orthogonal) dictionaries via l1 minimization , in Proc. Natl. Acad. Sci. USA , 100 ( 2003 ), pp. 2197 -- 2202 . CrossrefISIGoogle Scholar

  • 14.  A. Szameit , Sparsity-based single-shot sub-wavelength coherent diffractive imaging , Nature Materials , 11 ( 2012 ), pp. 455 -- 459 . CrossrefISIGoogle Scholar

  • 15.  J. R. Fienup , Phase retrieval algorithms: A comparison , Appl. Optics , 21 ( 1982 ), pp. 2758 -- 2769 . CrossrefISIGoogle Scholar

  • 16.  R. W. Gerchberg and  W. O. Saxton , A practical algorithm for the determination of phase from image and diffraction plane pictures , Optik , 35 ( 1972 ), pp. 237 -- 246 . ISIGoogle Scholar

  • 17.  I. F. Gorodnitsky and  B. D. Rao , Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm , IEEE Trans. Signal Process. , 45 ( 1997 ), pp. 600 -- 616 . CrossrefISIGoogle Scholar

  • 18.  N. Hurt, Phase Retrieval and Zero Crossings, Kluwer Academic, Norwell, MA, 1989. Google Scholar

  • 19.  M. A. Davenport, M. F. Duarte, Y. C. Eldar, and G. Kutyniok, Introduction to Compressed Sensing, in Compressed Sensing: Theory and Applications, Cambridge University Press, Cambridge, MA, 2012. Google Scholar

  • 20.  S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way, Academic Press, New York, 2008. Google Scholar

  • 21.  S. Mallat and  Z. Zhang , Matching pursuits with time-frequency dictionaries , IEEE Trans. Signal Process. , 41 ( 1993 ), pp. 3397 -- 3415 . CrossrefISIGoogle Scholar

  • 22.  B. Olshausen and  D. Field , Emergence of simple-cell receptive field properties by learning a sparse representation , Nature , 381 ( 1996 ), pp. 607 -- 609 . CrossrefISIGoogle Scholar

  • 23.  S. Scholtes , Nonconvex structures in nonlinear programming , Oper. Res. , 52 ( 2004 ), pp. 368 -- 383 . CrossrefISIGoogle Scholar

  • 24.  Y. Shechtman Y. C. Eldar A. Szameit and  M. Segev , Sparsity-based sub-wavelength imaging with partially spatially incoherent light via quadratic compressed sensing , Optics Express , 19 ( 2011 ), pp. 14807 -- 14822 . CrossrefISIGoogle Scholar

  • 25.  D. Taubman and M. Marcellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice, Kluwer, Dordrecht, Netherlands, 2001. Google Scholar

  • 26.  R. Tibshirani , Regression shrinkage and selection via the lasso , J. Royal Statist. Soc. B , 58 ( 1996 ), pp. 267 -- 288 . ISIGoogle Scholar

  • 27.  J. Tropp , Greed is good: Algorithmic results for sparse approximation , IEEE Trans. Inform. Theory , 50 ( 2004 ), pp. 2231 -- 2242 . CrossrefISIGoogle Scholar

  • 28.  J. Tropp and  S. J. Wright , Computational methods for sparse solution of linear inverse problems , Proc. IEEE , 98 ( 2010 ), pp. 948 -- 958 . CrossrefISIGoogle Scholar

  • 29.  R. Vershynin, Introduction to the Non-asymptotic Analysis of Random Matrices, in Compressed Sensing: Theory and Applications, Cambridge University Press, Cambridge, UK, 2012. Google Scholar