Abstract

Algorithms for solving variational regularization of ill-posed inverse problems usually involve operators that depend on a collection of continuous parameters. When the operators enjoy some (local) regularity, these parameters can be selected using the so-called Stein Unbiased Risk Estimator (SURE). While this selection is usually performed by an exhaustive search, we address in this work the problem of using the SURE to efficiently optimize for a collection of continuous parameters of the model. When considering nonsmooth regularizers, such as the popular $\ell_1$-norm corresponding to soft-thresholding mapping, the SURE is a discontinuous function of the parameters preventing the use of gradient descent optimization techniques. Instead, we focus on an approximation of the SURE based on finite differences as proposed by Ramani and Unser for the Monte-Carlo SURE approach. Under mild assumptions on the estimation mapping, we show that this approximation is a weakly differentiable function of the parameters and its weak gradient, coined the Stein Unbiased GrAdient estimator of the Risk (SUGAR), provides an asymptotically (with respect to the data dimension) unbiased estimate of the gradient of the risk. Moreover, in the particular case of soft-thresholding, it is proved to also be a consistent estimator. This gradient estimate can then be used as a basis for performing a quasi-Newton optimization. The computation of the SUGAR relies on the closed-form (weak) differentiation of the nonsmooth function. We provide its expression for a large class of iterative methods including proximal splitting methods and apply our strategy to regularizations involving nonsmooth convex structured penalties. Illustrations of various image restoration and matrix completion problems are given.

Keywords

  1. inverse problems
  2. risk estimation
  3. SURE
  4. parameter selection
  5. proximal splitting
  6. sparsity

MSC codes

  1. 68U10
  2. 49N45
  3. 65K10
  4. 90C31

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
H. Attouch and B. F. Svaiter, A continuous dynamical Newton-like approach to solving monotone inclusions, SIAM J. Control Optim., 49 (2011), pp. 574--598.
2.
H. Avron and S. Toledo, Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix, J. ACM, 58 (2011), 8.
3.
H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, CMS Books Math./Ouvrages Math. SMC, Springer, New York, 2011.
4.
A. Benazza-Benyahia and J.-C. Pesquet, Building robust wavelet estimators for multicomponent images using Stein's principle, IEEE Trans. Image Process., 14 (2005), pp. 1814--1830.
5.
J. Bennett and S. Lanning, The Netflix prize, in Proceedings of KDD Cup and Workshop 2007, ACM, New York, 2007, pp. 3--6.
6.
T. Blu and F. Luisier, The SURE-LET approach to image denoising, IEEE Trans. Image Process., 16 (2007), pp. 2778--2786.
7.
T. T. Cai and H. H. Zhou, A data-driven block thresholding approach to wavelet estimation, Ann. Statist., 37 (2009), pp. 569--595.
8.
E. J. Candès and B. Recht, Exact matrix completion via convex optimization, Found. Comput. Math., 9 (2009), pp. 717--772.
9.
E. J. Candès, C. A. Sing-Long, and J. D. Trzasko, Unbiased risk estimates for singular value thresholding and spectral estimators, IEEE Trans. Signal Process., 61 (2013), pp. 4643--4657.
10.
A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), pp. 120--145.
11.
C. Chaux, L. Duval, A. Benazza-Benyahia, and J.-C. Pesquet, A nonlinear Stein-based estimator for multichannel image denoising, IEEE Trans. Signal Process., 56 (2008), pp. 3855--3870.
12.
P. L. Combettes and J.-C. Pesquet, A Douglas-Rachford splitting approach to nonsmooth convex variational signal recovery, IEEE J. Sel. Topics Signal Process., 1 (2007), pp. 564--574.
13.
P. L. Combettes and J.-C. Pesquet, Proximal splitting methods in signal processing, in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Springer, New York, 2011, pp. 185--212.
14.
P. L. Combettes and V. R. Wajs, Signal recovery by proximal forward-backward splitting, Multiscale Model. Simul., 4 (2005), pp. 1168--1200.
15.
I. Daubechies, Orthonormal bases of compactly supported wavelets, Comm. Pure Appl. Math., 41 (1988), pp. 909--996.
16.
C.-A. Deledalle, V. Duval, and J. Salmon, Non-local methods with shape-adaptive patches (NLM-SAP), J. Math. Imaging Vision, 43 (2012), pp. 103--120.
17.
C.-A. Deledalle, G. Peyré, and J. Fadili, Stein COnsistent Risk Estimator (SCORE) for Hard Thresholding, preprint, http://arxiv.org/abs/1301.5874arXiv:1301.5874v1 [math.ST], 2013.
18.
C.-A. Deledalle, F. Tupin, and L. Denis, Poisson NL means: Unsupervised non local means for Poisson noise, in Proceedings of the 2010 17th IEEE International Conference on Image Processing, 2010, pp. 801--804.
19.
C.-A. Deledalle, S. Vaiter, G. Peyré, J. Fadili, and C. Dossal, Proximal splitting derivatives for risk estimation, J. Phys.: Conf. Ser., 386 (2012), 012003.
20.
C.-A. Deledalle, S. Vaiter, G. Peyré, J. Fadili, and C. Dossal, Risk Estimation for Matrix Recovery with Spectral Regularization, preprint, http://arxiv.org/abs/1205.1482arXiv:1205.1482v3 [math.OC], 2012.
21.
C.-A. Deledalle, S. Vaiter, G. Peyré, J. Fadili, and C. Dossal, Unbiased risk estimation for sparse analysis regularization, in Proceedings of the 2012 19th IEEE International Conference on Image Processing, 2012, pp. 3053--3056.
22.
S. J. Dong and K. F. Liu, Stochastic estimation with $Z_2$ noise, Phys. Lett. B, 328 (1994), pp. 130--136.
23.
D. L. Donoho and I. M. Johnstone, Adapting to unknown smoothness via wavelet shrinkage, J. Amer. Statist. Assoc., 90 (1995), pp. 1200--1224.
24.
C. Dossal, M. Kachour, M. J. Fadili, G. Peyré, and C. Chesneau, The degrees of freedom of the lasso for general design matrix, Statist. Sinica, 23 (2013), pp. 809--828.
25.
V. Duval, J.-F. Aujol, and Y. Gousseau, A bias-variance approach for the nonlocal means, SIAM J. Imaging Sci., 4 (2011), pp. 760--788.
26.
A. Edelman, Jacobians of Matrix Transforms (with Wedge Products), Handout 3, MIT, Cambridge, MA, 2005.
27.
B. Efron, How biased is the apparent error rate of a prediction rule?, J. Amer. Statist. Assoc., 81 (1986), pp. 461--470.
28.
Y. C. Eldar, Generalized SURE for exponential families: Applications to regularization, IEEE Trans. Signal Process., 57 (2009), pp. 471--481.
29.
L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.
30.
M. Fazel, Matrix Rank Minimization with Applications, Ph.D. thesis, Department of Electrical Engineering, Stanford University, Stanford, CA, 2002.
31.
D. Gilbarg and N. S. Trudinger, Elliptic Partial Differential Equations of Second Order, 2nd ed., Classics Math. 517, Springer-Verlag, Berlin, 1998.
32.
D. A. Girard, A fast “Monte Carlo cross-validation” procedure for large least squares problems with noisy data, Numer. Math., 56 (1989), pp. 1--23.
33.
R. Giryes, M. Elad, and Y. C. Eldar, The projected GSURE for automatic parameter tuning in iterative shrinkage methods, Appl. Comput. Harmon. Anal., 30 (2011), pp. 407--422.
34.
G. H. Golub, M. Heath, and G. Wahba, Generalized cross-validation as a method for choosing a good ridge parameter, Technometrics, 21 (1979), pp. 215--223.
35.
A. Griewank and A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, SIAM, Philadelphia, 2008.
36.
J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms, Vols. I--II, Springer-Verlag, Berlin, 2001.
37.
H. M. Hudson, A natural identity for exponential families with applications in multiparameter estimation, Ann. Statist., 6 (1978), pp. 473--484.
38.
H. M. Hudson and T. Lee, Maximum likelihood restoration and choice of smoothing parameter in deconvolution of image data subject to Poisson noise, Comput. Statist. Data Anal., 26 (1998), pp. 393--410.
39.
M. F. Hutchinson, A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines, Comm. Statist. Simulation Comput., 18 (1989), pp. 1059--1076.
40.
K. Kato, On the degrees of freedom in shrinkage estimation, J. Multivariate Anal., 100 (2009), pp. 1338--1352.
41.
N. Komodakis and J.-C. Pesquet, Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems, preprint, http://arxiv.org/abs/1406.5429arXiv:1406.5429v1 [cs.NA], 2014.
42.
A. S. Lewis and M. L. Overton, Nonsmooth optimization via quasi-Newton methods, Math. Program., 141 (2013), pp. 135--163.
43.
A. S. Lewis and H. S. Sendov, Twice differentiable spectral functions, SIAM J. Matrix Anal. Appl., 23 (2001), pp. 368--386.
44.
K.-C. Li, From Stein's unbiased risk estimates to the method of generalized cross validation, Ann. Statist., 13 (1985), pp. 1352--1377.
45.
F. Luisier, T. Blu, and M. Unser, SURE-LET for orthonormal wavelet-domain video denoising, IEEE Trans. Circuits Syst. Video Technol., 20 (2010), pp. 913--919.
46.
S. Mallat, A Wavelet Tour of Signal Processing. The Sparse Way, 3rd ed., Elsevier/Academic Press, Amsterdam, 2009.
47.
U. Naumann, Optimal Jacobian accumulation is NP-complete, Math. Program., 112 (2008), pp. 427--441.
48.
J.-C. Pesquet, A. Benazza-Benyahia, and C. Chaux, A SURE approach for digital signal/image deconvolution problems, IEEE Trans. Signal Process., 57 (2009), pp. 4616--4632.
49.
J.-C. Pesquet and D. Leporini, A new wavelet estimator for image denoising, in Proceedings of the Sixth International Conference on Image Processing and Its Applications, IET, Stevenage, UK, 1997, pp. 249--253.
50.
H. Raguet, J. Fadili, and G. Peyré, A generalized forward-backward splitting, SIAM J. Imaging Sci., 6 (2013), pp. 1199--1226.
51.
S. Ramani, T. Blu, and M. Unser, Monte-Carlo SURE: A black-box optimization of regularization parameters for general denoising algorithms, IEEE Trans. Image Process., 17 (2008), pp. 1540--1554.
52.
S. Ramani, Z. Liu, J. Rosen, J.-F. Nielsen, and J. A. Fessler, Regularization parameter selection for nonlinear iterative image restoration and MRI reconstruction using GCV and SURE-based methods, IEEE Trans. Image Process., 21 (2012), pp. 3659--3672.
53.
S. Ramani, J. Rosen, Z. Liu, and J. A. Fessler, Iterative weighted risk estimation for nonlinear image restoration with analysis priors, in Proceedings of SPIE, Vol. 8296, Computational Imaging X, SPIE, Bellingham, WA, 2012, 82960N.
54.
M. Raphan and E. P. Simoncelli, Optimal denoising in redundant representations, IEEE Trans. Image Process., 17 (2008), pp. 1342--1352.
55.
M. Raphan and E. P. Simoncelli, Least squares estimation without priors or supervision, Neural Comput., 23 (2011), pp. 374--420.
56.
J. Rice, Choice of smoothing parameter in deconvolution problems, Contemp. Math., 59 (1986), pp. 137--151.
57.
R. T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM J. Control Optim., 14 (1976), pp. 877--898.
58.
F. Roosta-Khorasani and U. Ascher, Improved Bounds on Sample Size for Implicit Matrix Trace Estimators, preprint, http://arxiv.org/abs/1308.2475arXiv:1308.2475 [cs.NA], 2013.
59.
L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, 60 (1992), pp. 259--268.
60.
X. Shen and J. Ye, Adaptive model selection, J. Amer. Statist. Assoc., 97 (2002), pp. 210--221.
61.
V. Solo and M. Ulfarsson, Threshold selection for group sparsity, in Proceedings of the 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, 2010, pp. 3754--3757.
62.
C. M. Stein, Estimation of the mean of a multivariate normal distribution, Ann. Statist., 9 (1981), pp. 1135--1151.
63.
D. Sun and J. Sun, Nonsmooth Matrix Valued Functions Defined by Singular Values, Technical report, Department of Decision Sciences, National University of Singapore, Singapore, 2003.
64.
R. J. Tibshirani and J. Taylor, The solution path of the generalized lasso, Ann. Statist., 39 (2011), pp. 1335--1371.
65.
R. J. Tibshirani and J. Taylor, Degrees of freedom in lasso problems, Ann. Statist., 40 (2012), pp. 1198--1232.
66.
S. Vaiter, C.-A. Deledalle, G. Peyré, C. Dossal, and J. Fadili, Local behavior of sparse analysis regularization: Applications to risk estimation, Appl. Comput. Harmon. Anal., 35 (2013), pp. 433--451.
67.
S. Vaiter, C.-A. Deledalle, G. Peyré, J. M Fadili, and C. Dossal, The Degrees of Freedom of Partly Smooth Regularizers, preprint, arXiv:1404.5557v2 [math.ST], 2014.
68.
D. Van De Ville and M. Kocher, SURE-based non-local means, IEEE Signal Process. Lett., 16 (2009), pp. 973--976.
69.
D. Van De Ville and M. Kocher, Non-local means with dimensionality reduction and SURE-based parameter selection, IEEE Trans. Image Process., 9 (2011), pp. 2683--2690.
70.
C. Vonesch, S. Ramani, and M. Unser, Recursive risk estimation for non-linear image deconvolution with a wavelet-domain sparsity constraint, in Proceedings of the 15th IEEE International Conference on Image Processing, 2008, pp. 665--668.
71.
J. Ye, On measuring and correcting the effects of data mining and model selection, J. Amer. Statist. Assoc., 93 (1998), pp. 120--131.
72.
M. Yuan and Y. Lin, Model selection and estimation in regression with grouped variables, J. R. Stat. Soc. Ser. B Stat. Methodol., 68 (2006), pp. 49--67.
73.
H. Zou, T. Hastie, and R. Tibshirani, On the “degrees of freedom” of the lasso, Ann. Statist., 35 (2007), pp. 2173--2192.

Information & Authors

Information

Published In

cover image SIAM Journal on Imaging Sciences
SIAM Journal on Imaging Sciences
Pages: 2448 - 2487
ISSN (online): 1936-4954

History

Submitted: 6 May 2014
Accepted: 19 August 2014
Published online: 25 November 2014

Keywords

  1. inverse problems
  2. risk estimation
  3. SURE
  4. parameter selection
  5. proximal splitting
  6. sparsity

MSC codes

  1. 68U10
  2. 49N45
  3. 65K10
  4. 90C31

Authors

Affiliations

Charles-Alban Deledalle

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

View Options

View options

PDF

View PDF

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media