Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Nonconvex Optimization
Abstract
Backtracking line-search is an old yet powerful strategy for finding better step sizes to be used in proximal gradient algorithms. The main principle is to locally find a simple convex upper bound of the objective function, which in turn controls the step size that is used. In case of inertial proximal gradient algorithms, the situation becomes much more difficult and usually leads to very restrictive rules on the extrapolation parameter. In this paper, we show that the extrapolation parameter can be controlled by also locally finding a simple concave lower bound of the objective function. This gives rise to a double convex-concave backtracking procedure which allows for an adaptive choice of both the step size and extrapolation parameters. We apply this procedure to the class of inertial Bregman proximal gradient methods, and prove that any sequence generated by these algorithms converges globally to a critical point of the function at hand. Numerical experiments on a number of challenging nonconvex problems in image processing and machine learning were conducted and show the power of combining inertial step and double backtracking strategy in achieving improved performances.
Keywords
MSC codes
Formats available
You can view the full content in the following formats:
Supplementary Material
PLEASE NOTE: These supplementary files have not been peer-reviewed.
Index of Supplementary Materials
Title of paper: Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Non-Convex Optimization
Authors: Mahesh Chandra Mukkamala, Peter Ochs, Thomas Pock, and Shoham Sabach
File: supplement.pdf
Type: PDF
Contents: Additional proofs.
References
1.
H. Attouch and J. Bolte, On the convergence of the proximal algorithm for nonsmooth functions involving analytic features, Math. Program., 116 (2009), pp. 5--16.
2.
H. Attouch, J. Bolte, P. Redont, and A. Soubeyran, Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality, Math. Oper. Res., 35 (2010), pp. 438--457.
3.
H. Attouch, J. Bolte, and B. F. Svaiter, Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward--backward splitting, and regularized Gauss--Seidel methods, Math. Program., 137 (2013), pp. 91--129.
4.
A. Auslender and M. Teboulle, Interior gradient and proximal methods for convex and conic optimization, SIAM J. Optim., 16 (2006), pp. 697--725, https://doi.org/10.1137/S1052623403427823.
5.
H. H. Bauschke, J. Bolte, and M. Teboulle, A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications, Math. Oper. Res., 42 (2017), pp. 330--348.
6.
H. H. Bauschke and J. M. Borwein, Legendre functions and the method of random Bregman projections, J. Convex Anal., 4 (1997), pp. 27--67.
7.
A. Beck and M. Teboulle, A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J. Imaging Sci., 2 (2009), pp. 183--202, https://doi.org/10.1137/080716542.
8.
J. Bolte, A. Daniilidis, and A. Lewis, The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems, SIAM J. Optim., 17 (2006), pp. 1205--1223, https://doi.org/10.1137/050644641.
9.
J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota, Clarke subgradients of stratifiable functions, SIAM J. Optim., 18 (2007), pp. 556--572, https://doi.org/10.1137/060670080.
10.
J. Bolte, A. Daniilidis, O. Ley, and L. Mazet, Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity, Trans. Amer. Math. Soc., 362 (2010), pp. 3319--3363.
11.
J. Bolte, S. Sabach, and M. Teboulle, Proximal alternating linearized minimization for nonconvex and nonsmooth problems, Math. Program., 146 (2014), pp. 459--494.
12.
J. Bolte, S. Sabach, M. Teboulle, and Y. Vaisbourd, First order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems, SIAM J. Optim., 28 (2018), pp. 2131--2151, https://doi.org/10.1137/17M1138558.
13.
R. I. BoŢ, E. R. Csetnek, and S. C. László, An inertial forward--backward algorithm for the minimization of the sum of two nonconvex functions, EURO J. Comput. Optim., 4 (2016), pp. 3--25.
14.
L. M. Bregman, The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming, Ž. Vyčisl. Mat i Mat. Fiz., 7 (1967), pp. 200--217.
15.
E. J. Candes, X. Li, and M. Soltanolkotabi, Phase retrieval via Wirtinger flow: Theory and algorithms, IEEE Trans. Inform. Theory, 61 (2015), pp. 1985--2007.
16.
E. J. Candes, M. B. Wakin, and S. Boyd, Enhancing sparsity by reweighted $\ell_1$ minimization, J. Fourier Anal. Appl., 14 (2008), pp. 877--905.
17.
Y. Censor and A. Lent, An iterative row-action method for interval convex programming, J. Optim. Theory Appl., 34 (1981), pp. 321--353.
18.
Y. Censor and S. A. Zenios, Proximal minimization algorithm with D-functions, J. Optim. Theory Appl., 73 (1992), pp. 451--464.
19.
H. Chang, S. Marchesini, Y. Lou, and T. Zeng, Variational phase retrieval with globally convergent preconditioned proximal algorithm, SIAM J. Imaging Sci., 11 (2018), pp. 56--93, https://doi.org/10.1137/17M1120439.
20.
G. Chen and M. Teboulle, Convergence analysis of a proximal-like minimization algorithm using Bregman functions, SIAM J. Optim., 3 (1993), pp. 538--543, https://doi.org/10.1137/0803026.
21.
J. Duchi and F. Ruan, Solving (Most) of a Set of Quadratic Equalities: Composite Optimization for Robust Phase Retrieval, preprint, https://arxiv.org/abs/1705.02356, 2017.
22.
J. Eckstein, Nonlinear proximal point algorithms using Bregman functions, with applications to convex programming, Math. Oper. Res., 18 (1993), pp. 202--226.
23.
J. Friedman, T. Hastie, and R. Tibshirani, The Elements of Statistical Learning, in Data mining, Inference, and Prediction, Springer Ser. Statist., Springer-Verlag, New York, 2001.
24.
P. Gong, C. Zhang, L. Zhaosong, J. Z. Huang, and J. Ye, A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems, in Proceedings of the 30th International Conference on Machine Learning, S. Dasgupta and D. McAllester, eds., PMLR, 28 (2013), pp. 37--45.
25.
J.-B. Hiriart-Urruty and C. Lemarechal, Fundamentals of Convex Analysis, Springer-Verlag, Berlin, Heidelberg, 2012.
26.
K. Kurdyka, On gradients of functions definable in o-minimal structures, Ann. Inst. Fourier (Grenoble), 48 (1998), pp. 769--783.
27.
S. Łojasiewicz, Une propriété topologique des sous-ensembles analytiques réels, in Les Équations aux Dérivées Partielles (Paris, 1962), Éditions du Centre National de la Recherche Scientifique, Paris, 1963, pp. 87--89.
28.
D. R. Luke, Phase retrieval, What's new?, SIAG/OPT Views and News, 25 (2017), pp. 1--5.
29.
J. J. Moreau, Proximité et dualité dans un espace hilbertien, Bull. Soc. Math. France, 93 (1965), pp. 273--299.
30.
M. C. Mukkamala and P. Ochs, Beyond Alternating Updates for Matrix Factorization with Inertial Bregman Proximal Gradient Algorithms, preprint, https://arxiv.org/abs/1905.09050, 2019.
31.
M. C. Mukkamala, F. Westerkamp, E. Laude, D. Cremers, and P. Ochs, Bregman Proximal Framework for Deep Linear Neural Networks, preprint, https://arxiv.org/abs/1910.03638, 2019.
32.
Y. E. Nesterov, A method for solving the convex programming problem with convergence rate $O(1/k\sp{2})$, Dokl. Akad. Nauk SSSR, 269 (1983), pp. 543--547.
33.
M. Nikolova, Analysis of the recovery of edges in images and signals by minimizing nonconvex regularized least-squares, Multiscale Model. Simul., 4 (2005), pp. 960--991, https://doi.org/10.1137/040619582.
34.
P. Ochs, Long Term Motion Analysis for Object Level Grouping and Nonsmooth Optimization Methods, Ph.D. thesis, Albert-Ludwigs-Universität Freiburg, Freiburg, Germany, 2015, http://lmb.informatik.uni-freiburg.de//Publications/2015/Och15.
35.
P. Ochs, Local convergence of the heavy-ball method and iPiano for non-convex optimization, J. Optim. Theory Appl., 177 (2018), pp. 153--180.
36.
P. Ochs, Unifying abstract inexact convergence theorems and block coordinate variable metric iPiano, SIAM J. Optim., 29 (2019), pp. 541--570, https://doi.org/10.1137/17M1124085.
37.
P. Ochs, Y. Chen, T. Brox, and T. Pock, iPiano: inertial proximal algorithm for nonconvex optimization, SIAM J. Imaging Sci., 7 (2014), pp. 1388--1419, https://doi.org/10.1137/130942954.
38.
P. Ochs, J. Fadili, and T. Brox, Non-smooth non-convex Bregman minimization: Unification and new algorithms, J. Optim. Theory Appl., 181 (2019), pp. 244--278.
39.
T. Pock and S. Sabach, Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems, SIAM J. Imaging Sci., 9 (2016), pp. 1756--1787, https://doi.org/10.1137/16M1064064.
40.
B. T. Polyak, Some methods of speeding up the convergence of iterative methods, Ž. Vyčisl. Mat i Mat. Fiz., 4 (1964), pp. 791--803.
41.
R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Grundlehren Math. Wiss. 317, Springer-Verlag, Berlin, 1998.
42.
W. Su, S. Boyd, and E. Candes, A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights, in Proceedings of the 27th International Conference on Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds., Curran Associates, Red Hook, NY, 2014, pp. 2510--2518.
43.
M. Teboulle, Entropic proximal mappings with application to nonlinear programming, Math. Oper. Res., 17 (1992), pp. 670--690.
44.
M. Teboulle, A simplified view of first order methods for optimization, Math. Program., 170 (2018), pp. 67--96.
45.
G. Wang, G. B. Giannakis, and Y. C. Eldar, Solving systems of random quadratic equations via truncated amplitude flow, IEEE Trans. Inform. Theory, 64 (2018), pp. 773--794.
46.
B. Wen, X. Chen, and T. K. Pong, Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems, SIAM J. Optim., 27 (2017), pp. 124--145, https://doi.org/10.1137/16M1055323.
47.
F. Wenand, L. Chu, P. Liu, and R. C. Qiu, A Survey on Nonconvex Regularization Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learning, preprint, https://arxiv.org/abs/1808.05403, 2018.
Information & Authors
Information
Published In
SIAM Journal on Mathematics of Data Science
Pages: 658 - 682
DOI: 10.1137/19M1298007
ISSN (online): 2577-0187
Copyright
© 2020, Society for Industrial and Applied Mathematics.
History
Submitted: 5 November 2019
Accepted: 14 May 2020
Published online: 6 August 2020
Keywords
MSC codes
Authors
Funding Information
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : OC 150/1-1
H2020 European Research Council https://doi.org/10.13039/100010663 : 640156
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Federated Learning and Meta Learning: Approaches, Applications, and DirectionsIEEE Communications Surveys & Tutorials, Vol. 26, No. 1 | 1 Sep 2025
- A Unified Bregman Alternating Minimization Algorithm for Generalized DC Programs with Application to ImagingJournal of Scientific Computing, Vol. 101, No. 3 | 7 November 2024
- Momentum-based variance-reduced stochastic Bregman proximal gradient methods for nonconvex nonsmooth optimizationExpert Systems with Applications, Vol. 18 | 1 Dec 2024
- A stochastic two-step inertial Bregman proximal alternating linearized minimization algorithm for nonconvex and nonsmooth problemsNumerical Algorithms, Vol. 97, No. 1 | 9 November 2023
- A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problemsJournal of Global Optimization, Vol. 89, No. 4 | 5 March 2024
- Automatic calculation of step size and inertia parameter for convolutional dictionary learningPattern Recognition, Vol. 152 | 1 Aug 2024
- Difference-of-Convex Algorithm with Extrapolation for Nonconvex, Nonsmooth Optimization ProblemsMathematics of Operations Research, Vol. 49, No. 3 | 1 Aug 2024
- Tight Differentiable Relaxation of Sum Ergodic Capacity Maximization for Clustered Cell-Free Networking2024 IEEE International Symposium on Information Theory (ISIT) | 7 Jul 2024
- Bregman proximal point type algorithms for quasiconvex minimizationOptimization, Vol. 73, No. 3 | 18 August 2022
- Convergence Analysis for Bregman Iterations in Minimizing a Class of Landau Free Energy FunctionalsSIAM Journal on Numerical Analysis, Vol. 62, No. 1 | 7 February 2024
- A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problemsNumerical Algorithms, Vol. 95, No. 1 | 30 June 2023
- Provable Phase Retrieval with Mirror DescentSIAM Journal on Imaging Sciences, Vol. 16, No. 3 | 14 July 2023
- A Bregman stochastic method for nonconvex nonsmooth problem beyond global Lipschitz gradient continuityOptimization Methods and Software, Vol. 38, No. 5 | 19 May 2023
- An alternating structure-adapted Bregman proximal gradient descent algorithm for constrained nonconvex nonsmooth optimization problems and its inertial variantJournal of Global Optimization, Vol. 87, No. 1 | 24 June 2023
- Stochastic Composition Optimization of Functions Without Lipschitz Continuous GradientJournal of Optimization Theory and Applications, Vol. 198, No. 1 | 10 March 2023
- New Bregman proximal type algorithms for solving DC optimization problemsComputational Optimization and Applications, Vol. 83, No. 3 | 23 September 2022
- Global convergence of model function based Bregman proximal minimization algorithmsJournal of Global Optimization, Vol. 83, No. 4 | 1 December 2021
- Optimal complexity and certification of Bregman first-order methodsMathematical Programming, Vol. 194, No. 1-2 | 21 April 2021
- A refined inertial DC algorithm for DC programmingOptimization and Engineering, Vol. 9 | 5 April 2022
- Block Bregman Majorization Minimization with ExtrapolationSIAM Journal on Mathematics of Data Science, Vol. 4, No. 1 | 13 January 2022
- Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge LearningProceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing | 26 July 2021
- Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorizationComputational Optimization and Applications, Vol. 79, No. 3 | 9 June 2021
- Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problemsJournal of Global Optimization, Vol. 79, No. 3 | 19 August 2020
- A Bregman Forward-Backward Linesearch Algorithm for Nonconvex Composite Optimization: Superlinear Convergence to Nonisolated Local MinimaSIAM Journal on Optimization, Vol. 31, No. 1 | 22 February 2021
- Learning Consistent Discretizations of the Total VariationSIAM Journal on Imaging Sciences, Vol. 14, No. 2 | 16 June 2021
- Bregman Proximal Gradient Algorithms for Deep Matrix FactorizationScale Space and Variational Methods in Computer Vision | 30 April 2021
- Bregman Proximal Mappings and Bregman–Moreau Envelopes Under Relative Prox-RegularityJournal of Optimization Theory and Applications, Vol. 184, No. 3 | 31 January 2020
View Options
View options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].