Abstract.

We consider the problem of minimizing the composition of a nonsmooth function with a smooth mapping in the case where the proximity operator of the nonsmooth function can be explicitly computed. We first show that this proximity operator can provide the exact smooth substructure of minimizers, not only of the nonsmooth function, but also of the full composite function. We then exploit this proximal identification by proposing an algorithm which combines proximal steps with sequential quadratic programming steps. We show that our method locally identifies the optimal smooth substructure and then converges quadratically. We illustrate its behavior on two problems: the minimization of a maximum of quadratic functions and the minimization of the maximal eigenvalue of a parametrized matrix.

Keywords

  1. nonsmooth optimization
  2. proximal operator
  3. partial smoothness
  4. manifold identification
  5. maximum eigenvalue minimization
  6. sequential quadratic programming

MSC codes

  1. 65K10
  2. 90C26
  3. 49Q12
  4. 90C55

Get full access to this article

View all available purchase options and get full access to this article.

Acknowledgments.

We thank the three anonymous referees and the associate editor for their improvement suggestions, which led to better readability and exposition of the paper.

References

1.
G. Bareilles, F. Iutzeler, and J. Malick, Newton acceleration on manifolds identified by proximal gradient methods, Math. Program., 200 (2023), pp. 37–70, https://doi.org/10.1007/s10107-022-01873-w.
2.
J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, Julia: A fresh approach to numerical computing, SIAM Rev., 59 (2017), pp. 65–98.
3.
J. Bolte, Z. Chen, and E. Pauwels, The multiproximal linearization method for convex composite problems, Math. Program., 182 (2020), pp. 1–36, https://doi.org/10.1007/s10107-019-01382-3.
4.
J. Bolte and E. Pauwels, Majorization-minimization procedures and convergence of SQP methods for semi-algebraic and tame programs, Math. Oper. Res., 41 (2016), pp. 442–465, https://doi.org/10.1287/moor.2015.0735.
5.
J.-F. Bonnans, J. C. Gilbert, C. Lemaréchal, and C. A. Sagastizábal, Numerical Optimization: Theoretical and Practical Aspects, Springer, New York, 2006.
6.
N. Boumal, An Introduction to Optimization on Smooth Manifolds, Cambridge University Press, New York, 2023, https://www.nicolasboumal.net/book.
7.
J. V. Burke, F. E. Curtis, A. S. Lewis, M. L. Overton, and L. E. Simões, Gradient sampling methods for nonsmooth optimization, in Numerical Nonsmooth Optimization, Springer, New York, 2020, pp. 201–225.
8.
A. Daniilidis, W. Hare, and J. Malick, Geometrical interpretation of the predictor-corrector type algorithms in structured optimization problems, Optimization, 55 (2006), pp. 481–503.
9.
D. Drusvyatskiy, A. D. Ioffe, and A. S. Lewis, Nonsmooth optimization using Taylor-like models: Error bounds, convergence, and termination criteria, Math. Program., 185 (2021), pp. 357–383, https://doi.org/10.1007/s10107-019-01432-w.
10.
X. Y. Han and A. S. Lewis, Survey descent: A multipoint generalization of gradient descent for nonsmooth optimization, SIAM J. Optim., 33 (2023), pp. 36–62.
11.
W. Hare and A. S. Lewis, Identifying active constraints via partial smoothness and prox-regularity, J. Convex Anal., 11 (2004), pp. 251–266.
12.
W. Hare and C. Sagastizábal, Computing proximal points of nonconvex functions, Math. Program., 116 (2009), pp. 221–258.
13.
C. Helmberg, M. Overton, and F. Rendl, The spectral bundle method with second-order information, Optim. Methods Softw., 29 (2014), pp. 855–876, https://doi.org/10.1080/10556788.2013.858155.
14.
J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms, Springer-Verlag, Heidelberg, 1993.
15.
C.-p. Lee, Accelerating Inexact Successive Quadratic Approximation for Regularized Optimization through Manifold Identification, https://arxiv.org/abs/2012.02522, 2021.
16.
J. M. Lee, Introduction to Smooth Manifolds, Grad. Texts in Math. 218, Springer, New York, 2003, https://doi.org/10.1007/978-0-387-21752-9.
17.
A. Lewis and T. Tian, Identifiability, the KL Property in Metric Spaces, and Subgradient Curves, preprint, https://arxiv.org/abs/2205.02868, 2022.
18.
A. Lewis and C. Wylie, A Simple Newton Method for Local Nonsmooth Optimization, https://arxiv.org/abs/1907.11742, 2019.
19.
A. S. Lewis, Active sets, nonsmoothness, and sensitivity, SIAM J Optim., 13 (2002), pp. 702–725.
20.
A. S. Lewis and M. L. Overton, Nonsmooth optimization via quasi-Newton methods, Math. Program., 141 (2013), pp. 135–163, https://doi.org/10.1007/s10107-012-0514-2.
21.
A. S. Lewis and S. J. Wright, A proximal method for composite minimization, Math. Program., 158 (2016), pp. 501–546.
22.
A. S. Lewis and S. Zhang, Partial smoothness, tilt stability, and generalized Hessians, SIAM J Optim., 23 (2013), pp. 74–94, https://doi.org/10.1137/110852103.
23.
R. Mifflin and C. Sagastizábal, A \(\mathcal{VU}\) -algorithm for convex minimization, Math. Program., 104 (2005), pp. 583–608.
24.
S. A. Miller and J. Malick, Newton methods for nonsmooth convex minimization: Connections among \(\mathcal{U}\) -Lagrangian, Riemannian Newton and SQP methods, Math. Program., 104 (2005), pp. 609–633.
25.
J. Nocedal and S. Wright, Numerical Optimization, Springer, New York, 2006.
26.
D. Noll and P. Apkarian, Spectral bundle methods for non-convex maximum eigenvalue functions: Second-order methods, Math. Program., 104 (2005), pp. 729–747.
27.
D. Noll and P. Apkarian, Spectral bundle methods for non-convex maximum eigenvalue functions: Second-order methods, Math. Program., 104 (2005), pp. 729–747, https://doi.org/10.1007/s10107-005-0635-y.
28.
F. Oustry, The \(\mathcal{U}\) -Lagrangian of the maximum eigenvalue function, SIAM J. Optim., 9 (1999), pp. 526–549, https://doi.org/10.1137/S1052623496311776.
29.
R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Grundlehren Math. Wiss. 317, Springer, New York, 2009.
30.
C. Sagastizábal, Composite proximal bundle method, Math. Program., 140 (2013), pp. 189–233, https://doi.org/10.1007/s10107-012-0600-5.
31.
A. Shapiro, On a class of nonsmooth composite functions, Math. Oper. Res., 28 (2003), pp. 677–692, https://doi.org/10.1287/moor.28.4.677.20512.
32.
A. Shapiro and M. K. H. Fan, On eigenvalue optimization, SIAM J. Optim., 5 (1995), pp. 552–569, https://doi.org/10.1137/0805028.
33.
R. S. Womersley and R. Fletcher, An algorithm for composite nonsmooth optimization problems, J. Optim. Theory Appl., 48 (1986), pp. 493–523, https://doi.org/10.1007/BF00940574.

Information & Authors

Information

Published In

cover image SIAM Journal on Optimization
SIAM Journal on Optimization
Pages: 2222 - 2247
ISSN (online): 1095-7189

History

Submitted: 28 June 2022
Accepted: 19 March 2023
Published online: 23 August 2023

Keywords

  1. nonsmooth optimization
  2. proximal operator
  3. partial smoothness
  4. manifold identification
  5. maximum eigenvalue minimization
  6. sequential quadratic programming

MSC codes

  1. 65K10
  2. 90C26
  3. 49Q12
  4. 90C55

Authors

Affiliations

Gilles Bareilles Contact the author
Université Grenoble Alpes, LJK, 38400 Saint-Martin-d’Hères, France.
Université Grenoble Alpes, LJK, 38400 Saint-Martin-d’Hères, France.
Université Grenoble Alpes, CNRS, Grenoble INP, LJK, 38400 Saint-Martin-d’Hères, France.

Funding Information

Agence Nationale de la Recherche (ANR): ANR-19-CE23-0008, ANR-19-P3IA-0003
Funding: This work is funded by ANR JCJC project STROLL (ANR-19-CE23-0008) and MIAI@Grenoble Alpes (ANR-19-P3IA-0003).

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

There are no citations for this item

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media

The SIAM Publications Library now uses SIAM Single Sign-On for individuals. If you do not have existing SIAM credentials, create your SIAM account https://my.siam.org.