A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity
Abstract
In this work, we propose a simple modification of the forward-backward splitting method for finding a zero in the sum of two monotone operators. Our method converges under the same assumptions as Tseng's forward-backward-forward method, namely, it does not require cocoercivity of the single-valued operator. Moreover, each iteration only uses one forward evaluation rather than two as is the case for Tseng's method. Variants of the method incorporating a linesearch, relaxation and inertia, or a structured three operator inclusion are also discussed.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
F. Alvarez and H. Attouch, An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping, Set-Valued Anal., 9 (2001), pp. 3--11.
2.
A. Antipin, On a method for convex programs using a symmetrical modification of the Lagrange function, Ekon. Mat. Metody, 12 (1976), pp. 1164--1173.
3.
K. J. Arrow, L. Hurwicz, and H. Uzawa, Studies in Linear and Non-Linear Programming, Stanford Mathematical Studies in the Social Sciences, Stanford University Press, Stanford, CA, 1958.
4.
H. Attouch and A. Cabot, Convergence of a relaxed inertial forward-backward algorithm for structured monotone inclusions, Appl. Math. Optim., (2019), pp. 1--52.
5.
H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Springer, New York, 2011.
6.
J. Bello Cruz and R. Díaz Millán, A variant of forward-backward splitting method for the sum of two monotone operators with a new search strategy, Optimization, 64 (2015), pp. 1471--1486.
7.
R. I. Boţ and E. R. Csetnek, An inertial forward-backward-forward primal-dual splitting algorithm for solving monotone inclusion problems, Numer. Algorithms, 71 (2016), pp. 519--540.
8.
R. I. Boţ, E. R. Csetnek, and A. Heinrich, A primal-dual splitting algorithm for finding zeros of sums of maximal monotone operators, SIAM J. Optim., 23 (2013), pp. 2011--2036.
9.
L. M. Bricen͂o-Arias and P. L. Combettes, A monotone+ skew splitting model for composite monotone inclusions in duality, SIAM J. Optim., 21 (2011), pp. 1230--1250.
10.
L. M. Bricen͂o-Arias and D. Davis, Forward-backward-half forward algorithm for solving monotone inclusions, SIAM J. Optim., 28 (2018), pp. 2839--2871.
11.
A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vis., 40 (2011), pp. 120--145.
12.
G. H. Chen and R. T. Rockafellar, Convergence rates in forward--backward splitting, SIAM J. Optim., 7 (1997), pp. 421--444.
13.
P. L. Combettes, Quasi-Fejérian analysis of some optimization algorithms, in Inherently Parallel Algortihms in Feasibility and Optimization and Their Applications, Stud. Comput. Math. 8, Elsevier, Amsterdam, 2001, pp. 115--152.
14.
P. L. Combettes and J.-C. Pesquet, Stochastic quasi-Fejér block-coordinate fixed point iterations with random sweeping, SIAM J. Optim., 25 (2015), pp. 1221--1248.
15.
E. R. Csetnek, Y. Malitsky, and M. K. Tam, Shadow Douglas--Rachford splitting for monotone inclusions, Appl. Math. Optim., 80 (2019), pp. 665--678.
16.
C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng, Training GANs with optimism, in Proceedings of ICLR, 2018.
17.
D. Davis and W. Yin, A three-operator splitting scheme and its optimization applications, Set-Valued Var. Anal., 25 (2017), pp. 829--858.
18.
G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien, A variational inequality perspective on generative adversarial networks, in Proceedings of ICLR, 2019.
19.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, Generative adversarial nets, in Advances in Neural Information Processing Systems, 2014, pp. 2672--2680.
20.
E. Y. Hamedani and N. S. Aybat, A Primal-Dual Algorithm for General Convex-Concave Saddle Point Problems, arXiv:1803.01401, 2018.
21.
Y.-G. Hsieh, F., J. Malick, and P. Mertikopoulos, On the convergence of single-call stochastic extra-gradient methods, in Proceedings of NeurIPS, 2019, pp. 6936--6946.
22.
P. R. Johnstone and J. Eckstein, Projective Splitting with Forward Steps: Asynchronous and Block-Iterative Operator Splitting, arXiv:1803.07043, 2018.
23.
G. M. Korpelevich, The extragradient method for finding saddle points and other problems, Ekon. Mat. Metody, 12 (1976), pp. 747--756.
24.
P. L. Lions and B. Mercier, Splitting algorithms for the sum of two nonlinear operators, SIAM J. Numer. Anal., 16 (1979), pp. 964--979.
25.
D. Lorenz and T. Pock, An inertial forward-backward algorithm for monotone inclusions, J. Math. Imaging Vis., 51 (2015), pp. 311--325.
26.
Y. Malitsky, Reflected projected gradient method for solving monotone variational inequalities, SIAM J. Optim., 25 (2015), pp. 502--520.
27.
Y. Malitsky, Golden ratio algorithms for variational inequalities, Math. Program., (2019), https://doi.org/10.1007/s10107-019-01416-w.
28.
P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, and G. Piliouras, Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile, in Proceedings of ICLR, 2019.
29.
K. Mishchenko, D. Kovalev, E. Shulgin, P. Richtárik, and Y. Malitsky, Revisiting Stochastic Extragradient, arXiv:1905.11373, 2019.
30.
A. Moudafi and M. Oliny, Convergence of a splitting inertial proximal method for monotone operators, J. Comput. Appl. Math., 155 (2003), pp. 447--454.
31.
Y. Nesterov, Gradient methods for minimizing composite functions, Math. Program., 140 (2013), pp. 125--161.
32.
Y. E. Nesterov, A method for solving the convex programming problem with convergence rate ${O}(1/k^2)$, Dokl. Akad. Nauk SSSR, 269 (1983), pp. 543--547.
33.
L. D. Popov, A modification of the Arrow--Hurwicz method for finding saddle points, Math. Notes, 28 (1980), pp. 845--848.
34.
J. Rieger and M. K. Tam, Backward-forward-reflected-backward splitting for three operator monotone inclusions, Appl. Math. Comput., 381 (2020), 125248, https://doi.org/10.1016/j.amc.2020.125248.
35.
H. Robbins and D. Siegmund, A convergence theorem for non negative almost supermartingales and some applications, in Optimizing Methods in Statistics, Academic Press, New York, 1971, pp. 233--257.
36.
R. T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM J. Control Optim., 14 (1976), pp. 877--898.
37.
E. K. Ryu and B. C. Vu, Finding the forward-Douglas--Rachford-forward method, J. Optim. Theory. Appl., 184 (2000), pp. 858--876.
38.
E. K. Ryu, K. Yuan, and W. Yin, ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs, arXiv:1905.10899, 2019.
39.
B. F. Svaiter, On weak convergence of the Douglas--Rachford method, SIAM J. Control Optim., 49 (2011), pp. 280--287.
40.
P. Tseng, A modified forward-backward splitting method for maximal monotone mappings, SIAM J. Control Optim., 38 (2000), pp. 431--446.
Information & Authors
Information
Published In
Copyright
© 2020, Society for Industrial and Applied Mathematics.
History
Submitted: 13 August 2018
Accepted: 17 March 2020
Published online: 21 May 2020
Keywords
MSC codes
Authors
Funding Information
Alexander von Humboldt-Stiftung https://doi.org/10.13039/100005156
Funding Information
Australian Research Council https://doi.org/10.13039/501100000923
Funding Information
Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659 : SFB755-A4
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Alternating Proximal-Gradient Steps for (Stochastic) Nonconvex-Concave Minimax ProblemsSIAM Journal on Optimization, Vol. 33, No. 3 | 2 August 2023
- Accelerated First-Order Methods for Convex Optimization with Locally Lipschitz Continuous GradientSIAM Journal on Optimization, Vol. 33, No. 3 | 30 August 2023
- Proximal Splitting Algorithms for Convex Optimization: A Tour of Recent Advances, with New TwistsSIAM Review, Vol. 65, No. 2 | 9 May 2023
- Two Steps at a Time---Taking GAN Training in Stride with Tseng's MethodSIAM Journal on Mathematics of Data Science, Vol. 4, No. 2 | 23 June 2022
- A Primal-Dual Algorithm with Line Search for General Convex-Concave Saddle Point ProblemsSIAM Journal on Optimization, Vol. 31, No. 2 | 11 May 2021
- Convergence Rate of $\mathcal{O}(1/k)$ for Optimistic Gradient and Extragradient Methods in Smooth Convex-Concave Saddle Point ProblemsSIAM Journal on Optimization, Vol. 30, No. 4 | 3 December 2020
- Newton-like Inertial Dynamics and Proximal Algorithms Governed by Maximally Monotone OperatorsSIAM Journal on Optimization, Vol. 30, No. 4 | 3 December 2020