Randomized Kaczmarz with Geometrically Smoothed Momentum
Abstract.
This paper studies the effect of adding geometrically smoothed momentum to the randomized Kaczmarz algorithm, which is an instance of stochastic gradient descent on a linear least squares loss function. We prove a result about the expected error in the direction of singular vectors of the matrix defining the least squares loss. We present several numerical examples illustrating the utility of our result and pose several questions.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgment.
We thank the anonymous reviewers, whose suggestions significantly improved the exposition of the results.
References
1.
R. Bollapragada, T. Chen, and R. Ward, On the fast convergence of minibatch heavy ball momentum, IMA J. Numer. Anal., (2024), drae033, https://doi.org/10.1093/imanum/drae033.
2.
N. Coria, J. Haddock, and J. Pacheco, On Quantile Randomized Kaczmarz for Linear Systems with Time-Varying Noise and Corruption, preprint, arXiv:2403.19874, 2024.
3.
M. Dereziński and E. Rebrova, Sharp analysis of sketch-and-project methods via a connection to randomized singular value decomposition, SIAM J. Math. Data Sci., 6 (2024), https://doi.org/10.1137/23M1545537.
4.
R. Gordon, R. Bender, and G. T. Herman, Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and x-ray photography, J. Theoret. Biol., 29 (1970), pp. 471–481, https://doi.org/10.1016/0022-5193(70)90109-8.
5.
K. Gupta, J. Siegel, and S. Wojtowytsch, Achieving Acceleration Despite Very Noisy Gradients, arXiv:2302.05515, 2023.
6.
J. Haddock, D. Needell, E. Rebrova, and W. Swartworth, Quantile-based iterative methods for corrupted systems of linear equations, SIAM J. Matrix Anal. Appl., 43 (2022), pp. 605–637, https://doi.org/10.1137/21M1429187.
7.
D. Han, Y. Su, and J. Xie, Randomized Douglas-Rachford methods for linear systems: Improved accuracy and efficiency, SIAM J. Optim., 34 (2024), https://doi.org/10.1137/23M1567503.
8.
B. Jarman and D. Needell, Quantilerk: Solving large-scale linear systems with corrupted, noisy data, in 2021 55th Asilomar Conference on Signals, Systems, and Computers, IEEE, 2021, pp. 1312–1316.
9.
S. Kaczmarz, Angenäherte Auflösung von Systemen linearer Gleichungen, Bull. Int. Acad. Polon. Sci. Lett. A, (1937), pp. 335–357.
10.
R. Kidambi, P. Netrapalli, P. Jain, and S. Kakade, On the insufficiency of existing momentum schemes for stochastic optimization, in 2018 Information Theory and Applications Workshop (ITA), IEEE, 2018, pp. 1–9.
11.
D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, arXiv:1412.6980, 2014.
12.
J. Liu and S. J. Wright, An accelerated randomized Kaczmarz algorithm, Math. Comp., 85 (2016), pp. 153–178, https://doi.org/10.1090/mcom/2971.
13.
N. Loizou and P. Richtárik, Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods, Comput. Optim. Appl., 77 (2020), pp. 653–710, https://doi.org/10.1007/s10589-020-00220-z.
14.
A. Ma, D. Needell, and A. Ramdas, Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods, SIAM J. Matrix Anal. Appl., 36 (2015), pp. 1590–1604, https://doi.org/10.1137/15M1014425.
15.
N. F. Marshall and O. Mickelin, An optimal scheduled learning rate for a randomized Kaczmarz algorithm, SIAM J. Matrix Anal. Appl., 44 (2023), pp. 312–330, https://doi.org/10.1137/22M148803X.
16.
J. D. Moorman, T. K. Tu, D. Molitor, and D. Needell, Randomized Kaczmarz with averaging, BIT, 61 (2020), pp. 337–359, https://doi.org/10.1007/s10543-020-00824-1.
17.
M. S. Morshed, M. S. Islam, and M. Noor-E-Alam, Sampling Kaczmarz-Motzkin method for linear feasibility problems: Generalization and acceleration, Math. Program., 194 (2022), pp. 719–779, https://doi.org/10.1007/s10107-021-01649-8.
18.
I. Necoara, Faster randomized block Kaczmarz algorithms, SIAM J. Matrix Anal. Appl., 40 (2019), pp. 1425–1452, https://doi.org/10.1137/19M1251643.
19.
D. Needell, Randomized Kaczmarz solver for noisy linear systems, BIT, 50 (2010), pp. 395–403, https://doi.org/10.1007/s10543-010-0265-5.
20.
D. Needell, N. Srebro, and R. Ward, Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm, Math. Program., 155 (2015), pp. 549–573, https://doi.org/10.1007/s10107-015-0864-7.
21.
D. Needell and J. A. Tropp, Paved with good intentions: Analysis of a randomized block Kaczmarz method, Linear Algebra Appl., 441 (2014), pp. 199–221, https://doi.org/10.1016/j.laa.2012.12.022.
22.
D. Needell, R. Zhao, and A. Zouzias, Randomized block Kaczmarz method with projection for solving least squares, Linear Algebra Appl., 484 (2015), pp. 322–343, https://doi.org/10.1016/j.laa.2015.06.027.
23.
Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Springer, 2004.
24.
B. T. Polyak, Some methods of speeding up the convergence of iteration methods, USSR Comput. Math. Math. Phys., 4 (1964), pp. 1–17, https://doi.org/10.1016/0041-5553(64)90137-5.
25.
S. Steinerberger, Randomized Kaczmarz converges along small singular vectors, SIAM J. Matrix Anal. Appl., 42 (2021), pp. 608–615, https://doi.org/10.1137/20M1350947.
26.
S. Steinerberger, Quantile-based random Kaczmarz for corrupted linear systems of equations, Inf. Inference, 12 (2023), pp. 448–465, https://doi.org/10.1093/imaiai/iaab029.
27.
T. Strohmer and R. Vershynin, A randomized Kaczmarz algorithm with exponential convergence, J. Fourier Anal. Appl., 15 (2009), pp. 262–278, https://doi.org/10.1007/s00041-008-9030-4.
28.
L. Tondji, I. Tondji, and D. A. Lorenz, Adaptive Bregman-Kaczmarz: An Approach to Solve Linear Inverse Problems with Independent Noise Exactly, Inverse Problems, 40 (2024), https://doi.org/10.1088/1361-6420/ad5fb1.
29.
R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, Cambridge University Press, 2018.
30.
Y. Zeng, D. Han, Y. Su, and J. Xie, On Adaptive Stochastic Heavy Ball Momentum for Solving Linear Systems, arXiv:2305.05482, 2023.
Information & Authors
Information
Published In
SIAM Journal on Matrix Analysis and Applications
Pages: 2287 - 2313
DOI: 10.1137/24M1633820
ISSN (online): 1095-7162
Copyright
© 2024 Society for Industrial and Applied Mathematics.
History
Submitted: 24 January 2024
Accepted: 28 August 2024
Published online: 2 December 2024
Keywords
MSC codes
Authors
Funding Information
Funding: The third author was supported in part by a start-up grant from Oregon State University.
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
There are no citations for this item
View Options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].