Efficiency of the Accelerated Coordinate Descent Method on Structured Optimization Problems
Abstract
In this paper we prove a new complexity bound for a variant of the accelerated coordinate descent method [Yu. Nesterov, SIAM J. Optim., 22 (2012), pp. 341--362]. We show that this method often outperforms the standard fast gradient methods (FGM [Yu. Nesterov, Dokl. Akad. Nauk SSSR, 269 (1983), pp. 542--547; Math. Program. (A), 103 (2005), pp. 127--152]) on optimization problems with dense data. In many important situations, the computational expenses of oracle and method itself at each iteration of our scheme are perfectly balanced (both depend linearly on dimensions of the problem). As application examples, we consider unconstrained convex quadratic minimization and the problems arising in the smoothing technique [Nesterov, Math. Program. (A), 103 (2005), pp. 127--152]. On some special problem instances, the provable acceleration factor with respect to FGM can reach the square root of the number of variables. Our theoretical conclusions are confirmed by numerical experiments.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Z. Allen-Zhu, Z. Qu, P. Richtárik, and Y. Yuan, Even faster accelerated coordinate descent using non-uniform sampling, in Proceedings of the 33rd International Conference on Machine Learning, 2016, pp. 1110--1119.
2.
Y. T. Lee and A. Sidford, Efficient Accelerated Coordinate Descent Methods and Fastest Algorithms for Solving Linear Systems, in FOCS, 2013, https://doi.org/10.1109/FOCS.2013.24.
3.
Yu. Nesterov, A method for unconstrained convex minimization problem with the rate of convergence $O({1\over k^2})$, Dokl. AN SSSR (translated as Soviet Math. Docl.), 269 (1983), pp. 543--547.
4.
Yu. Nesterov, Smooth minimization of non-smooth functions, Math. Program. (A), 103 (2005), pp. 127--152, https://doi.org/10.1007/s10107-004-0552-5.
5.
Yu. Nesterov, Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim., 22 (2012), pp. 341--362, https://doi.org/10.1137/100802001.
6.
Yu. Nesterov, Introductory Lectures on Convex Optimization, The Basic Course, Kluwer, Boston, 2004.
7.
Yu. Nesterov, Universal gradient methods for convex optimization problems, Math. Program. (A), 152 (2015), pp. 381--404, https://doi.org/10.1007/s10107-014-0790-0.
8.
P. Richtarik and M. Takac͂, Parallel coordinate descent methods for big data optimization, Math. Program., (2012), pp. 1--51, https://doi.org/10.1007/s10107-015-0901-6.
9.
P. Richtarik and M. Takac͂, Distributed coordinate descent method for learning with big data, J. Machine Learning Res., 17 (2016), 75.
Information & Authors
Information
Published In
Copyright
© 2017, Society for Industrial and Applied Mathematics.
History
Submitted: 5 February 2016
Accepted: 12 September 2016
Published online: 26 January 2017
Keywords
MSC codes
Authors
Funding Information
Fédération Wallonie-Bruxelles http://dx.doi.org/10.13039/501100002910 : ARC 04/09-315
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung http://dx.doi.org/10.13039/501100001711
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Coordinate-update algorithms can efficiently detect infeasible optimization problemsJournal of Mathematical Analysis and Applications, Vol. 543, No. 2 | 1 Mar 2025
- New aspects of black box conditional gradient: Variance reduction and one point feedbackChaos, Solitons & Fractals, Vol. 189 | 1 Dec 2024
- Information FOMO: The Unhealthy Fear of Missing Out on Information—A Method for Removing Misleading Data for Healthier ModelsEntropy, Vol. 26, No. 10 | 30 September 2024
- On Cyclic Block Coordinate Descent Method for Solving Large Inconsistent Linear SystemsCommunications on Applied Mathematics and Computation, Vol. 124 | 19 September 2024
- Random coordinate descent: A simple alternative for optimizing parameterized quantum circuitsPhysical Review Research, Vol. 6, No. 3 | 8 July 2024
- Coordinate descent methods beyond smoothness and separabilityComputational Optimization and Applications, Vol. 88, No. 1 | 13 February 2024
- Randomized Gradient-Free Methods in Convex OptimizationEncyclopedia of Optimization | 8 September 2023
- Cyclic Coordinate Dual Averaging with ExtrapolationSIAM Journal on Optimization, Vol. 33, No. 4 | 25 October 2023
- Factor-$$\sqrt{2}$$ Acceleration of Accelerated Gradient MethodsApplied Mathematics & Optimization, Vol. 88, No. 3 | 23 August 2023
- On Accelerated Coordinate Descent Methods for Searching Equilibria in Two-Stage Transportation Equilibrium Traffic Flow Distribution ModelProgramming and Computer Software, Vol. 49, No. 6 | 1 December 2023
- On Accelerated Coordinate Descent Methods for Searching Equilibria in Two-Stage Transportation Equilibrium Traffic Flow Distribution ModelПрограммирование, No. 6 | 1 November 2023
- Multi-Objective Binary Coordinate Search for Feature Selection2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) | 1 Oct 2023
- Random Coordinate Descent Methods for Nonseparable Composite OptimizationSIAM Journal on Optimization, Vol. 33, No. 3 | 17 August 2023
- Conjugate Gradients Acceleration of Coordinate Descent for Linear SystemsJournal of Scientific Computing, Vol. 96, No. 3 | 5 August 2023
- On the Worst-Case Analysis of Cyclic Coordinate-Wise Algorithms on Smooth Convex Functions2023 European Control Conference (ECC) | 13 Jun 2023
- Near-linear convergence of the Random Osborne algorithm for Matrix BalancingMathematical Programming, Vol. 198, No. 1 | 30 May 2022
- On multi-step greedy randomized coordinate descent method for solving large linear least-squares problemsComputational and Applied Mathematics, Vol. 42, No. 1 | 7 January 2023
- Unifying Framework for Accelerated Randomized Methods in Convex OptimizationFoundations of Modern Statistics | 17 July 2023
- Pick your Neighbor: Local Gauss-Southwell Rule for Fast Asynchronous Decentralized Optimization2022 IEEE 61st Conference on Decision and Control (CDC) | 6 Dec 2022
- Exact gradient methods with memoryOptimization Methods and Software, Vol. 37, No. 6 | 20 July 2022
- Oracle Complexity Separation in Convex OptimizationJournal of Optimization Theory and Applications, Vol. 193, No. 1-3 | 27 April 2022
- On adapting Nesterov's scheme to accelerate iterative methods for linear problemsNumerical Linear Algebra with Applications, Vol. 29, No. 2 | 17 October 2021
- Accelerated Proximal Envelopes: Application to Componentwise MethodsComputational Mathematics and Mathematical Physics, Vol. 62, No. 2 | 9 March 2022
- A New Homotopy Proximal Variable-Metric Framework for Composite Convex MinimizationMathematics of Operations Research, Vol. 47, No. 1 | 1 Feb 2022
- Greedy randomized and maximal weighted residual Kaczmarz methods with oblique projectionElectronic Research Archive, Vol. 30, No. 4 | 1 Jan 2022
- Gauss–Seidel method with oblique directionResults in Applied Mathematics, Vol. 12 | 1 Nov 2021
- Decentralized and parallel primal and dual accelerated methods for stochastic convex programming problemsJournal of Inverse and Ill-posed Problems, Vol. 29, No. 3 | 22 January 2021
- An accelerated directional derivative method for smooth stochastic convex optimizationEuropean Journal of Operational Research, Vol. 290, No. 2 | 1 Apr 2021
- On Adaptive Sketch-and-Project for Solving Linear SystemsSIAM Journal on Matrix Analysis and Applications, Vol. 42, No. 2 | 22 June 2021
- An Optimal Algorithm for Decentralized Finite-Sum OptimizationSIAM Journal on Optimization, Vol. 31, No. 4 | 1 November 2021
- On the Computational Efficiency of Catalyst Accelerated Coordinate DescentMathematical Optimization Theory and Operations Research | 14 June 2021
- Solving Smooth Min-Min and Min-Max Problems by Mixed Oracle AlgorithmsMathematical Optimization Theory and Operations Research: Recent Trends | 21 September 2021
- Adaptive Catalyst for Smooth Convex OptimizationOptimization and Applications | 5 November 2021
- Adaptive Gradient-Free Method for Stochastic OptimizationAdvances in Optimization and Applications | 9 December 2021
- First-Order Methods for Convex OptimizationEURO Journal on Computational Optimization, Vol. 9 | 1 Jan 2021
- Accelerated Meta-Algorithm for Convex Optimization ProblemsComputational Mathematics and Mathematical Physics, Vol. 61, No. 1 | 28 February 2021
- A symmetric alternating minimization algorithm for total variation minimizationSignal Processing, Vol. 176 | 1 Nov 2020
- Coordinate Methods for Matrix Games2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) | 1 Nov 2020
- Accelerated First-Order Optimization Algorithms for Machine LearningProceedings of the IEEE, Vol. 108, No. 11 | 1 Nov 2020
- Randomness and permutations in coordinate descent methodsMathematical Programming, Vol. 181, No. 2 | 30 September 2019
- A Randomized Coordinate Descent Method with Volume SamplingSIAM Journal on Optimization, Vol. 30, No. 3 | 14 July 2020
- Strongly Convex Optimization for the Dual Formulation of Optimal TransportMathematical Optimization Theory and Operations Research | 14 September 2020
- A Generalized Accelerated Composite Gradient Method: Uniting Nesterov's Fast Gradient Method and FISTAIEEE Transactions on Signal Processing, Vol. 68 | 1 Jan 2020
- On greedy randomized coordinate descent methods for solving large linear least‐squares problemsNumerical Linear Algebra with Applications, Vol. 26, No. 4 | 21 March 2019
- Distributed multi-step subgradient optimization for multi-agent systemSystems & Control Letters, Vol. 128 | 1 Jun 2019
- A block symmetric Gauss–Seidel decomposition theorem for convex composite quadratic programming and its applicationsMathematical Programming, Vol. 175, No. 1-2 | 23 February 2018
- Provably Accelerated Randomized Gossip AlgorithmsICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 1 May 2019
- Computing the Best Approximation over the Intersection of a Polyhedral Set and the Doubly Nonnegative ConeSIAM Journal on Optimization, Vol. 29, No. 4 | 31 October 2019
- Coordinate Methods for Accelerating ℓ∞ Regression and Faster Approximate Maximum Flow2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) | 1 Oct 2018
- Stochastic Gradient Descent: Recent TrendsRecent Advances in Optimization and Modeling of Contemporary Problems | 19 October 2018
View Options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].