Stochastic Block Mirror Descent Methods for Nonsmooth and Stochastic Optimization
Abstract
In this paper, we present a new stochastic algorithm, namely, the stochastic block mirror descent (SBMD) method for solving large-scale nonsmooth and stochastic optimization problems. The basic idea of this algorithm is to incorporate block coordinate decomposition and an incremental block averaging scheme into the classic (stochastic) mirror descent method, in order to significantly reduce the cost per iteration of the latter algorithm. We establish the rate of convergence of the SBMD method along with its associated large-deviation results for solving general nonsmooth and stochastic optimization problems. We also introduce variants of this method and establish their rate of convergence for solving strongly convex, smooth, and composite optimization problems, as well as certain nonconvex optimization problems. To the best of our knowledge, all these developments related to the SBMD methods are new in the stochastic optimization literature. Moreover, some of our results seem to be new for block coordinate descent methods for deterministic optimization.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
A. Beck and M. Teboulle, Mirror descent and nonlinear projected subgradient methods for convex optimization, Oper. Res. Lett., 31 (2003), pp. 167--175.
2.
A. Beck and L. Tetruashvili, On the convergence of block coordinate descent type methods. SIAM J. Optim, 23 (2013), pp. 2037--2060.
3.
L. M. Bregman, The relaxation method of finding the common point convex sets and its application to the solution of problems in convex programming, USSR Comput. Math. Phys., 7 (1967), pp. 200--217.
4.
O. Fercoq and P. Richtárik, Smooth Minimization of Nonsmooth Functions with Parallel Coordinate Descent Methods, ArXiv:1309.5885, 2013.
5.
S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, I: A generic algorithmic framework, SIAM J. Optim., 22 (2012), pp. 1469--1492.
6.
S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: Shrinking procedures and optimal algorithms, SIAM J. Optim., 23 (2013), pp. 2061--2089.
7.
S. Ghadimi and G. Lan, Stochastic first- and zeroth-order methods for nonconvex stochastic programming, SIAM J. Optim., 23 (2013), pp. 2341--2368.
8.
S. Ghadimi, G. Lan, and H. Zhang, Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization, Math. Program. (2014), pp. 1--39.
9.
A. Juditsky and Y. E. Nesterov, Primal-dual subgradient methods for minimizing uniformly convex functions, arXiv:1401.1792 [math.OC], 2014.
10.
G. Lan, An optimal method for stochastic composite optimization, Math. Program., 133 (2012), pp. 365--397.
11.
G. Lan, Bundle-level type methods uniformly optimal for smooth and non-smooth convex optimization, Math. Program., 149 (2015), pp. 1--45.
12.
G. Lan and R. D. C. Monteiro, Iteration-complexity of first-order augmented Lagrangian methods for convex programming,
Math. Program., (2014).
13.
G. Lan and R. D. C. Monteiro, Iteration-complexity of first-order penalty methods for convex programming, Math. Program., 138 (2013), pp. 115--139.
14.
G. Lan, A. S. Nemirovski, and A. Shapiro, Validation analysis of mirror descent stochastic approximation method, Math. Program., 134 (2012), pp. 425--458.
15.
D. Leventhal and A. S. Lewis, Randomized methods for linear constraints: Convergence rates and conditioning, Math. Oper. Res., 35 (2010), pp. 641--654.
16.
Z. Lu and L. Xiao, On the complexity analysis of randomized block-coordinate descent methods, Math. Program. (2014), pp. 1--28.
17.
Z. Q. Luo and P. Tseng, On the convergence of a matrix splitting algorithm for the symmetric monotone linear complementarity problem, SIAM J. Control Optim., 29 (1991), pp. 1037--1060.
18.
A. Nedić and S. Lee, On stochastic subgradient mirror-descent algorithm with weighted averaging, SIAM J. Optim., 24 (2014), pp. 84--107.
19.
A. S. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2009), pp. 1574--1609.
20.
A. S. Nemirovski and D. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Intersci. Ser. Discrete Math. 15, John Wiley, New York, 1983.
21.
Y. E. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer, Boston, 2004.
22.
Y. E. Nesterov, Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems, Technical report, Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, 2010.
23.
Y. E. Nesterov, Subgradient Methods for Huge-Scale Optimization Problems, Technical report, Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, 2012.
24.
B. T. Polyak, New stochastic approximation type procedures, Avtomat. i Telemekh., 7 (1990), pp. 98--107.
25.
B. T. Polyak and A. B. Juditsky, Acceleration of stochastic approximation by averaging, SIAM J. Control Optim., 30 (1992), pp. 838--855.
26.
P. Richtárik and M. Takác̆, Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function, Math. Program., 144 (2014), pp. 1--38.
27.
H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), pp. 400--407.
28.
S. Shalev-Shwartz and A. Tewari, Stochastic methods for $l_1$ regularized loss minimization, J. Mach. Learn. Res., 12 (2011), pp. 1865--1892.
29.
S. Shalev-Shwartz and T. Zhang, Stochastic dual coordinate ascent methods for regularized loss minimization, J. Mach. Learn. Res., 14 (2013), pp. 567--599.
30.
P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program., 117 (2009), pp. 387--423.
31.
S. J. Wright, Accelerated Block-Coordinate Relaxation for Regularized Optimizations, Manuscript, University of Wisconsin-Madison, Madison, WI, 2010.
32.
E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell, Distance metric learning, with application to clustering with side-information, in Adv. Neural Information Processing Systems 15, MIT Press, Cambridge, MA, 2002, pp. 505--512.
Information & Authors
Information
Published In
Copyright
© 2015, Society for Industrial and Applied Mathematics.
History
Submitted: 10 September 2013
Accepted: 5 February 2015
Published online: 30 April 2015
Keywords
MSC codes
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Inertial accelerated stochastic mirror descent for large-scale generalized tensor CP decompositionComputational Optimization and Applications, Vol. 91, No. 1 | 17 February 2025
- Bregman distance regularization for nonsmooth and nonconvex optimizationCanadian Mathematical Bulletin, Vol. 67, No. 2 | 26 October 2023
- Differentiable Logic Policy for Interpretable Deep Reinforcement Learning: A Study From an Optimization PerspectiveIEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 10 | 1 Oct 2023
- Block Policy Mirror DescentSIAM Journal on Optimization, Vol. 33, No. 3 | 30 August 2023
- A stochastic variance reduction algorithm with Bregman distances for structured composite problemsOptimization, Vol. 72, No. 6 | 31 January 2022
- Block coordinate type methods for optimization and learningAnalysis and Applications, Vol. 21, No. 03 | 28 December 2022
- Block Mirror Stochastic Gradient Method For Stochastic OptimizationJournal of Scientific Computing, Vol. 94, No. 3 | 8 February 2023
- Unifying Framework for Accelerated Randomized Methods in Convex OptimizationFoundations of Modern Statistics | 17 July 2023
- Smoothed Variable Sample-Size Accelerated Proximal Methods for Nonsmooth Stochastic Convex ProgramsStochastic Systems, Vol. 12, No. 4 | 1 Dec 2022
- An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model IdentificationProceedings of the 31st ACM International Conference on Information & Knowledge Management | 17 October 2022
- On the local convergence of a stochastic semismooth Newton method for nonsmooth nonconvex optimizationScience China Mathematics, Vol. 65, No. 10 | 30 March 2022
- Stochastic batch size for adaptive regularization in deep network optimizationPattern Recognition, Vol. 129 | 1 Sep 2022
- A fully stochastic second-order trust region methodOptimization Methods and Software, Vol. 37, No. 3 | 25 November 2020
- Asynchronous variance-reduced block schemes for composite non-convex stochastic optimization: block-specific steplengths and adapted batch-sizesOptimization Methods and Software, Vol. 37, No. 1 | 13 April 2020
- Recent Theoretical Advances in Non-Convex OptimizationHigh-Dimensional Optimization and Probability | 5 April 2022
- Stochastic Mirror Descent for Low-Rank Tensor Decomposition Under Non-Euclidean LossesIEEE Transactions on Signal Processing, Vol. 70 | 1 Jan 2022
- Zeroth-order randomized block methods for constrained minimization of expectation-valued Lipschitz continuous functions2021 Seventh Indian Control Conference (ICC) | 20 Dec 2021
- A Robust Stochastic Quasi-Newton Method with the Application in Machine Learning2021 International Conference on Culture-oriented Science & Technology (ICCST) | 1 Nov 2021
- Distributed constraint-coupled optimization via primal decomposition over random time-varying graphsAutomatica, Vol. 131 | 1 Sep 2021
- Randomized Block Proximal Methods for Distributed Stochastic Big-Data OptimizationIEEE Transactions on Automatic Control, Vol. 66, No. 9 | 1 Sep 2021
- Fastest rates for stochastic mirror descent methodsComputational Optimization and Applications, Vol. 79, No. 3 | 9 June 2021
- Fiber-Sampled Stochastic Mirror Descent for Tensor Decomposition with β-DivergenceICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 6 Jun 2021
- An accelerated directional derivative method for smooth stochastic convex optimizationEuropean Journal of Operational Research, Vol. 290, No. 2 | 1 Apr 2021
- A Unified Convergence Analysis of Stochastic Bregman Proximal Gradient and Extragradient MethodsJournal of Optimization Theory and Applications, Vol. 188, No. 3 | 8 January 2021
- Efficient Search of First-Order Nash Equilibria in Nonconvex-Concave Smooth Min-Max ProblemsSIAM Journal on Optimization, Vol. 31, No. 4 | 21 October 2021
- A Method with Convergence Rates for Optimization Problems with Variational Inequality ConstraintsSIAM Journal on Optimization, Vol. 31, No. 3 | 31 August 2021
- Stochastic Bigger Subspace Algorithms for Nonconvex Stochastic OptimizationIEEE Access, Vol. 9 | 1 Jan 2021
- Efficiency of Coordinate Descent Methods for Structured Nonconvex OptimizationMachine Learning and Knowledge Discovery in Databases | 25 February 2021
- Point Process Estimation with Mirror Prox AlgorithmsApplied Mathematics & Optimization, Vol. 82, No. 3 | 26 November 2019
- A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With ApplicationsIEEE Transactions on Neural Networks and Learning Systems, Vol. 31, No. 11 | 1 Nov 2020
- On the Linear Convergence Rate of the Distributed Block Proximal MethodIEEE Control Systems Letters, Vol. 4, No. 3 | 1 Jul 2020
- Greedy coordinate descent method on non-negative quadratic programming2020 IEEE 11th Sensor Array and Multichannel Signal Processing Workshop (SAM) | 1 Jun 2020
- Stochastic Convex OptimizationFirst-order and Stochastic Optimization Methods for Machine Learning | 16 May 2020
- Markov chain block coordinate descentComputational Optimization and Applications, Vol. 75, No. 1 | 22 October 2019
- Distributed Constraint-Coupled Optimization over Random Time-Varying Graphs via Primal Decomposition and Block Subgradient Approaches2019 IEEE 58th Conference on Decision and Control (CDC) | 1 Dec 2019
- A Randomized Block Subgradient Approach to Distributed Big Data Optimization2019 IEEE 58th Conference on Decision and Control (CDC) | 1 Dec 2019
- Self-Tuned Mirror Descent Schemes for Smooth and Nonsmooth High-Dimensional Stochastic OptimizationIEEE Transactions on Automatic Control, Vol. 64, No. 10 | 1 Oct 2019
- Optimization-Based Calibration of Simulation Input ModelsOperations Research, Vol. 67, No. 5 | 1 Sep 2019
- A Stochastic Trust Region Algorithm Based on Careful Step NormalizationINFORMS Journal on Optimization, Vol. 1, No. 3 | 1 Jul 2019
- Randomized Primal–Dual Proximal Block Coordinate UpdatesJournal of the Operations Research Society of China, Vol. 7, No. 2 | 19 December 2018
- One method for minimization a convex Lipschitz-continuous function of two variables on a fixed squareComputer Research and Modeling, Vol. 11, No. 3 | 1 June 2019
- On the Convergence of Asynchronous Parallel Iteration with Unbounded DelaysJournal of the Operations Research Society of China, Vol. 7, No. 1 | 9 December 2017
- A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex OptimizationSIAM Journal on Optimization, Vol. 29, No. 4 | 19 November 2019
- Accelerated Stochastic Algorithms for Nonconvex Finite-Sum and Multiblock OptimizationSIAM Journal on Optimization, Vol. 29, No. 4 | 31 October 2019
- Conditional gradient type methods for composite nonlinear and stochastic optimizationMathematical Programming, Vol. 173, No. 1-2 | 24 January 2018
- Informative Feature Selection for Domain AdaptationIEEE Access, Vol. 7 | 1 Jan 2019
- On Stochastic Mirror-prox Algorithms for Stochastic Cartesian Variational Inequalities: Randomized Block Coordinate and Optimal Averaging SchemesSet-Valued and Variational Analysis, Vol. 26, No. 4 | 20 March 2018
- Stochastic Gradient Descent: Recent TrendsRecent Advances in Optimization and Modeling of Contemporary Problems | 19 October 2018
- A Fast Algorithm for Posterior Inference with Latent Dirichlet AllocationIntelligent Information and Database Systems | 14 February 2018
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic OptimizationSIAM Journal on Optimization, Vol. 27, No. 2 | 18 May 2017
- Accelerated Stochastic Block Coordinate Descent with Optimal SamplingProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining | 13 August 2016
- Block Stochastic Gradient Iteration for Convex and Nonconvex OptimizationSIAM Journal on Optimization, Vol. 25, No. 3 | 26 August 2015
View Options
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].