Methods and Algorithms for Scientific Computing

Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints


To construct a parallel approach for solving optimization problems with orthogonality constraints is usually regarded as an extremely difficult mission, due to the low scalability of the orthonormalization procedure. However, such a demand is particularly huge in some application areas such as materials computation. In this paper, we propose a proximal linearized augmented Lagrangian algorithm (PLAM) for solving optimization problems with orthogonality constraints. Unlike the classical augmented Lagrangian methods, in our algorithm, the prime variables are updated by minimizing a proximal linearized approximation of the augmented Lagrangian function; meanwhile the dual variables are updated by a closed-form expression which holds at any first-order stationary point. The orthonormalization procedure is only invoked once at the last step of the above-mentioned algorithm if high-precision feasibility is needed. Consequently, the main parts of the proposed algorithm can be parallelized naturally. We establish global subsequence convergence, worst-case complexity, and local convergence rate for PLAM under some mild assumptions. To reduce the sensitivity of the penalty parameter, we put forward a modification of PLAM, which is called parallelizable columnwise block minimization of PLAM (PCAL). Numerical experiments in serial illustrate that the novel updating rule for the Lagrangian multipliers significantly accelerates the convergence of PLAM and makes it comparable with the existent feasible solvers for optimization problems with orthogonality constraints, and the performance of PCAL does not highly rely on the choice of the penalty parameter. Numerical experiments under parallel environment demonstrate that PCAL attains good performance and high scalability in solving discretized Kohn--Sham total energy minimization problems.


  1. orthogonality constraint
  2. Stiefel manifold
  3. augmented Lagrangian method
  4. parallel computing

MSC codes

  1. 15A18
  2. 65F15
  3. 65K05
  4. 90C06

Get full access to this article

View all available purchase options and get full access to this article.


P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds, Princeton University Press, Princeton, NJ, 2009.
J. Barzilai and J. M. Borwein, Two-point step size gradient methods, IMA J. Numer. Anal., 8 (1988), pp. 141--148.
D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods, Academic Press, New York, 2014.
J. Bolte, S. Sabach, and M. Teboulle, Proximal alternating linearized minimization or nonconvex and nonsmooth problems, Math. Program., 146 (2014), pp. 459--494.
S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends. Mach. Learn., 3 (2011), pp. 1--122.
X. Dai, Z. Liu, L. Zhang, and A. Zhou, A conjugate gradient method for electronic structure calculations, SIAM J. Sci. Comput., 39 (2017), pp. A2702--A2740.
Y.-H. Dai and R. Fletcher, Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming, Numer. Math., 100 (2005), pp. 21--47.
E. D. Dolan and J. J. Moré, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), pp. 201--213.
A. Edelman, T. A. Arias, and S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl., 20 (1998), pp. 303--353.
B. Gao, X. Liu, X. Chen, and Y.-x. Yuan, A new first-order algorithmic framework for optimization problems with orthogonality constraints, SIAM J. Optim., 28 (2018), pp. 302--332.
B. Jiang and Y.-H. Dai, A framework of constraint preserving update schemes for optimization on Stiefel manifold, Math. Program., 153 (2015), pp. 535--575.
W. Kohn and L. J. Sham, Self-consistent equations including exchange and correlation effects, Phys. Rev., 140 (1965), A1133.
R. Lai and S. Osher, A splitting method for orthogonality constrained problems, J. Sci. Comput., 58 (2014), pp. 431--449.
Y. Li, Z. Wen, C. Yang, and Y. Yuan, A Semi-Smooth Newton Method for Solving Semidefinite Programs in Electronic Structure Calculations, 2017,
J. Liu, S. J. Wright, C. Ré, V. Bittorf, and S. Sridhar, An asynchronous parallel stochastic coordinate descent algorithm, J. Mach. Learn. Res., 16 (2015), pp. 285--322.
X. Liu, X. Wang, Z. Wen, and Y. Yuan, On the convergence of the self-consistent field iteration in Kohn--Sham density functional theory, SIAM J. Matrix Anal. Appl., 35 (2014), pp. 546--558.
X. Liu, Z. Wen, X. Wang, M. Ulbrich, and Y. Yuan, On the analysis of the discretized Kohn--Sham density functional theory, SIAM J. Numer. Anal., 53 (2015), pp. 1758--1785.
X. Liu, Z. Wen, and Y. Zhang, An efficient Gauss--Newton algorithm for symmetric low-rank product matrix approximations, SIAM J. Optim., 25 (2015), pp. 1571--1608.
M. A. Marques, A. Castro, G. F. Bertsch, and A. Rubio, octopus: A first-principles tool for excited electron--ion dynamics, Comput Phys. Commun., 151 (2003), pp. 60--78.
Y. Nishimori and S. Akaho, Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold, Neurocomputing, 67 (2005), pp. 106--135.
J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer, New York, 2006.
Z. Peng, Y. Xu, M. Yan, and W. Yin, ARock: An algorithmic framework for asynchronous parallel coordinate updates, SIAM J. Sci. Comput., 38 (2016), pp. A2851--A2879.
Z. Peng, M. Yan, and W. Yin, Parallel and distributed sparse optimization, in Proceedings of the Asilomar Conference on Signals, Systems and Computers, IEEE, 2013, pp. 659--646.
J. P. Perdew and A. Zunger, Self-interaction correction to density-functional approximations for many-electron systems, Phys. Rev., B, 23 (1981), 5048.
M. J. Powell, A method for nonlinear constraints in minimization problems, in Optimization, R. Fletcher, ed., Academic Press, London, 1969, pp. 283--298.
B. Recht, C. Re, S. Wright, and F. Niu, Hogwild: A lock-free approach to parallelizing stochastic gradient descent, in Advances in Neural Information Processing Systems, NIPS, 2011, pp. 693--701.
M. Ulbrich, Z. Wen, C. Yang, D. Klockner, and Z. Lu, A proximal gradient method for ensemble density functional theory, SIAM J. Sci. Comput., 37 (2015), pp. A1975--A2002.
Z. Wen, A. Milzarek, M. Ulbrich, and H. Zhang, Adaptive regularized self-consistent field iteration with exact Hessian for electronic structure calculation, SIAM J. Sci. Comput., 35 (2013), pp. A1299--A1324.
Z. Wen, C. Yang, X. Liu, and Y. Zhang, Trace-penalty minimization for large-scale eigenspace computation, J. Sci. Comput., 66 (2016), pp. 1175--1203.
Z. Wen and W. Yin, A feasible method for optimization with orthogonality constraints, Math. Program., 142 (2013), pp. 397--434.
C. Yang, W. Gao, and J. C. Meza, On the convergence of the self-consistent field iteration for a class of nonlinear eigenvalue problems, SIAM J. Matrix Anal. Appl., 30 (2009), pp. 1773--1788.
C. Yang, J. C. Meza, B. Lee, and L.-W. Wang, KSSOLV--a MATLAB toolbox for solving the Kohn-Sham equations, ACM Trans. Math. Softw, 36 (2009), p. 10.
C. Yang, J. C. Meza, and L.-W. Wang, A constrained optimization algorithm for total energy minimization in electronic structure calculations, J. Comput. Phys., 217 (2006), pp. 709--721.
C. Yang, J. C. Meza, and L.-W. Wang, A trust region direct constrained minimization algorithm for the Kohn--Sham equation, SIAM J. Sci. Comput., 29 (2007), pp. 1854--1875.
X. Zhang, J. Zhu, Z. Wen, and A. Zhou, Gradient type optimization methods for electronic structure calculations, SIAM J. Sci. Comput., 36 (2014), pp. C265--C289.

Information & Authors


Published In

cover image SIAM Journal on Scientific Computing
SIAM Journal on Scientific Computing
Pages: A1949 - A1983
ISSN (online): 1095-7197


Submitted: 19 October 2018
Accepted: 25 March 2019
Published online: 20 June 2019


  1. orthogonality constraint
  2. Stiefel manifold
  3. augmented Lagrangian method
  4. parallel computing

MSC codes

  1. 15A18
  2. 65F15
  3. 65K05
  4. 90C06



Funding Information

Chinese Academy of Sciences Key Project : QYZDJ-SSW-SYS010
National Natural Science Foundation of China : 11622112, 11471325, 91530204, 11688101, 11331012, 11461161005

Metrics & Citations



If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

View Options

View options


View PDF







Copy the content Link

Share with email

Email a colleague

Share on social media