In this paper, we consider a class of optimization problems with orthogonality constraints, the feasible region of which is called the Stiefel manifold. Our new framework combines a function value reduction step with a correction step. Different from the existing approaches, the function value reduction step of our algorithmic framework searches along the standard Euclidean descent directions instead of the vectors in the tangent space of the Stiefel manifold, and the correction step further reduces the function value and guarantees a symmetric dual variable at the same time. We construct two types of algorithms based on this new framework. The first type is based on gradient reduction including the gradient reflection (GR) and the gradient projection (GP) algorithms. The other one adopts a columnwise block coordinate descent (CBCD) scheme with a novel idea for solving the corresponding CBCD subproblem inexactly. We prove that both GR/GP with a fixed step size and CBCD belong to our algorithmic framework, and any clustering point of the iterates generated by the proposed framework is a first-order stationary point. Preliminary experiments illustrate that our new framework is of great potential.


  1. orthogonality constraint
  2. Stiefel manifold
  3. Householder transformation
  4. gradient projection
  5. block coordinate descent

MSC codes

  1. 15A18
  2. 65F15
  3. 65K05
  4. 90C06

Get full access to this article

View all available purchase options and get full access to this article.


T. E. Abrudan, J. Eriksson, and V. Koivunen, Steepest descent algorithms for optimization under unitary matrix constraint, IEEE Trans. Signal Process., 56 (2008), pp. 1134--1147.
T. E. Abrudan, J. Eriksson, and V. Koivunen, Conjugate gradient algorithm for optimization under unitary matrix constraint, Signal Process., 89 (2009), pp. 1704--1714.
P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds, Princeton University Press, Princeton, NJ, 2009.
P.-A. Absil and J. Malick, Projection-like retractions on matrix manifolds, SIAM J. Optim., 22 (2012), pp. 135--158.
J. Barzilai and J. M. Borwein, Two-point step size gradient methods, IMA J. Numer. Anal., 8 (1988), pp. 141--148.
A. Caboussat, R. Glowinski, and V. Pons, An augmented Lagrangian approach to the numerical solution of a non-smooth eigenvalue problem, J. Numer. Math., 17 (2009), pp. 3--26.
Y.-H. Dai and R. Fletcher, Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming, Numer. Math., 100 (2005), pp. 21--47.
A. d'Aspremont, L. El Ghaoui, M. I. Jordan, and G. R. G. Lanckriet, A direct formulation for sparse PCA using semidefinite programming, SIAM Rev., 49 (2007), pp. 434--448.
E. D. Dolan and J. J. Moré, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), pp. 201--213.
A. Edelman, T. A. Arias, and S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl., 20 (1998), pp. 303--353.
L. Eldén and H. Park, A Procrustes problem on the Stiefel manifold, Numer. Math., 82 (1999), pp. 599--619.
C. Fraikin, Y. Nesterov, and P. Van Dooren, A gradient-type algorithm optimizing the coupling between matrices, Linear Algebra Appl., 429 (2008), pp. 1229--1242.
D. Goldfarb, Z. Wen, and W. Yin, A curvilinear search method for p-harmonic flows on spheres, SIAM J. Imaging Sci., 2 (2009), pp. 84--109.
I. Grubišić and R. Pietersz, Efficient rank reduction of correlation matrices, Linear Algebra Appl., 422 (2007), pp. 629--653.
W. Huang, P.-A. Absil and K. A. Gallivan, A Riemannian BFGS method for nonconvex optimization problems, in Numerical Mathematics and Advanced Applications, Lecture Notes Comput. Sci. Eng. 112, Springer, Cham, Switzerland, 2016, pp. 627--634, https://doi.org/10.1007/978-3-319-39929-4_60.
W. Huang, K. A. Gallivan and P.-A. Absil, A Broyden class of quasi-Newton methods for Riemannian optimization, SIAM J. Optim., 25 (2015), pp. 1660--1685.
B. Jiang and Y.-H. Dai, A framework of constraint preserving update schemes for optimization on Stiefel manifold, Math. Program., 153 (2015), pp. 535--575.
W. Kohn and L. J. Sham, Self-consistent equations including exchange and correlation effects, Phys. Rev. (2), 140 (1965), pp. A1133--1138.
R. Lai and S. Osher, A splitting method for orthogonality constrained problems, J. Sci. Comput., 58 (2014), pp. 431--449.
X. Liu, X. Wang, Z. Wen, and Y. Yuan, On the convergence of the self-consistent field iteration in Kohn--Sham density functional theory, SIAM J. Matrix Anal. Appl., 35 (2014), pp. 546--558.
X. Liu, Z. Wen, and Y. Zhang, Limited memory block Krylov subspace optimization for computing dominant singular value decompositions, SIAM J. Sci. Comput., 35 (2013), pp. A1641--A1668.
X. Liu, Z. Wen, and Y. Zhang, An efficient Gauss--Newton algorithm for symmetric low-rank product matrix approximations, SIAM J. Optim., 25 (2015), pp. 1571--1608.
J. H. Manton, Optimization algorithms exploiting unitary constraints, IEEE Trans. Signal Process., 50 (2002), pp. 635--650.
Y. Nishimori and S. Akaho, Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold, Neurocomputing, 67 (2005), pp. 106--135.
J. Nocedal and S. Wright, Numerical Optimization, Springer, Berlin, 2006.
B. Savas and L.-H. Lim, Quasi-Newton methods on Grassmannians and multilinear approximations of tensors, SIAM J. Sci. Comput., 32 (2010), pp. 3352--3393.
P. H. Schönemann, A generalized solution of the orthogonal Procrustes problem, Psychometrika, 31 (1966), pp. 1--10.
E. Stiefel, Richtungsfelder und fernparallelismus in n-dimensionalen mannigfaltigkeiten, Comment. Math. Helv., 8 (1935), pp. 305--353.
W. Sun and Y.-X. Yuan, Optimization Theory and Methods: Nonlinear Programming, Vol. 1, Springer, New York, 2006.
L. N. Trefethen and D. Bau, Numerical Linear Algebra, SIAM, Philadelphia, 1997.
M. Ulbrich, Z. Wen, C. Yang, D. Klöckner, and Z. Lu, A proximal gradient method for ensemble density functional theory, SIAM J. Sci. Comput., 37 (2015), pp. A1975--A2002.
Z. Wen, C. Yang, X. Liu, and Y. Zhang, Trace-penalty minimization for large-scale eigenspace computation, J. Sci. Comput., 66 (2016), pp. 1175--1203.
Z. Wen and W. Yin, A feasible method for optimization with orthogonality constraints, Math. Program., 142 (2013), pp. 397--434.
C. Yang, J. C. Meza, B. Lee, and L.-W. Wang, KSSOLV--a MATLAB toolbox for solving the Kohn-Sham equations, ACM Trans. Math. Software, 36 (2009), 10.
C. Yang, J. C. Meza, and L.-W. Wang, A constrained optimization algorithm for total energy minimization in electronic structure calculations, J. Comput. Phys., 217 (2006), pp. 709--721.
C. Yang, J. C. Meza, and L.-W. Wang, A trust region direct constrained minimization algorithm for the Kohn-Sham equation, SIAM J. Sci. Comput., 29 (2007), pp. 1854--1875.
H. Zhang and W. W. Hager, A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim., 14 (2004), pp. 1043--1056.
X. Zhang, J. Zhu, Z. Wen, and A. Zhou, Gradient type optimization methods for electronic structure calculations, SIAM J. Sci. Comput., 36 (2014), pp. C265--C289.
H. Zou, T. Hastie, and R. Tibshirani, Sparse principal component analysis, J. Comput. Graph, Statist., 15 (2006), pp. 265--286.

Information & Authors


Published In

cover image SIAM Journal on Optimization
SIAM Journal on Optimization
Pages: 302 - 332
ISSN (online): 1095-7189


Submitted: 13 October 2016
Accepted: 6 October 2017
Published online: 1 February 2018


  1. orthogonality constraint
  2. Stiefel manifold
  3. Householder transformation
  4. gradient projection
  5. block coordinate descent

MSC codes

  1. 15A18
  2. 65F15
  3. 65K05
  4. 90C06



Funding Information

Hong Kong Research Council : N PolyU504/14
Chinese Academy of Sciences https://doi.org/10.13039/501100002367
Chinese Academy of Sciences Key Project https://doi.org/10.13039/501100005151
National Natural Science Foundation of China https://doi.org/10.13039/501100001809 : 11471325, 91530204, 11622112, 11331012, 11461161005

Metrics & Citations



If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

View Options

View options


View PDF







Copy the content Link

Share with email

Email a colleague

Share on social media

On May 28, 2024, our site will enter Read Only mode for a limited time in order to complete a platform upgrade. As a result, the following functions will be temporarily unavailable: registering new user accounts, any updates to existing user accounts, access token activations, and shopping cart transactions. Contact [email protected] with any questions.