Abstract

Sequential quadratic programming (SQP) methods have proved highly effective for solving constrained optimization problems with smooth nonlinear functions in the objective and constraints. Here we consider problems with general inequality constraints (linear and nonlinear). We assume that first derivatives are available and that the constraint gradients are sparse. Second derivatives are assumed to be unavailable or too expensive to calculate.
We discuss an SQP algorithm that uses a smooth augmented Lagrangian merit function and makes explicit provision for infeasibility in the original problem and the QP subproblems. The Hessian of the Lagrangian is approximated using a limited-memory quasi-Newton method.
SNOPT is a particular implementation that uses a reduced-Hessian semidefinite QP solver (SQOPT) for the QP subproblems. It is designed for problems with many thousands of constraints and variables but is best suited for problems with a moderate number of degrees of freedom (say, up to 2000). Numerical results are given for most of the CUTEr and COPS test collections (about 1020 examples of all sizes up to 40000 constraints and variables, and up to 20000 degrees of freedom).

MSC codes

  1. 49J20
  2. 49J15
  3. 49M37
  4. 49D37
  5. 65F05
  6. 65K05
  7. 90C30

Keywords

  1. large-scale optimization
  2. nonlinear programming
  3. nonlinear inequality constraints
  4. sequential quadratic programming
  5. quasi-Newton methods
  6. limited-memory methods

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
AMPL Home Page, http://www.ampl.com.
2.
ARKI Consulting & Development A/S, http://www.conopt.com.
3.
Richard Bartels, A stabilization of the simplex method, Numer. Math., 16 (1970/1971), 414–434
4.
Richard Bartels, A penalty linear programming method using reduced‐gradient basis‐exchange techniques, Linear Algebra Appl., 29 (1980), 17–32
5.
J. Betts, P. Frank, A sparse nonlinear optimization algorithm, J. Optim. Theory Appl., 82 (1994), 519–541
6.
Lorenz Biegler, Jorge Nocedal, Claudia Schmid, A reduced Hessian method for large‐scale constrained optimization, SIAM J. Optim., 5 (1995), 314–347
7.
M. Biggs, Constrained minimization using recursive equality quadratic programming, Academic Press, London, 1972, 411–428
8.
Paul Boggs, Anthony Kearsley, Jon Tolle, A global convergence analysis of an algorithm for large‐scale nonlinear optimization problems, SIAM J. Optim., 9 (1999), 833–862, Dedicated to John E. Dennis, Jr., on his 60th birthday
9.
Paul Boggs, Anthony Kearsley, Jon Tolle, A practical algorithm for general large scale nonlinear optimization problems, SIAM J. Optim., 9 (1999), 755–778
10.
A. Bondarenko, D. Bortz, and J. J. Moré, COPS: Large‐Scale Nonlinearly Constrained Optimization Problems, Technical Report ANL/MCS‐TM‐237, Mathematics and Computer Science division, Argonne National Laboratory, Argonne, IL, 1998.
Revised October 1999.
11.
I. Bongartz, A. R. Conn, N. I. M. Gould, and Ph. L. Toint, CUTE: Constrained and unconstrained testing environment, ACM Trans. Math. Software, 21 (1995), pp. 123–160.
12.
I. Bongartz, A. R. Conn, N. I. M. Gould, M. A. Saunders, and Ph. L. Toint, A Numerical Comparison between the LANCELOT and MINOS Packages for Large‐Scale Constrained Optimization, Report 97/13, Département de Mathématique, Facultés Universitaires de Namur, 1997.
13.
I. Bongartz, A. R. Conn, N. I. M. Gould, M. A. Saunders, and Ph. L. Toint, A Numerical Comparison between the LANCELOT and MINOS Packages for Large‐Scale Constrained Optimization: The Complete Numerical Results, Report 97/14, Département de Mathématique, Facultés Universitaires de Namur, 1997.
14.
K. Brodlie, A. Gourlay, J. Greenstadt, Rank‐one and rank‐two corrections to positive definite matrices expressed in product form, J. Inst. Math. Appl., 11 (1973), 73–82
15.
G. G. Brown and G. W. Graves, Elastic Programming: A New Approach to Large‐Scale Mixed‐Integer Optimization, 1975.
Presented at the ORSA/TIMS meeting, Las Vegas, NV.
16.
G. G. Brown and G. W. Graves, The XS Mathematical Programming System, working paper, Department of Operations Research, Naval Postgraduate School, Monterey, CA, 1975.
17.
A. Buckley, A. Lenir, QN‐like variable storage conjugate gradients, Math. Programming, 27 (1983), 155–175
18.
A. Buckley and A. LeNir, BBVSCG—a variable storage algorithm for function minimization, ACM Trans. Math. Software, 11 (1985), pp. 103–119.
19.
R. H. Byrd, Robust Trust‐Region Methods for Constrained Optimization. Presented at the SIAM Conference on Optimization, Houston, TX, 1987.
20.
Richard Byrd, Jean Gilbert, Jorge Nocedal, A trust region method based on interior point techniques for nonlinear programming, Math. Program., 89 (2000), 149–185
21.
Richard Byrd, Mary Hribar, Jorge Nocedal, An interior point algorithm for large‐scale nonlinear programming, SIAM J. Optim., 9 (1999), 877–900, Dedicated to John E. Dennis, Jr., on his 60th birthday
22.
Richard Byrd, Peihuang Lu, Jorge Nocedal, Ci Zhu, A limited memory algorithm for bound constrained optimization, SIAM J. Sci. Comput., 16 (1995), 1190–1208
23.
Richard Byrd, Jorge Nocedal, An analysis of reduced Hessian methods for constrained optimization, Math. Programming, 49 (1990/91), 285–323
24.
Richard Byrd, Jorge Nocedal, Robert Schnabel, Representations of quasi‐Newton matrices and their use in limited memory methods, Math. Programming, 63 (1994), 129–156
25.
Andrew Conn, Constrained optimization using a nondifferentiable penalty function, SIAM J. Numer. Anal., 10 (1973), 760–784
26.
A. Conn, Linear programming via a nondifferentiable penalty function, SIAM J. Numer. Anal., 13 (1976), 145–154
27.
A. Conn, N. Gould, Ph. Toint, LANCELOT, Springer Series in Computational Mathematics, Vol. 17, Springer‐Verlag, 1992xx+330, A Fortran package for large‐scale nonlinearoptimization (release A)
28.
Andrew Conn, Nicholas Gould, Philippe Toint, Trust‐region methods, MPS/SIAM Series on Optimization, Society for Industrial and Applied Mathematics (SIAM), 2000xx+959
29.
Ron Dembo, Stanley Eisenstat, Trond Steihaug, Inexact Newton methods, SIAM J. Numer. Anal., 19 (1982), 400–408
30.
J. Dennis, Jr., Robert Schnabel, A new derivation of symmetric positive definite secant updates, Academic Press, New York, 1981, 167–199
31.
E. D. Dolan and J. J. Moré, Benchmarking Optimization Software with COPS, Technical Memorandum ANL/MCS‐TM‐246, Argonne National Laboratory, Argonne, IL, 2000.
32.
Elizabeth Dolan, Jorge Moré, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), 201–213
33.
E. D. Dolan, J. J. Moré, and T. S. Munson, Benchmarking Optimization Software with COPS 3.0, Technical Memorandum ANL/MCS‐TM‐273, Argonne National Laboratory, Argonne, IL, 2004.
34.
Arne Drud, CONOPT: a GRG code for large sparse dynamic nonlinear optimization problems, Math. Programming, 31 (1985), 153–191
35.
I. S. Duff, MA28—A Set of Fortran Subroutines for Sparse Unsymmetric Linear Equations, Report AERE R8730, Atomic Energy Research Establishment, Harwell, England, 1977.
36.
S. K. Eldersveld, Large‐Scale Sequential Quadratic Programming Algorithms, Ph.D. thesis, Department of Operations Research, Stanford University, Stanford, CA, 1991.
37.
S. I. Feldman, D. M. Gay, M. W. Maimone, and N. L. Schryer, A Fortran‐to‐C Converter, Computing Science Technical Report 149, AT&T Bell Laboratories, Murray Hill, NJ, 1990.
38.
R. Fletcher, An l1 penalty method for nonlinear constraints, SIAM, Philadelphia, PA, 1985, 26–40
39.
R. Fletcher, Practical methods of optimization, A Wiley‐Interscience Publication, John Wiley & Sons Ltd., 1987xiv+436
40.
R. Fletcher and S. Leyffer, User Manual for FilterSQP, Technical Report NA/181, Department of Mathematics, University of Dundee, Scotland, 1998.
41.
Roger Fletcher, Sven Leyffer, Nonlinear programming without a penalty function, Math. Program., 91 (2002), 239–269
42.
Robert Fourer, A simplex algorithm for piecewise‐linear programming. I. Derivation and proof, Math. Programming, 33 (1985), 204–233
43.
Robert Fourer, A simplex algorithm for piecewise‐linear programming. II. Finiteness, feasibility and degeneracy, Math. Programming, 41 (1988), 281–315
44.
Robert Fourer, A simplex algorithm for piecewise‐linear programming. III. Computational analysis and applications, Math. Programming, 53 (1992), 213–235
45.
R. Fourer, D. M. Gay, and B. W. Kernighan, AMPL: A Modeling Language for Mathematical Programming, The Scientific Press, San Francisco, CA, 1993.
46.
R. Fourer, D. M. Gay, and B. W. Kernighan, AMPL: A Modeling Language for Mathematical Programming, Brooks/Cole—Thomson Learning, Pacific Grove, CA, 2003.
47.
M. P. Friedlander and M. A. Saunders, A globally convergent linearly constrained Lagrangian method for nonlinear optimization, SIAM J. Optim., to appear.
48.
GAMS Development Corp., http://www.gams.com.
49.
Jean Gilbert, Claude Lemaréchal, Some numerical experiments with variable‐storage quasi‐Newton algorithms, Math. Programming, 45 (1989), 407–435
50.
P. E. Gill, G. H. Golub, W. Murray, and M. A. Saunders, Methods for modifying matrix factorizations, Math. Comput., 28 (1974), pp. 505–535.
51.
Philip Gill, Michael Leonard, Limited‐memory reduced‐Hessian methods for large‐scale unconstrained optimization, SIAM J. Optim., 14 (2003), 380–401
52.
Philip Gill, Walter Murray, The computation of Lagrange‐multiplier estimates for constrained minimization, Math. Programming, 17 (1979), 32–60
53.
P. E. Gill, W. Murray, and M. A. Saunders, User’s Guide for SQOPT 5.3: A Fortran Package for Large‐Scale Linear and Quadratic Programming, Numerical Analysis Report NA 97‐4, Department of Mathematics, University of California, San Diego, La Jolla, CA, 1997.
54.
Philip Gill, Walter Murray, Michael Saunders, SNOPT: an SQP algorithm for large‐scale constrained optimization, SIAM J. Optim., 12 (2002), 979–1006
55.
P. E. Gill, W. Murray, and M. A. Saunders, User’s Guide for SNOPT 7.1: A Fortran Package for Large‐Scale Nonlinear Programming, Numerical Analysis Report NA 04‐1, Department of Mathematics, University of California, San Diego, La Jolla, CA, 2004.
56.
Philip Gill, Walter Murray, Michael Saunders, Margaret Wright, Sparse matrix methods in optimization, SIAM J. Sci. Statist. Comput., 5 (1984), 562–589
57.
P. E. Gill, W. Murray, M. A. Saunders, and M. H. Wright, User’s Guide for NPSOL (Version 4.0): A Fortran Package for Nonlinear Programming, Report SOL 86‐2, Department of Operations Research, Stanford University, Stanford, CA, 1986.
58.
Philip Gill, Walter Murray, Michael Saunders, Margaret Wright, Maintaining LU factors of a general sparse matrix, Linear Algebra Appl., 88/89 (1987), 239–270
59.
P. Gill, W. Murray, M. Saunders, M. Wright, Inertia‐controlling methods for general quadratic programming, SIAM Rev., 33 (1991), 1–36
60.
Panos Pardalos, Advances in optimization and parallel computing, North‐Holland Publishing Co., 1992xxiv+316, Honorary volume on the occasion of J. B. Rosen’s 70th birthday
61.
Philip Gill, Walter Murray, Margaret Wright, Practical optimization, Academic Press Inc. [Harcourt Brace Jovanovich Publishers], 1981xvi+401
62.
Donald Goldfarb, Factorized variable metric methods for unconstrained optimization, Math. Comp., 30 (1976), 796–811
63.
N. I. M. Gould, D. Orban, and Ph. L. Toint, CUTEr and SifDec: A constrained and unconstrained testing environment, revisited, ACM Trans. Math. Software, 29 (2003), pp. 373–394.
64.
Nicholas Gould, Dominique Orban, Philippe Toint, GALAHAD, a library of thread‐safe Fortran 90 packages for large‐scale nonlinear optimization, ACM Trans. Math.Software, 29 (2003), 353–372
65.
Nicholas Gould, Philippe Toint, SQP methods for large‐scale nonlinear programming, Kluwer Acad. Publ., Boston, MA, 2000, 149–178
66.
S. P. Han, Superlinearly convergent variable metric algorithms for general nonlinear programming problems, Math. Program., 11 (1976), pp. 263–282.
67.
C. R. Hargraves and S. W. Paris, Direct trajectory optimization using nonlinear programming and collocation, J. Guidance, Control, and Dynamics, 10 (1987), pp. 338–348.
68.
Nicholas Higham, FORTRAN codes for estimating the one‐norm of a real or complex matrix, with applications to condition estimation, ACM Trans. Math. Software, 14 (1988), 381–396
69.
Marucha Lalee, Jorge Nocedal, Todd Plantenga, On the implementation of an algorithm for large‐scale equality constrained optimization, SIAM J. Optim., 8 (1998), 682–706
70.
Dong Liu, Jorge Nocedal, On the limited memory BFGS method for large scale optimization, Math. Programming, 45 (1989), 503–528
71.
J. Morales, A numerical study of limited memory BFGS methods, Appl. Math. Lett., 15 (2002), 481–487
72.
José Morales, Jorge Nocedal, Automatic preconditioning by limited memory quasi‐Newton updating, SIAM J. Optim., 10 (2000), 1079–1096
73.
Walter Murray, Sequential quadratic programming methods for large‐scale problems, Comput. Optim. Appl., 7 (1997), 127–142, Computational issues in high performance software for nonlinear optimization (Capri, 1995)
74.
Walter Murray, Francisco Prieto, A sequential quadratic programming algorithm using an incomplete solution of the subproblem, SIAM J. Optim., 5 (1995), 590–640
75.
B. Murtagh, M. Saunders, Large‐Scale linearly constrained optimization, Math. Programming, 14 (1978), 41–72
76.
Bruce Murtagh, Michael Saunders, A projected Lagrangian algorithm and its implementation for sparse nonlinear constraints, Math. Programming Stud., (1982), 84–117, Algorithms for constrained minimization of smooth nonlinear functions
77.
B. A. Murtagh and M. A. Saunders, MINOS 5.5 User’s Guide, Report SOL 83‐20R, Department of Operations Research, Stanford University, Stanford, CA, 1998 (revised).
78.
Jorge Nocedal, Stephen Wright, Numerical optimization, Springer Series in Operations Research, Springer‐Verlag, 1999xxii+636
79.
E. O. Omojokun, Trust Region Algorithms for Nonlinear Equality and Inequality Constraints, Ph.D. thesis, Department of Computer Science, University of Colorado, Boulder, CO, 1989.
80.
M. J. O’Sullivan, New Methods for Dynamic Programming over an Infinite Time Horizon, Ph.D. thesis, Department of Management Science and Engineering, Stanford University, Stanford, CA, 2002.
81.
M. J. O’Sullivan and M. A. Saunders, Sparse Rank‐Revealing LU Factorization. Presented at the 7th SIAM Conference on Optimization, Toronto, Canada, 2002;
available online from http://www.stanford.edu/group/SOL/talks.html.
82.
M. J. O’Sullivan and M. A. Saunders, Sparse Rank‐Revealing LU Factorization (via Threshold Complete Pivoting and Threshold Rook Pivoting). Presented at Householder Symposium XV on Numerical Linear Algebra, Peebles, Scotland, 2002;
available online from http://www.stanford.edu/group/SOL/talks.html.
83.
C. C. Paige and M. A. Saunders, Solution of sparse indefinite systems of linear equations, SIAM J. Numer. Anal., 12 (1975), pp. 617–629.
84.
Christopher Paige, Michael Saunders, LSQR: an algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software, 8 (1982), 43–71
85.
M. J. D. Powell, A Fast Algorithm for Nonlinearly Constrained Optimization Calculations, Technical Report 77/NA 2, Department of Applied Mathematics and Theoretical Physics, University of Cambridge, England, 1977.
86.
M. Powell, Algorithms for nonlinear constraints that use Lagrangian functions, Math. Programming, 14 (1978), 224–248
87.
M. Powell, The convergence of variable metric methods for nonlinearly constrained optimization calculations, Academic Press, New York, 1978, 27–63
88.
J. K. Reid, Fortran Subroutines for Handling Sparse Linear Programming Bases, Report AERE R8269, Atomic Energy Research Establishment, Harwell, England, 1976.
89.
J. Reid, A sparsity‐exploiting variant of the Bartels‐Golub decomposition for linear programming bases, Math. Programming, 24 (1982), 55–69
90.
Stephen Robinson, A quadratically‐convergent algorithm for general nonlinear programming problems, Math. Programming, 3 (1972), 145–156
91.
R. Sargent, M. Ding, A new SQP algorithm for large‐scale nonlinear programming, SIAM J. Optim., 11 (2000/01), 716–747
92.
K. Schittkowski, NLPQL: a FORTRAN subroutine solving constrained nonlinear programming problems, Ann. Oper. Res., 5 (1986), 485–500
93.
P. Spellucci, Han’s method without solving QP, Lecture Notes in Control and Information Sci., Vol. 30, Springer, Berlin, 1981, 123–141
94.
P. Spellucci, A new technique for inconsistent QP problems in the SQP method, Math. Methods Oper. Res., 47 (1998), 355–400
95.
P. Spellucci, An SQP method for general nonlinear programs using only equality constrained subproblems, Math. Programming, 82 (1998), 413–448
96.
R. Tapia, A stable approach to Newton’s method for general mathematical programming problems in Rn, J. Optimization Theory Appl., 14 (1974), 453–476, Collection of articles dedicated to Magnus R. Hestenes
97.
I.‐B. Tjoa and L. T. Biegler, Simultaneous solution and optimization strategies for parameter estimation of differential algebraic equation systems, Ind. Eng. Chem. Res., 30 (1991), pp. 376–385.
98.
Kaoru Tone, Revisions of constraint approximations in the successive QP method for nonlinear programming problems, Math. Programming, 26 (1983), 144–152
99.
G. Van der Hoek, Asymptotic properties of reduction methods applying linearly equality constrained reduced problems, Math. Programming Stud., (1982), 162–189, Algorithms for constrained minimization of smooth nonlinear functions
100.
Robert Vanderbei, David Shanno, An interior‐point algorithm for nonconvex nonlinear programming, Comput. Optim. Appl., 13 (1999), 231–252, Computational optimization—a tribute to Olvi Mangasarian, Part II
101.
A. Wächter, L. T. Biegler, Y.‐D. Lang, and A. Raghunathan, IPOPT: An Interior Point Algorithm for Large‐Scale Nonlinear Optimization, http://www.coin‐or.org, 2002.
102.
R. B. Wilson, A Simplicial Method for Convex Programming, Ph.D. thesis, Harvard University, 1963.
103.
Ciyou Zhu, Richard Byrd, Peihuang Lu, Jorge Nocedal, Algorithm 778: L‐BFGS‐B: Fortran subroutines for large‐scale bound‐constrained optimization, ACM Trans. Math. Software, 23 (1997), 550–560

Information & Authors

Information

Published In

cover image SIAM Review
SIAM Review
Pages: 99 - 131
ISSN (online): 1095-7200

History

Published online: 4 August 2006

MSC codes

  1. 49J20
  2. 49J15
  3. 49M37
  4. 49D37
  5. 65F05
  6. 65K05
  7. 90C30

Keywords

  1. large-scale optimization
  2. nonlinear programming
  3. nonlinear inequality constraints
  4. sequential quadratic programming
  5. quasi-Newton methods
  6. limited-memory methods

Authors

Affiliations

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By