A Globally Convergent Linearly Constrained Lagrangian Method for Nonlinear Optimization
Abstract
For optimization problems with nonlinear constraints, linearly constrained Lagrangian (LCL) methods solve a sequence of subproblems of the form "minimize an augmented Lagrangian function subject to linearized constraints." Such methods converge rapidly near a solution but may not be reliable from arbitrary starting points. Nevertheless, the well-known software package MINOS has proved effective on many large problems. Its success motivates us to derive a related LCL algorithm that possesses three important properties: it is globally convergent, the subproblem constraints are always feasible, and the subproblems may be solved inexactly. The new algorithm has been implemented in MATLAB, with an option to use either MINOS or SNOPT (Fortran codes) to solve the linearly constrained subproblems. Only first derivatives are required. We present numerical results on a subset of the COPS, HS, and CUTE test problems, which include many large examples. The results demonstrate the robustness and efficiency of the stabilized LCL procedure.
MSC codes
Keywords
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Dimitri Bertsekas, Constrained optimization and Lagrange multiplier methods, Computer Science and Applied Mathematics, Academic Press Inc. [Harcourt Brace Jovanovich Publishers], 1982xiii+395
2.
M. Best, J. Bräuninger, K. Ritter, S. Robinson, A globally and quadratically convergent algorithm for general nonlinear programming problems, Computing, 26 (1981), 141–153
3.
C. Bischof, A. Carle, P. Hovland, P. Khademi, and A. Mauer, ADIFOR 2.0 Users’ Guide, Tech. report 192, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, 1998.
4.
C. Bischof, L. Roh, and A. Mauer, ADIC: An extensible automatic differentiation tool for ANSI‐C, Software—Practice and Experience, 27 (1997), pp. 1427–1456.
5.
I. Bongartz, A. R. Conn, N. I. M. Gould, and Ph. L. Toint, CUTE: Constrained and unconstrained testing environment, ACM Trans. Math. Software, 21 (1995), pp. 123–160.
6.
Jürgen Bräuninger, A modification of Robinson’s algorithm for general nonlinear programming problems requiring only approximate solutions of subproblems with linear equality const, Lecture Notes in Control and Information Sci., Vol. 7, Springer, Berlin, 1978, 33–41
7.
J. Bräuninger, A globally convergent version of Robinson’s algorithm for general nonlinear programming problems without using derivatives, J. Optim. Theory Appl., 35 (1981), 195–216
8.
A. Brooke, D. Kendrick, and A. Meeraus, GAMS: A User’s Guide, The Scientific Press, Redwood City, CA, 1988.
9.
A. Conn, N. Gould, A. Sartenaer, Ph. Toint, Convergence properties of an augmented Lagrangian algorithm for optimization with a combination of general equality and linear constraints, SIAM J. Optim., 6 (1996), 674–703
10.
Andrew Conn, Nicholas Gould, Philippe Toint, A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds, SIAM J. Numer. Anal., 28 (1991), 545–572
11.
A. R. Conn, N. I. M. Gould, and Ph. L. Toint, LANCELOT: A Fortran Package for Large‐Scale Nonlinear Optimization (Release A), Springer‐Verlag, Berlin, 1991.
12.
A. Conn, N. Gould, Ph. Toint, On the number of inner iterations per outer iteration of a globally convergent algorithm for optimization with general nonlinear constraints and simple bounds, Pitman Res. Notes Math. Ser., Vol. 260, Longman Sci. Tech., Harlow, 1992, 49–68
13.
Andrew Conn, Nicholas Gould, Philippe Toint, Trust‐region methods, MPS/SIAM Series on Optimization, Society for Industrial and Applied Mathematics (SIAM), 2000xx+959
14.
Ron Dembo, Stanley Eisenstat, Trond Steihaug, Inexact Newton methods, SIAM J. Numer. Anal., 19 (1982), 400–408
15.
E. D. Dolan and J. J. Moré, Benchmarking Optimization Software with COPS, Tech. report ANL/MCS‐246, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, 2000 (revised 2001).
16.
Elizabeth Dolan, Jorge Moré, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), 201–213
17.
R. Fletcher, An penalty method for nonlinear constraints, SIAM, Philadelphia, PA, 1985, 26–40
18.
R. Fletcher, Practical methods of optimization, A Wiley‐Interscience Publication, John Wiley & Sons Ltd., 1987xiv+436
19.
R. Fourer, D. M. Gay, and B. W. Kernighan, AMPL: A Modeling Language for Mathematical Programming, 2nd ed., Duxbury Press/Brooks/Cole, Pacific Grove, CA, 2003.
20.
M. P. Friedlander, A Globally Convergent Linearly Constrained Lagrangian Method for Nonlineary Constrained Optimization, Ph.D. thesis, Stanford University, Stanford, CA, 2002.
21.
M. P. Friedlander and M. A. Saunders, An LCL implementation for nonlinear optimization, presented at the 18th International Symposium on Mathematical Programming, Copenhagen, Denmark, 2003;
available online from http://www.stanford.edu/group/SOL/talks.html.
22.
D. M. Gay, Hooking Your Solver to AMPL, Tech. report 97‐4‐06, Computing Sciences Research Center, Bell Laboratories, Murray Hill, NJ, 1997.
23.
P. E. Gill, private communication, 2002.
24.
P. E. Gill, W. Murray, and M. A. Saunders, User’s Guide for SNOPT 5.3: A Fortran Package for Large‐scale Nonlinear Programming, Numerical Analysis Report 97‐5, Department of Mathematics, University of California, San Diego, La Jolla, CA, 1997.
25.
Philip Gill, Walter Murray, Michael Saunders, SNOPT: an SQP algorithm for large‐scale constrained optimization, SIAM J. Optim., 12 (2002), 979–1006
26.
Philip Gill, Walter Murray, Margaret Wright, Practical optimization, Academic Press Inc. [Harcourt Brace Jovanovich Publishers], 1981xvi+401
27.
N. I. M. Gould, D. Orban, and Ph. L. Toint, CUTEr and SifDec: A constrained and unconstrained testing environment, revisited, ACM Trans. Math. Software, 29 (2003), pp. 373–394.
28.
S. Han, O. Mangasarian, Exact penalty functions in nonlinear programming, Math. Programming, 17 (1979), 251–269
29.
Magnus Hestenes, Multiplier and gradient methods, J. Optimization Theory Appl., 4 (1969), 303–320
30.
Willi Hock, Klaus Schittkowski, Test examples for nonlinear programming codes, Lecture Notes in Economics and Mathematical Systems, Vol. 187, Springer‐Verlag, 1981v+177
31.
Olvi Mangasarian, Nonlinear programming, McGraw‐Hill Book Co., 1969xiii+220
32.
N. Maratos, Exact Penalty Function Algorithms for Finite Dimensional and Optimization Problems, Ph.D. thesis, Imperial College of Science and Technology, London, UK, 1978.
33.
MathWorks, MATLAB User’s Guide, The MathWorks, Natick, MA, 1992.
34.
MathWorks, MATLAB: External Interfaces, The MathWorks, Natick, MA, 1995.
35.
B. Murtagh, M. Saunders, Large‐Scale linearly constrained optimization, Math. Programming, 14 (1978), 41–72
36.
Bruce Murtagh, Michael Saunders, A projected Lagrangian algorithm and its implementation for sparse nonlinear constraints, Math. Programming Stud., (1982), 84–117, Algorithms for constrained minimization of smooth nonlinear functions
37.
Jorge Nocedal, Stephen Wright, Numerical optimization, Springer Series in Operations Research, Springer‐Verlag, 1999xxii+636
38.
J. Ortega, W. Rheinboldt, Iterative solution of nonlinear equations in several variables, Academic Press, 1970xx+572
39.
M. Powell, A method for nonlinear constraints in minimization problems, Academic Press, London, 1969, 283–298
40.
Stephen Robinson, A quadratically‐convergent algorithm for general nonlinear programming problems, Math. Programming, 3 (1972), 145–156
41.
S. M. Robinson, Perturbed Kuhn‐Tucker points and rates of convergence for a class of nonlinear‐programming algorithms, Math. Program., 7 (1974), pp. 1–16.
42.
J. Rosen, Two‐phase algorithm for nonlinear constraint problems, Academic Press, New York, 1978, 97–124
43.
J. Rosen, J. Kreuser, A gradient projection algorithm for non‐linear constraints, Academic Press, London, 1972, 297–300
44.
G. Van der Hoek, Asymptotic properties of reduction methods applying linearly equality constrained reduced problems, Math. Programming Stud., (1982), 162–189, Algorithms for constrained minimization of smooth nonlinear functions
45.
R. Vanderbei, Benchmarks for Nonlinear Optimization, http://www.princeton.edu/∼rvdb/bench.html (December 2002).
Information & Authors
Information
Published In
SIAM Journal on Optimization
Pages: 863 - 897
ISSN (online): 1095-7189
Copyright
Copyright © 2005 Society for Industrial and Applied Mathematics.
History
Published online: 17 February 2012
MSC codes
Keywords
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- A new Lagrangian-based first-order method for nonconvex constrained optimizationOperations Research Letters, Vol. 51, No. 3 | 1 May 2023
- Complexity of an inexact proximal-point penalty method for constrained smooth non-convex optimizationComputational Optimization and Applications, Vol. 82, No. 1 | 3 March 2022
- Battery-Conscious, Economic, and Prioritization-Based Electric Vehicle Residential SchedulingEnergies, Vol. 15, No. 10 | 19 May 2022
- Sequential Linearization Method for Bound-Constrained Mathematical Programs with Complementarity ConstraintsSIAM Journal on Optimization, Vol. 32, No. 1 | 5 January 2022
- Constrained Nonlinear Optimization Methods: An OverviewModern Numerical Nonlinear Optimization | 19 October 2022
- Stabilized Optimization Via an NCL AlgorithmNumerical Analysis and Optimization | 1 June 2018
- Penalty dual decomposition method with application in signal processing2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 1 Mar 2017
- IntroductionContinuous Nonlinear Optimization for Engineering Applications in GAMS Technology | 9 November 2017
- An Augmented Lagrangian Based Algorithm for Distributed NonConvex OptimizationSIAM Journal on Optimization, Vol. 26, No. 2 | 3 May 2016
- A Sequential Method for a Class of Stable Mathematical Programming ProblemsSIAM Journal on Optimization, Vol. 26, No. 2 | 1 June 2016
- Augmented Likelihood Image ReconstructionIEEE Transactions on Medical Imaging, Vol. 35, No. 1 | 1 Jan 2016
- Critical Lagrange multipliers: what we currently know about them, how they spoil our lives, and what we can do about itTOP, Vol. 23, No. 1 | 27 February 2015
- Newton-Type Methods: A Broader ViewJournal of Optimization Theory and Applications, Vol. 164, No. 2 | 28 May 2014
- STABILIZED SEQUENTIAL QUADRATIC PROGRAMMING: A SURVEYPesquisa Operacional, Vol. 34, No. 3 | 1 Dec 2014
- Manifold learning by preserving distance ordersPattern Recognition Letters, Vol. 38 | 1 Mar 2014
- On R-linear convergence of semi-monotonic inexact augmented Lagrangians for bound and equality constrained quadratic programming problems with applicationComputers & Mathematics with Applications, Vol. 67, No. 3 | 1 Feb 2014
- Robust Face Recognition by Group Sparse Representation That Uses Samples from List of SubjectsArtificial Intelligence and Soft Computing | 1 Jan 2014
- Equations and Unconstrained OptimizationNewton-Type Methods for Optimization and Variational Problems | 22 January 2014
- Constrained Optimization: Local MethodsNewton-Type Methods for Optimization and Variational Problems | 22 January 2014
- A practical relative error criterion for augmented LagrangiansMathematical Programming, Vol. 141, No. 1-2 | 24 March 2012
- Abstract Newtonian Frameworks and Their ApplicationsSIAM Journal on Optimization, Vol. 23, No. 4 | 3 December 2013
- An Optimal Control Approach to Find Sparse Data for Laplace InterpolationEnergy Minimization Methods in Computer Vision and Pattern Recognition | 1 Jan 2013
- Multiplier methods for optimization problems with Lipschitzian derivativesComputational Mathematics and Mathematical Physics, Vol. 52, No. 12 | 14 December 2012
- Sequential Quadratic Programming MethodsMixed Integer Nonlinear Programming | 15 November 2011
- A primal-dual augmented LagrangianComputational Optimization and Applications, Vol. 51, No. 1 | 3 August 2010
- On attraction of linearly constrained Lagrangian methods and of stabilized and quasi-Newton SQP methods to critical multipliersMathematical Programming, Vol. 126, No. 2 | 16 April 2009
- Software For Nonlinearly Constrained OptimizationWiley Encyclopedia of Operations Research and Management Science | 15 February 2011
- Foundations of Constrained OptimizationWiley Encyclopedia of Operations Research and Management Science | 15 February 2011
- Inexact Josephy–Newton framework for generalized equations and its applications to local analysis of Newtonian methods for constrained optimizationComputational Optimization and Applications, Vol. 46, No. 2 | 13 June 2009
- Convergence Properties of Modified and Partially-Augmented Lagrangian Methods for Mathematical Programs with Complementarity ConstraintsJournal of Optimization Theory and Applications, Vol. 145, No. 3 | 29 January 2010
- Optimal Management of a Bioreactor for Eutrophicated Water Treatment: A Numerical ApproachJournal of Scientific Computing, Vol. 43, No. 1 | 31 December 2009
- Numerical optimization for constrained image registrationNumerical Linear Algebra with Applications, Vol. 17, No. 2-3 | 5 March 2010
- A Truncated SQP Method Based on Inexact Interior-Point Solutions of SubproblemsSIAM Journal on Optimization, Vol. 20, No. 5 | 13 July 2010
- Sharp Primal Superlinear Convergence Results for Some Newtonian Methods for Constrained OptimizationSIAM Journal on Optimization, Vol. 20, No. 6 | 9 November 2010
- Global and Finite Termination of a Two-Phase Augmented Lagrangian Filter Method for General Quadratic ProgramsSIAM Journal on Scientific Computing, Vol. 30, No. 4 | 23 April 2008
- Parameter range reduction for ODE models using cumulative backward differentiation formulasJournal of Computational and Applied Mathematics, Vol. 203, No. 1 | 1 Jun 2007
- A two-sided relaxation scheme for Mathematical Programs with Equilibrium ConstraintsSIAM Journal on Optimization, Vol. 16, No. 2 | 28 July 2006