Tackling Box-Constrained Optimization via a New Projected Quasi-Newton Approach

Numerous scientific applications across a variety of fields depend on box-constrained convex optimization. Box-constrained problems therefore continue to attract research interest. We address box-constrained (strictly convex) problems by deriving two new quasi-Newton algorithms. Our algorithms are positioned between the projected-gradient [J. B. Rosen, J. SIAM, 8 (1960), pp. 181–217] and projected-Newton [D. P. Bertsekas, SIAM J. Control Optim., 20 (1982), pp. 221–246] methods. We also prove their convergence under a simple Armijo step-size rule. We provide experimental results for two particular box-constrained problems: nonnegative least squares (NNLS), and nonnegative Kullback–Leibler (NNKL) minimization. For both NNLS and NNKL our algorithms perform competitively as compared to well-established methods on medium-sized problems; for larger problems our approach frequently outperforms the competition.

  • [1]  D. P. Bertsekas, Projected Newton methods for optimization problems with simple constraints, SIAM J. Control Optim., 20 (1982), pp. 221–246. SJCODC 0363-0129 LinkISIGoogle Scholar

  • [2]  Google Scholar

  • [3]  M. Bierlaire, P. L. Toint and , and D. Tuyttens, On iterative algorithms for linear least squares problems with bound constraints, Linear Algebra Appl., 143 (1991), pp. 111–143. LAAPAW 0024-3795 CrossrefISIGoogle Scholar

  • [4]  R. Bro and  and S. De Jong, A fast non-negativity-constrained least squares algorithm, J. Chemometrics, 11 (1997), pp. 393–401. JOCHEU 0886-9383 CrossrefISIGoogle Scholar

  • [5]  R. Byrd, P. Lu, J. Nocedal and , and C. Zhu, A limited memory algorithm for bound constrained optimization, SIAM J. Sci. Comput., 16 (1995), pp. 1190–1208. SJOCE3 1064-8275 LinkISIGoogle Scholar

  • [6]  J. C. Dunn, Global and asymptotic convergence rate estimates for a class of projected gradient processes, SIAM J. Control Optim., 19 (1981), pp. 368–400. SJCODC 0363-0129 LinkISIGoogle Scholar

  • [7]  Google Scholar

  • [8]  Google Scholar

  • [9]  H. M. Hudson and  and R. S. Larkin, Accelerated image reconstruction using ordered subsets of projection data, IEEE Trans. Med. Imaging, 13 (1994), pp. 601–609. ITMID4 0278-0062 CrossrefISIGoogle Scholar

  • [10]  Google Scholar

  • [11]  Google Scholar

  • [12]  R. M. Lewitt and  and S. Matej, Overview of methods for image reconstruction from projections in emission computed tomography, Proc. of the IEEE, 91 (2003), pp. 1588–1611. IEEPAD 0018-9219 CrossrefISIGoogle Scholar

  • [13]  C.-J. Lin and  and J. J. Moré, Newton's method for large bound-constrained optimization problems, SIAM J. Optim., 9 (1999), pp. 1100–1127. SJOPE8 1052-6234 LinkISIGoogle Scholar

  • [14]  Google Scholar

  • [15]  M. Merritt and  and Y. Zhang, Interior-point gradient method for large-scale totally nonnegative least squares problems, J. Optim. Theory Appl., 126 (2005), pp. 191–202. JOTABN 0022-3239 CrossrefISIGoogle Scholar

  • [16]  N. Molinari, J.-F. Durand and , and R. Sabatier, Bounded optimal knots for regression splines, Comput. Statist. Data Anal., 45 (2004), pp. 159–178. CSDADW 0167-9473 CrossrefISIGoogle Scholar

  • [17]  J. J. Moré and  and D. C. Sorensen, Computing a trust region step, SIAM J. Sci. Statist. Comput., 3 (1983), pp. 553–572. SIJCD4 0196-5204 LinkISIGoogle Scholar

  • [18]  J. Nagy and  and Z. Strakos, Enforcing nonnegativity in image reconstruction algorithms, Math. Model., Estim. Imaging, 4121 (2000), pp. 182–190. CrossrefGoogle Scholar

  • [19]  Google Scholar

  • [20]  J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comp., 95 (1980), pp. 339–353. MCMPAF 0025-5718 Google Scholar

  • [21]  Google Scholar

  • [22]  Google Scholar

  • [23]  Google Scholar

  • [24]  J. B. Rosen, The gradient projection method for nonlinear programming. Part I. Linear constraints, J. Soc. Indus. Appl. Math., 8 (1960), pp. 181–217. JSIMAV 0368-4245 LinkISIGoogle Scholar

  • [25]  L. K. Saul, F. Sha and , and D. D. Lee, Statistical signal processing with nonnegativity constraints, in Proc. EuroSpeech 2003, 2 (2003), pp. 1001–1004. Google Scholar

  • [26]  Google Scholar

  • [27]  Y. Vardi and  and D. Lee, From image deblurring to optimal investment portfolios: Maximum likelihood solutions for positive linear problems, J. Roy. Statist. Soc. Ser. B, 55 (1993), pp. 569–612. Google Scholar