A PAC Approach to Application-Specific Algorithm Selection

The best algorithm for a computational problem generally depends on the “relevant inputs,” a concept that depends on the application domain and often defies formal articulation. While there is a large body of literature on empirical approaches to selecting the best algorithm for a given application domain, there has been surprisingly little theoretical analysis of the problem. This paper adapts concepts from statistical and online learning theory to reason about application-specific algorithm selection. Our models capture several state-of-the-art empirical and theoretical approaches to the problem, ranging from self-improving algorithms to empirical performance models, and our results identify conditions under which these approaches are guaranteed to perform well. We present one framework that models algorithm selection as a statistical learning problem, and our work here shows that dimension notions from statistical learning theory, historically used to measure the complexity of classes of binary- and real-valued functions, are relevant in a much broader algorithmic context. We also study the online version of the algorithm selection problem, and give possibility and impossibility results for the existence of no-regret learning algorithms.

  • 1.  N. Ailon, B. Chazelle, S. Comandur, and D. Liu, Self-improving algorithms, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2006, pp. 261--270. Google Scholar

  • 2.  M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge University Press, Cambridge, 1999. Google Scholar

  • 3.  V. Arya N. Garg R. Khandekar A. Meyerson K. Munagala and  V. Pandit , Local search heuristics for $k$-median and facility location problems , SIAM J. Comput. , 33 ( 2004 ), pp. 544 -- 562 . LinkISIGoogle Scholar

  • 4.  J. Bergstra and  Y. Bengio , Random search for hyper-parameter optimization , J. Mach. Learn. Res. , 13 ( 2012 ), pp. 281 -- 305 . ISIGoogle Scholar

  • 5.  A. Borodin M. N. Nielsen and  C. Rackoff , (Incremental) priority algorithms , Algorithmica , 37 ( 2003 ), pp. 295 -- 326 . CrossrefISIGoogle Scholar

  • 6.  S. Boyd and L. Vandenberghe, Convex optimization, Cambridge University Press, Cambridge, 2004. Google Scholar

  • 7.  The Budget and Economic Outlook: 2015 to 2025, U. S. Congressional Budget Office, 2014. Google Scholar

  • 8.  N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, and Games, Cambridge University Press, Cambridge, 2006. Google Scholar

  • 9.  K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for convex hulls, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2010, pp. 1546--1565. Google Scholar

  • 10.  K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for coordinate-wise maxima, in Proceedings of the Symposium on Computational Geometry (SoCG), 2012, pp. 277--286. Google Scholar

  • 11.  K. L. Clarkson and C. Seshadhri, Self-improving algorithms for Delaunay triangulations, in Proceedings of the Symposium on Computational Geometry (SoCG), 2008, pp. 148--155. Google Scholar

  • 12.  L. Devroye, Lectures Notes on Bucket Algorithms, Birkhäuser, Basel, 1986. Google Scholar

  • 13.  E. Fink , How to solve it automatically: Selection among problem solving methods , in Proceedings of the International Conference on Artificial Intelligence Planning Systems , 1998 , pp. 128 -- 136 . Google Scholar

  • 14.  A. Gathmann, Lectures Notes on Algebraic Geometry, TU Kaiserslautern, 2014. Google Scholar

  • 15.  D. Haussler , Decision theoretic generalizations of the PAC model for neural net and other learning applications , Inf. Comput. , 100 ( 1992 ), pp. 78 -- 150 . CrossrefISIGoogle Scholar

  • 16.  E. Horvitz, Y. Ruan, C. P. Gomes, H. A. Kautz, B. Selman, and D. M. Chickering, A Bayesian approach to tackling hard computational problems, in Proceedings of the Conference in Uncertainty in Artificial Intelligence (UAI), 2001, pp. 235--244. Google Scholar

  • 17.  L. Huang, J. Jia, B. Yu, B. Chun, P. Maniatis, and M. Naik, Predicting execution time of computer programs using sparse polynomial regression, in Proceedings of Advances in Neural Information Processing Systems (NIPS), 2010, pp. 883--891. Google Scholar

  • 18.  F. Hutter L. Xu H. H. Hoos and  K. Leyton-Brown , Algorithm runtime prediction: Methods & evaluation , Artif. Intell. , 206 ( 2014 ), pp. 79 -- 111 . CrossrefISIGoogle Scholar

  • 19.  G. J. . Jameson, Counting zeros of generalized polynomials: Descartes’ rule of signs and Laguerre’s extensions , Math. Gaz. , 90 ( 2006 ), pp. 223 -- 234 . CrossrefGoogle Scholar

  • 20.  D. S. Johnson and  L. . McGeoch, The traveling salesman problem: A case study in local optimization, in Local Search in Combinatorial Optimization, E. Aarts and J. K. Lenstra, eds., Wiley , New York , 1997 , pp. 215 -- 310 . Reprinted by Princeton University Press, Princeton, 2003. Google Scholar

  • 21.  D. E. Knuth , Estimating the efficiency of backtrack programs , Math. Comput. , 29 ( 1975 ), pp. 121 -- 136 . CrossrefISIGoogle Scholar

  • 22.  L. Kotthoff I. P. Gent and  I. Miguel , An evaluation of machine learning in algorithm selection for search problems , AI Commun. , 25 ( 2012 ), pp. 257 -- 270 . ISIGoogle Scholar

  • 23.  D. Lehmann L. I. O'Callaghan and  Y. Shoham , Truth revelation in approximately efficient combinatorial auctions , J. ACM , 49 ( 2002 ), pp. 577 -- 602 . CrossrefISIGoogle Scholar

  • 24.  K. Leyton-Brown E. Nudelman and  Y. Shoham , Empirical hardness models: Methodology and a case study on combinatorial auctions , J. ACM , 56 ( 2009 ). ISIGoogle Scholar

  • 25.  N. Littlestone and  M. K. Warmuth , The weighted majority algorithm , Inf. Comput. , 108 ( 1994 ), pp. 212 -- 261 . CrossrefISIGoogle Scholar

  • 26.  P. M. Long, Using the pseudo-dimension to analyze approximation algorithms for integer programming, in Proceedings of the International Workshop on Algorithms and Data Structures (WADS), 2001, pp. 26--37. Google Scholar

  • 27.  P. Milgrom and  I. Segal , Deferred-acceptance auctions and radio spectrum reallocation, in Proceedings of the Fifteenth ACM Conference on Economics and Computation, Palo Alto, CA , Association for Computing Machinery , 2014 , pp. 185 -- 186 . Google Scholar

  • 28.  M. Mohri and A. M. Medina, Learning theory and algorithms for revenue optimization in second price auctions with reserve, in Proceedings of the International Conference on Machine Learning (ICML), 2014, pp. 262--270. Google Scholar

  • 29.  J. Morgenstern and  T. Roughgarden , The pseudo-dimension of near-optimal auctions , in Proceedings of Advances in Neural Information Processing Systems , 2015 , pp. 136 -- 144 . Google Scholar

  • 30.  S. Sakai M. Togasaki and  K. Yamazaki , A note on greedy algorithms for the maximum weighted independent set problem , Discrete Appl. Math. , 126 ( 2003 ), pp. 313 -- 322 . CrossrefISIGoogle Scholar

  • 31.  D. A. Spielman and  S. Teng , Smoothed analysis: an attempt to explain the behavior of algorithms in practice , Commun. ACM , 52 ( 2009 ), pp. 76 -- 84 . CrossrefISIGoogle Scholar

  • 32.  N. Srebro and S. Ben-David, Learning bounds for support vector machines with learned kernels, in Proceedings of the 19th Annual Conference on Learning Theory, 2006, pp. 169--183. Google Scholar

  • 33.  L. Xu F. Hutter H. H. Hoos and  K. Leyton-Brown : Portfolio-based algorithm selection for SAT , J. Artif. Intell. Res. , 32 ( 2008 ), pp. 565 -- 606 . CrossrefISIGoogle Scholar

  • 34.  L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming, in Proceedings of the RCRA Workshop on Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI), 2011, pp. 16--30. Google Scholar

  • 35.  L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, SATzilla 2012: Improved algorithm selection based on cost-sensitive classification models, in Proceedings of the International Conference on Theory and Applications of Satisfiability Testing (SAT), 2012. Google Scholar