Abstract

The best algorithm for a computational problem generally depends on the “relevant inputs,” a concept that depends on the application domain and often defies formal articulation. While there is a large body of literature on empirical approaches to selecting the best algorithm for a given application domain, there has been surprisingly little theoretical analysis of the problem. This paper adapts concepts from statistical and online learning theory to reason about application-specific algorithm selection. Our models capture several state-of-the-art empirical and theoretical approaches to the problem, ranging from self-improving algorithms to empirical performance models, and our results identify conditions under which these approaches are guaranteed to perform well. We present one framework that models algorithm selection as a statistical learning problem, and our work here shows that dimension notions from statistical learning theory, historically used to measure the complexity of classes of binary- and real-valued functions, are relevant in a much broader algorithmic context. We also study the online version of the algorithm selection problem, and give possibility and impossibility results for the existence of no-regret learning algorithms.

Keywords

  1. algorithm selection
  2. parameter tuning
  3. PAC learning
  4. online learning
  5. meta-algorithms

MSC codes

  1. 68Q32

Get full access to this article

View all available purchase options and get full access to this article.

References

1.
N. Ailon, B. Chazelle, S. Comandur, and D. Liu, Self-improving algorithms, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2006, pp. 261--270.
2.
M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge University Press, Cambridge, 1999.
3.
V. Arya, N. Garg, R. Khandekar, A. Meyerson, K. Munagala, and V. Pandit, Local search heuristics for $k$-median and facility location problems, SIAM J. Comput., 33 (2004), pp. 544--562.
4.
J. Bergstra and Y. Bengio, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13 (2012), pp. 281--305.
5.
A. Borodin, M. N. Nielsen, and C. Rackoff, (Incremental) priority algorithms, Algorithmica, 37 (2003), pp. 295--326.
6.
S. Boyd and L. Vandenberghe, Convex optimization, Cambridge University Press, Cambridge, 2004.
7.
The Budget and Economic Outlook: 2015 to 2025, U. S. Congressional Budget Office, 2014.
8.
N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, and Games, Cambridge University Press, Cambridge, 2006.
9.
K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for convex hulls, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2010, pp. 1546--1565.
10.
K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for coordinate-wise maxima, in Proceedings of the Symposium on Computational Geometry (SoCG), 2012, pp. 277--286.
11.
K. L. Clarkson and C. Seshadhri, Self-improving algorithms for Delaunay triangulations, in Proceedings of the Symposium on Computational Geometry (SoCG), 2008, pp. 148--155.
12.
L. Devroye, Lectures Notes on Bucket Algorithms, Birkhäuser, Basel, 1986.
13.
E. Fink, How to solve it automatically: Selection among problem solving methods, in Proceedings of the International Conference on Artificial Intelligence Planning Systems, 1998, pp. 128--136.
14.
A. Gathmann, Lectures Notes on Algebraic Geometry, TU Kaiserslautern, 2014.
15.
D. Haussler, Decision theoretic generalizations of the PAC model for neural net and other learning applications, Inf. Comput., 100 (1992), pp. 78--150.
16.
E. Horvitz, Y. Ruan, C. P. Gomes, H. A. Kautz, B. Selman, and D. M. Chickering, A Bayesian approach to tackling hard computational problems, in Proceedings of the Conference in Uncertainty in Artificial Intelligence (UAI), 2001, pp. 235--244.
17.
L. Huang, J. Jia, B. Yu, B. Chun, P. Maniatis, and M. Naik, Predicting execution time of computer programs using sparse polynomial regression, in Proceedings of Advances in Neural Information Processing Systems (NIPS), 2010, pp. 883--891.
18.
F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown, Algorithm runtime prediction: Methods & evaluation, Artif. Intell., 206 (2014), pp. 79--111.
19.
G. J. O. Jameson, Counting zeros of generalized polynomials: Descartes’ rule of signs and Laguerre’s extensions, Math. Gaz., 90 (2006), pp. 223--234.
20.
D. S. Johnson and L. A. McGeoch, The traveling salesman problem: A case study in local optimization, in Local Search in Combinatorial Optimization, E. Aarts and J. K. Lenstra, eds., Wiley, New York, 1997, pp. 215--310. Reprinted by Princeton University Press, Princeton, 2003.
21.
D. E. Knuth, Estimating the efficiency of backtrack programs, Math. Comput., 29 (1975), pp. 121--136.
22.
L. Kotthoff, I. P. Gent, and I. Miguel, An evaluation of machine learning in algorithm selection for search problems, AI Commun., 25 (2012), pp. 257--270.
23.
D. Lehmann, L. I. O'Callaghan, and Y. Shoham, Truth revelation in approximately efficient combinatorial auctions, J. ACM, 49 (2002), pp. 577--602.
24.
K. Leyton-Brown, E. Nudelman, and Y. Shoham, Empirical hardness models: Methodology and a case study on combinatorial auctions, J. ACM, 56 (2009).
25.
N. Littlestone and M. K. Warmuth, The weighted majority algorithm, Inf. Comput., 108 (1994), pp. 212--261.
26.
P. M. Long, Using the pseudo-dimension to analyze approximation algorithms for integer programming, in Proceedings of the International Workshop on Algorithms and Data Structures (WADS), 2001, pp. 26--37.
27.
P. Milgrom and I. Segal, Deferred-acceptance auctions and radio spectrum reallocation, in Proceedings of the Fifteenth ACM Conference on Economics and Computation, Palo Alto, CA, Association for Computing Machinery, 2014, pp. 185--186.
28.
M. Mohri and A. M. Medina, Learning theory and algorithms for revenue optimization in second price auctions with reserve, in Proceedings of the International Conference on Machine Learning (ICML), 2014, pp. 262--270.
29.
J. Morgenstern and T. Roughgarden, The pseudo-dimension of near-optimal auctions, in Proceedings of Advances in Neural Information Processing Systems, 2015, pp. 136--144.
30.
S. Sakai, M. Togasaki, and K. Yamazaki, A note on greedy algorithms for the maximum weighted independent set problem, Discrete Appl. Math., 126 (2003), pp. 313--322.
31.
D. A. Spielman and S. Teng, Smoothed analysis: an attempt to explain the behavior of algorithms in practice, Commun. ACM, 52 (2009), pp. 76--84.
32.
N. Srebro and S. Ben-David, Learning bounds for support vector machines with learned kernels, in Proceedings of the 19th Annual Conference on Learning Theory, 2006, pp. 169--183.
33.
L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, SATzilla: Portfolio-based algorithm selection for SAT, J. Artif. Intell. Res., 32 (2008), pp. 565--606.
34.
L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming, in Proceedings of the RCRA Workshop on Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI), 2011, pp. 16--30.
35.
L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, SATzilla 2012: Improved algorithm selection based on cost-sensitive classification models, in Proceedings of the International Conference on Theory and Applications of Satisfiability Testing (SAT), 2012.

Information & Authors

Information

Published In

cover image SIAM Journal on Computing
SIAM Journal on Computing
Pages: 992 - 1017
ISSN (online): 1095-7111

History

Submitted: 30 November 2015
Accepted: 25 January 2017
Published online: 13 June 2017

Keywords

  1. algorithm selection
  2. parameter tuning
  3. PAC learning
  4. online learning
  5. meta-algorithms

MSC codes

  1. 68Q32

Authors

Affiliations

Funding Information

National Science Foundation https://doi.org/10.13039/100000001 : CCF-1215965, CCF-1524062

Metrics & Citations

Metrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited By

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share on social media