A PAC Approach to Application-Specific Algorithm Selection
The best algorithm for a computational problem generally depends on the “relevant inputs,” a concept that depends on the application domain and often defies formal articulation. While there is a large body of literature on empirical approaches to selecting the best algorithm for a given application domain, there has been surprisingly little theoretical analysis of the problem. This paper adapts concepts from statistical and online learning theory to reason about application-specific algorithm selection. Our models capture several state-of-the-art empirical and theoretical approaches to the problem, ranging from self-improving algorithms to empirical performance models, and our results identify conditions under which these approaches are guaranteed to perform well. We present one framework that models algorithm selection as a statistical learning problem, and our work here shows that dimension notions from statistical learning theory, historically used to measure the complexity of classes of binary- and real-valued functions, are relevant in a much broader algorithmic context. We also study the online version of the algorithm selection problem, and give possibility and impossibility results for the existence of no-regret learning algorithms.
1. N. Ailon, B. Chazelle, S. Comandur, and D. Liu, Self-improving algorithms, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2006, pp. 261--270.
2. M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge University Press, Cambridge, 1999.
3. , Local search heuristics for $k$-median and facility location problems , SIAM J. Comput. , 33 ( 2004 ), pp. 544 -- 562 .
4. , Random search for hyper-parameter optimization , J. Mach. Learn. Res. , 13 ( 2012 ), pp. 281 -- 305 .
5. , (Incremental) priority algorithms , Algorithmica , 37 ( 2003 ), pp. 295 -- 326 .
6. S. Boyd and L. Vandenberghe, Convex optimization, Cambridge University Press, Cambridge, 2004.
7. The Budget and Economic Outlook: 2015 to 2025, U. S. Congressional Budget Office, 2014.
8. N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, and Games, Cambridge University Press, Cambridge, 2006.
9. K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for convex hulls, in Proceedings of the Symposium on Discrete Algorithms (SODA), 2010, pp. 1546--1565.
10. K. L. Clarkson, W. Mulzer, and C. Seshadhri, Self-improving algorithms for coordinate-wise maxima, in Proceedings of the Symposium on Computational Geometry (SoCG), 2012, pp. 277--286.
11. K. L. Clarkson and C. Seshadhri, Self-improving algorithms for Delaunay triangulations, in Proceedings of the Symposium on Computational Geometry (SoCG), 2008, pp. 148--155.
12. L. Devroye, Lectures Notes on Bucket Algorithms, Birkhäuser, Basel, 1986.
13. , How to solve it automatically: Selection among problem solving methods , in Proceedings of the International Conference on Artificial Intelligence Planning Systems , 1998 , pp. 128 -- 136 .
14. A. Gathmann, Lectures Notes on Algebraic Geometry, TU Kaiserslautern, 2014.
15. , Decision theoretic generalizations of the PAC model for neural net and other learning applications , Inf. Comput. , 100 ( 1992 ), pp. 78 -- 150 .
16. E. Horvitz, Y. Ruan, C. P. Gomes, H. A. Kautz, B. Selman, and D. M. Chickering, A Bayesian approach to tackling hard computational problems, in Proceedings of the Conference in Uncertainty in Artificial Intelligence (UAI), 2001, pp. 235--244.
17. L. Huang, J. Jia, B. Yu, B. Chun, P. Maniatis, and M. Naik, Predicting execution time of computer programs using sparse polynomial regression, in Proceedings of Advances in Neural Information Processing Systems (NIPS), 2010, pp. 883--891.
18. , Algorithm runtime prediction: Methods & evaluation , Artif. Intell. , 206 ( 2014 ), pp. 79 -- 111 .
19. . Jameson, Counting zeros of generalized polynomials: Descartesâ rule of signs and Laguerreâs extensions , Math. Gaz. , 90 ( 2006 ), pp. 223 -- 234 .
20. . McGeoch, The traveling salesman problem: A case study in local optimization, in Local Search in Combinatorial Optimization, E. Aarts and J. K. Lenstra, eds., Wiley , New York , 1997 , pp. 215 -- 310 . Reprinted by Princeton University Press, Princeton, 2003.
21. , Estimating the efficiency of backtrack programs , Math. Comput. , 29 ( 1975 ), pp. 121 -- 136 .
22. , An evaluation of machine learning in algorithm selection for search problems , AI Commun. , 25 ( 2012 ), pp. 257 -- 270 .
23. , Truth revelation in approximately efficient combinatorial auctions , J. ACM , 49 ( 2002 ), pp. 577 -- 602 .
24. , Empirical hardness models: Methodology and a case study on combinatorial auctions , J. ACM , 56 ( 2009 ).
25. , The weighted majority algorithm , Inf. Comput. , 108 ( 1994 ), pp. 212 -- 261 .
26. P. M. Long, Using the pseudo-dimension to analyze approximation algorithms for integer programming, in Proceedings of the International Workshop on Algorithms and Data Structures (WADS), 2001, pp. 26--37.
27. , Deferred-acceptance auctions and radio spectrum reallocation, in Proceedings of the Fifteenth ACM Conference on Economics and Computation, Palo Alto, CA , Association for Computing Machinery , 2014 , pp. 185 -- 186 .
28. M. Mohri and A. M. Medina, Learning theory and algorithms for revenue optimization in second price auctions with reserve, in Proceedings of the International Conference on Machine Learning (ICML), 2014, pp. 262--270.
29. , The pseudo-dimension of near-optimal auctions , in Proceedings of Advances in Neural Information Processing Systems , 2015 , pp. 136 -- 144 .
30. , A note on greedy algorithms for the maximum weighted independent set problem , Discrete Appl. Math. , 126 ( 2003 ), pp. 313 -- 322 .
31. , Smoothed analysis: an attempt to explain the behavior of algorithms in practice , Commun. ACM , 52 ( 2009 ), pp. 76 -- 84 .
32. N. Srebro and S. Ben-David, Learning bounds for support vector machines with learned kernels, in Proceedings of the 19th Annual Conference on Learning Theory, 2006, pp. 169--183.
33. : Portfolio-based algorithm selection for SAT , J. Artif. Intell. Res. , 32 ( 2008 ), pp. 565 -- 606 .
34. L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming, in Proceedings of the RCRA Workshop on Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI), 2011, pp. 16--30.
35. L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown, SATzilla 2012: Improved algorithm selection based on cost-sensitive classification models, in Proceedings of the International Conference on Theory and Applications of Satisfiability Testing (SAT), 2012.