The Forgetron: A Kernel-Based Perceptron on a Budget
Abstract
The Perceptron algorithm, despite its simplicity, often performs well in online classification tasks. The Perceptron becomes especially effective when it is used in conjunction with kernel functions. However, a common difficulty encountered when implementing kernel-based online algorithms is the amount of memory required to store the online hypothesis, which may grow unboundedly as the algorithm progresses. Moreover, the running time of each online round grows linearly with the amount of memory used to store the hypothesis. In this paper, we present the Forgetron family of kernel-based online classification algorithms, which overcome this problem by restricting themselves to a predefined memory budget. We obtain different members of this family by modifying the kernel-based Perceptron in various ways. We also prove a unified mistake bound for all of the Forgetron algorithms. To our knowledge, this is the first online kernel-based learning paradigm which, on one hand, maintains a strict limit on the amount of memory it uses and, on the other hand, entertains a relative mistake bound. We conclude with experiments using real datasets, which underscore the merits of our approach.
MSC codes
Keywords
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
N. Cesa-Bianchi and C. Gentile, Tracking the best hyperplane with a simple budget perceptron, in Learning Theory, Lecture Notes in Comput. Sci. 4005, Springer-Verlag, Berlin, 2006, pp. 483–498.
2.
K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer, Online passive aggressive algorithms, J. Mach. Learn. Res., 7 (2006), pp. 551–585.
3.
K. Crammer, J. Kandola, and Y. Singer, Online classification on a budget, in Advances in Neural Information Processing Systems 16, The MIT Press, Cambridge, MA, 2004, pp. 225–232.
4.
O. Dekel, S. Shalev-Shwartz, and Y. Singer, The Forgetron: A kernel-based perceptron on a fixed budget, in Advances in Neural Information Processing Systems 18, The MIT Press, Cambridge, MA, 2006, pp. 259–266.
5.
C. Gentile, A new approximate maximal margin classification algorithm, J. Mach. Learn. Res., 2 (2001), pp. 213–242.
6.
D. P. Helmbold, J. Kivinen, and M. Warmuth, Relative loss bounds for single neurons, IEEE Trans. Neural Networks, 10 (1999), pp. 1291–1304.
7.
J. Kivinen, A. J. Smola, and R. C. Williamson, Online learning with kernels, IEEE Trans. Signal Process., 52 (2002), pp. 2165–2176.
8.
J. Kivinen and M. Warmuth, Exponentiated gradient versus gradient descent for linear predictors, Inform. and Comput., 132 (1997), pp. 1–64.
9.
Y. Li and P. M. Long, The relaxed online maximum margin algorithm, Machine Learning, 46 (2002), pp. 361–387.
10.
F. Rosenblatt, The perceptron: A probabilistic model for information storage and organization in the brain, Psychological Review, 65 (1958), pp. 386–407.
11.
V. N. Vapnik, Statistical Learning Theory, John Wiley, New York, 1998.
12.
J. Weston, A. Bordes, and L. Bottou, Online (and offline) on an even tighter budget, in Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Barbados, 2005, pp. 413–420.
Information & Authors
Information
Published In
Copyright
Copyright © 2008 Society for Industrial and Applied Mathematics.
History
Submitted: 7 August 2006
Accepted: 5 June 2007
Published online: 16 January 2008
MSC codes
Keywords
Authors
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
- Adaptive supervised learning on data streams in reproducing kernel Hilbert spaces with data sparsity constraintStat, Vol. 12, No. 1 | 2 January 2023
- Quick continual kernel learning on bounded memory space based on balancing between adaptation and forgettingEvolving Systems, Vol. 14, No. 3 | 30 December 2022
- Non-invasive grading of brain tumors using online support vector machine with dynamic fuzzy rule-based parameters optimizationProceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, Vol. 25 | 26 May 2023
- Addressing modern and practical challenges in machine learning: a survey of online federated and transfer learningApplied Intelligence, Vol. 53, No. 9 | 30 August 2022
- Worst-case regret analysis of computationally budgeted online kernel selectionMachine Learning, Vol. 111, No. 3 | 22 January 2022
- Graph-based Adversarial Online Kernel Learning with Adaptive Embedding2021 IEEE International Conference on Data Mining (ICDM) | 1 Dec 2021
- Population Coding of Generative Neuronal Cells for Collaborative Decision Making in UAV-Based SLAM OperationsJournal of the Indian Society of Remote Sensing, Vol. 49, No. 3 | 7 November 2020
- High-Probability Kernel Alignment Regret Bounds for Online Kernel SelectionMachine Learning and Knowledge Discovery in Databases. Research Track | 10 September 2021
- Distributed and Quantized Online Multi-Kernel LearningIEEE Transactions on Signal Processing, Vol. 69 | 1 Jan 2021
- Online Budgeted Stochastic Coordinate Ascent for Large-Scale Kernelized Dual Support Vector Machine TrainingPattern Recognition Applications and Methods | 25 January 2020
- Budgeted Passive-Aggressive Learning for Online Multiclass ClassificationIEEE Access, Vol. 8 | 1 Jan 2020
- An Online Stochastic Kernel Machine for Robust Signal Classification2019 53rd Asilomar Conference on Signals, Systems, and Computers | 1 Nov 2019
- Budgeted Algorithm for Linearized Confidence-Weighted LearningProceedings of the 2019 3rd International Conference on Cloud and Big Data Computing | 28 August 2019
- A hybrid multi‐objective firefly and simulated annealing based algorithm for big data classificationConcurrency and Computation: Practice and Experience, Vol. 31, No. 14 | 28 November 2018
- Large-scale support vector regression with budgeted stochastic gradient descentInternational Journal of Machine Learning and Cybernetics, Vol. 10, No. 6 | 7 June 2018
- Large Scale Online Multiple Kernel Regression with Application to Time-Series PredictionACM Transactions on Knowledge Discovery from Data, Vol. 13, No. 1 | 23 January 2019
- Kernel MethodsNeural Networks and Statistical Learning | 13 September 2019
- Swap Kernel RegressionArtificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation | 9 September 2019
- Toward Budgeted Online Kernel Ridge Regression on Streaming DataIEEE Access, Vol. 7 | 1 Jan 2019
- Supervised online matrix factorization for histopathological multimodal retrieval14th International Symposium on Medical Information Processing and Analysis | 21 Dec 2018
- Variable learning rates kernel adaptive filter with single feedbackDigital Signal Processing, Vol. 83 | 1 Dec 2018
- A Noise-Resilient Online Learning Algorithm for Scene ClassificationRemote Sensing, Vol. 10, No. 11 | 20 November 2018
- Sparse Passive-Aggressive Learning for Bounded Online Kernel MethodsACM Transactions on Intelligent Systems and Technology, Vol. 9, No. 4 | 30 January 2018
- Decentralized Online Learning With KernelsIEEE Transactions on Signal Processing, Vol. 66, No. 12 | 15 Jun 2018
- Online Learning Adaptive to Dynamic and Adversarial Environments2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) | 1 Jun 2018
- Large-Scale Kernel-Based Feature Extraction via Low-Rank Subspace Tracking on a BudgetIEEE Transactions on Signal Processing, Vol. 66, No. 8 | 15 Apr 2018
- Online Nonlinear AUC Maximization for Imbalanced Data SetsIEEE Transactions on Neural Networks and Learning Systems, Vol. 29, No. 4 | 1 Apr 2018
- Feature Selection, Online Feature Selection Techniques for Big Data Classification: - A Review2018 International Conference on Current Trends towards Converging Technologies (ICCTCT) | 1 Mar 2018
- A Novel Manifold Regularized Online Semi-supervised Learning ModelCognitive Computation, Vol. 10, No. 1 | 2 August 2017
- A Local Online Learning Approach for Non-linear DataAdvances in Knowledge Discovery and Data Mining | 20 June 2018
- Differed Restructuring of Neural Connectome Using Evolutionary Neurodynamic Algorithm for Improved M2M Online LearningProcedia Computer Science, Vol. 133 | 1 Jan 2018
- Memory efficient low-rank non-linear subspace tracking2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) | 1 Dec 2017
- Prototype-based budget maintenance for tracking in depth videosMultimedia Tools and Applications, Vol. 76, No. 20 | 22 October 2016
- Scalable kernel-based learning via low-rank approximation of lifted data2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton) | 1 Oct 2017
- OMKT: Projection based bounded on-line multiple kernel tracker2017 International Joint Conference on Neural Networks (IJCNN) | 1 May 2017
- A Note on the Unification of Adaptive Online LearningIEEE Transactions on Neural Networks and Learning Systems, Vol. 28, No. 5 | 1 May 2017
- Classifier transfer with data selection strategies for online support vector machine classification with class imbalanceJournal of Neural Engineering, Vol. 14, No. 2 | 13 February 2017
- Application of Statistical Classifiers on Java Source CodeProceedings of the 2015 Federated Conference on Software Development and Object Technologies | 15 December 2016
- Stochastic Online Kernel Selection with Instantaneous Loss in Random Feature SpaceNeural Information Processing | 24 October 2017
- An empirical study on budget-aware online kernel algorithms for streams of graphsNeurocomputing, Vol. 216 | 1 Dec 2016
- One-pass online learning: A local approachPattern Recognition, Vol. 51 | 1 Mar 2016
- Online kernel learning with nearly constant support vectorsNeurocomputing, Vol. 179 | 1 Feb 2016
- Incremental Learning of Muscle Synergies: From Calibration to InteractionHuman and Robot Hands | 25 February 2016
- Multithreading Incremental Learning Scheme for Embedded System to Realize a High-ThroughputNeural Information Processing | 30 September 2016
- Risk Sensitive Reinforcement Learning Scheme Is Suitable for Learning on a BudgetNeural Information Processing | 29 September 2016
- Confidence-Weighted Bipartite RankingAdvanced Data Mining and Applications | 13 November 2016
- Robust compressive features based power quality events classification with Analog–Digital Mixing Network (ADMN)Neurocomputing, Vol. 171 | 1 Jan 2016
- Budget Online Learning Algorithm for Least Squares SVMIEEE Transactions on Neural Networks and Learning Systems | 1 Jan 2016
- Kernel-based low-rank feature extraction on a budget for big data streams2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP) | 1 Dec 2015
- KCRC-LCD: Discriminative kernel collaborative representation with locality constrained dictionary for visual categorizationPattern Recognition, Vol. 48, No. 10 | 1 Oct 2015
- Kernel Least Mean Square with Single FeedbackIEEE Signal Processing Letters, Vol. 22, No. 7 | 1 Jul 2015
- A Survey on Supervised Classification on Data StreamsBusiness Intelligence | 1 Jan 2015
- Simple randomized algorithms for online learning with kernelsNeural Networks, Vol. 60 | 1 Dec 2014
- Online multiple kernel regressionProceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining | 24 August 2014
- Incremental Learning on a Budget and its Application to Quick Maximum Power Point Tracking of Photovoltaic SystemsJournal of Advanced Computational Intelligence and Intelligent Informatics, Vol. 18, No. 4 | 20 July 2014
- On the Sample Complexity of Random Fourier Features for Online LearningACM Transactions on Knowledge Discovery from Data, Vol. 8, No. 3 | 1 June 2014
- Sparse Ridgelet Kernel Regressor and its online sequential extreme learningNeurocomputing, Vol. 134 | 1 Jun 2014
- Reliable Posterior Probability Estimation for Streaming Face Recognition2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops | 1 Jun 2014
- Towards the online learning with Kernels in classification and regressionEvolving Systems, Vol. 5, No. 1 | 14 July 2013
- Online Feature Selection and Its ApplicationsIEEE Transactions on Knowledge and Data Engineering, Vol. 26, No. 3 | 1 Mar 2014
- Effective Kernelized Online Learning in Language Processing TasksAdvances in Information Retrieval | 1 Jan 2014
- Other Kernel MethodsNeural Networks and Statistical Learning | 7 December 2013
- Robust Language Learning via Efficient Budgeted Online Algorithms2013 IEEE 13th International Conference on Data Mining Workshops | 1 Dec 2013
- Linear Online Learning over Structured Data with Distributed Tree Kernels2013 12th International Conference on Machine Learning and Applications | 1 Dec 2013
- Fixed budget quantized kernel least-mean-square algorithmSignal Processing, Vol. 93, No. 9 | 1 Sep 2013
- Semi-Supervised Learning and Domain Adaptation in Natural Language ProcessingSynthesis Lectures on Human Language Technologies, Vol. 6, No. 2 | 21 May 2013
- Real-time model learning using Incremental Sparse Spectrum Gaussian Process RegressionNeural Networks, Vol. 41 | 1 May 2013
- Dynamical memory control based on projection technique for online regressionSoft Computing, Vol. 17, No. 4 | 26 September 2012
- Interpretable models from distributed data via merging of decision trees2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM) | 1 Apr 2013
- Revised online learning with kernels for classification and regression2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM) | 1 Apr 2013
- Model-Based Online Learning With KernelsIEEE Transactions on Neural Networks and Learning Systems, Vol. 24, No. 3 | 1 Mar 2013
- Online Learning with (Multiple) Kernels: A ReviewNeural Computation, Vol. 25, No. 3 | 1 Mar 2013
- Online Multiple Kernel ClassificationMachine Learning, Vol. 90, No. 2 | 13 December 2012
- Incremental Learning on a Budget and Its Application to Power ElectronicsNeural Information Processing | 1 Jan 2013
- Incremental learning on a budget and its application to quick maximum power point tracking of photovoltaic systemsThe 6th International Conference on Soft Computing and Intelligent Systems, and The 13th International Symposium on Advanced Intelligence Systems | 1 Nov 2012
- A new method of online learning with kernels for regression2012 7th IEEE Conference on Industrial Electronics and Applications (ICIEA) | 1 Jul 2012
- Online learning with kernels in classification and regression2012 IEEE Conference on Evolving and Adaptive Intelligent Systems | 1 May 2012
- Online Sequential Extreme Learning of Sparse Ridgelet Kernel Regressor for Nonlinear Time-Series PredictionIntelligent Science and Intelligent Data Engineering | 1 Jan 2012
- BDUOL: Double Updating Online Learning on a Fixed BudgetMachine Learning and Knowledge Discovery in Databases | 1 Jan 2012
- A kernel-based Perceptron with dynamic memoryNeural Networks, Vol. 25 | 1 Jan 2012
- Quantized Kernel Least Mean Square AlgorithmIEEE Transactions on Neural Networks and Learning Systems, Vol. 23, No. 1 | 1 Jan 2012
- Efficient Classification with Conjunctive FeaturesJournal of Information Processing, Vol. 20, No. 1 | 1 Jan 2012
- A comparison study on kernel based online learning for moving object classification2011 Third Chinese Conference on Intelligent Visual Surveillance | 1 Dec 2011
- Incremental Learning From Stream DataIEEE Transactions on Neural Networks, Vol. 22, No. 12 | 1 Dec 2011
- Towards semi-supervised learning of semantic spatial concepts2011 IEEE International Conference on Robotics and Automation | 1 May 2011
- Incremental learning of robot dynamics using random features2011 IEEE International Conference on Robotics and Automation | 1 May 2011
- Real-Time Discriminative Background SubtractionIEEE Transactions on Image Processing, Vol. 20, No. 5 | 1 May 2011
- Learning to detect malicious URLsACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 3 | 6 May 2011
- Two One-Pass Algorithms for Data Stream Classification Using Approximate MEBsAdaptive and Natural Computing Algorithms | 1 Jan 2011
- Preference-Based Policy LearningMachine Learning and Knowledge Discovery in Databases | 1 Jan 2011
- : An adaptive classifier for data streamsPattern Recognition, Vol. 44, No. 1 | 1 Jan 2011
- Online training on a budget of support vector machines using twin prototypesStatistical Analysis and Data Mining: The ASA Data Science Journal, Vol. 3, No. 3 | 12 May 2010
- Fixed-budget kernel recursive least-squares2010 IEEE International Conference on Acoustics, Speech and Signal Processing | 1 Mar 2010
- An Online Framework for Learning Novel Concepts over Multiple CuesComputer Vision – ACCV 2009 | 1 Jan 2010
- Example-dependent Basis Vector Selection for Kernel-Based ClassifiersMachine Learning and Knowledge Discovery in Databases | 1 Jan 2010
- Tighter Perceptron with improved dual use of cached data for model representation and validation2009 International Joint Conference on Neural Networks | 1 Jun 2009
- Recent results on processing random-order streams and space-efficient sampling2008 46th Annual Allerton Conference on Communication, Control, and Computing | 1 Sep 2008