3 - Kernel-Based Adaptive Filtering
Published online by Cambridge University Press: 24 November 2022
Summary
This chapter explains the basic concepts of kernel-based methods, a widely used tool in machine learning. The idea is to present online parameter estimation of nonlinear models using kernel-based tools. The chapters aim is to introduce the kernel version of classical algorithms such as least mean square (LMS), recursive least squares (RLS), affine projection (AP), and set membership affine projection (SM-AP). In particular, we will discuss how to keep the dictionary of the kernel finite through a series of model reduction strategies. This way, all discussed kernel algorithms are tailored for online implementation.
- Type
- Chapter
- Information
- Online Learning and Adaptive Filters , pp. 60 - 108Publisher: Cambridge University PressPrint publication year: 2022
References
Aronszajn, N., Theory of reproducing kernels. Transactions of the American Mathematical Society 63, pp. 337-404 (1950).Google Scholar
Aizerman, M. A., Braverman, E. M., and L. I. Rozoner, Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control 25, pp. 821-837 (1964).Google Scholar
Diniz, P. S. R., Adaptive Filtering: Algorithms and Practical Implementations, 5th ed. (Springer, Cham, 2020).Google Scholar
Liu, W., P.P. Pokharel, and J.C. Principe, The kernel least-mean-square algorithm. IEEE Transactions on Signal Processing. 56, pp. 543-554 (2008).Google Scholar
Liu, W., J.C. Principe, S. Haykin, Kernel Adaptive Filtering: A Comprehensive Introduction (Wiley, Hoboken, 2010).CrossRefGoogle Scholar
Scholkopf, B. and Smola, A. L., Learning with Kernels: Support Vector Machine, Regularization, Optimization and Beyond (The MIT Press, Cambridge, 2001).Google Scholar
Murphy, K. P., Machine Learning: A Probabilistic Perspective (The MIT Press, Cambridge, 2012).Google Scholar
Theodoridis, S., Machine Learning: A Bayesian and Optimization Perspective (Academic Press, Oxford, 2015).Google Scholar
Abu-Mostafa, Y. S., Magdon-Ismail, M., and H.-T. Lin, Learning from Data (AMLbook.com, 2012).Google Scholar
Richard, C., C. M. Bermudez, J., and P. Honeine, Online prediction of time series data with kernels. IEEE Transactions on Signal Processing 57, pp. 1058-1067 (2009).CrossRefGoogle Scholar
Parreira, W. D., C. M. Bermudez, J., Richard, C., and J.-Y. Tourneret, Stochastic behavior analysis of the Gaussian kernel least-mean-square algorithm. IEEE Transactions on Signal Processing 60, pp. 2208-2222 (2012).Google Scholar
Ozeki, K., Theory of Affine Projection Algorithms for Adaptive Filtering (Springer, New York, 2015).Google Scholar
Albu, F., Coltuc, D., Rotaru, M., and K. Nishikawa, An efficient implementation of the kernel affine projection algorithm. 8th International Symposium on Image and Signal Processing and Analysis (ISPA 2013), Trieste, Italy, 2013, pp. 349353.Google Scholar
Honeine, P., Approximation errors of online sparsification criteria. IEEE Transactions on Signal Processing 63, pp. 4700-4709 (2015).Google Scholar
Engel, Y., Mannor, S., and R. Meir, The kernel recursive least-squares algorithm. IEEE Transactions on Signal Processing 52, pp. 2275-2285 (2004).Google Scholar
Van Vaerenbergh, S., Via, J., and I. Santamarla, A sliding-window kernel RLS and its application to nonlinear channel identification. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Toulouse, France, May 2006, pp. V-789-V-792.Google Scholar
Van Vaerenbergh, S., Via, J., and I. Santamarla, Nonlinear system identification using new sliding-window kernel RLS algorithm. Journal of Communications 2, pp. 1-8 (2007).Google Scholar
Van Vaerenbergh, S., Santamarla, I., Liu, W., and J. C. Principe, Fixed-budget kernel recursive least-squares. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Toulouse, France, May 2006, pp. V- 789-V-792.Google Scholar
Malipatil, A. V., Huang, Y.-F., Andra, S., and K. Bennett, Kernelized set- membership approach to nonlinear adaptive filtering. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, USA, May 2005, pp. IV-149-IV-152.Google Scholar
Flores, A. and C. de Lamare, R., Set-membership kernel adaptive algorithms. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, New Orleans, USA, May 2017, pp. 2676-2680.Google Scholar
Chen, K., Werner, S., Kuh, A., and Y.-F. Huang, Nonlinear adaptive filtering with kernel set-membership approach. IEEE Transactions on Signal Processing 68, pp. 1515-1528 (2020).Google Scholar
Rahimi, A. and Recht, B., Random features for large-scale kernel machines. NIPS'07: Proceedings of the 20th International Conference on Neural Information Processing Systems, Curran Associates, Inc., 2008, pp. 1-8.Google Scholar
Sriperumbudur, B. K. and Szabo, Z., Optimal rates for random Fourier features. NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 1144-1152.Google Scholar
Sutherland, D. J. and Schneider, J., On the error of random Fourier features. arXiv:1506.02785, 2015, pp. 1-10.Google Scholar
Muandet, K., Fukumizu, K., Sriperumbudur, B., and B. Scholkopf, Kernel Mean Embedding of Distributions: A Review and Beyond (NOW Publishers, Delft, 2017).Google Scholar
Diniz, P. S. R., On data-selective adaptive filtering. IEEE Transactions on Signal Processing 66, pp. 4239-4252 (2018).Google Scholar
Mendonca, M. O. K., Ferreira, J. O., Tsinos, C. G., S. R. Diniz, P., and T. N. Fer- reira, On fast converging data-selective adaptive filtering. Algorithms 12, pp. 115 (2019).Google Scholar
Diniz, P. S. R., Ferreira, J. O., O. K. Mendoncca, M., and T. N. Ferreira, Data selection kernel conjugate gradient algorithm. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020, pp. 1-5.Google Scholar
Ferreira, J. O., O. K. Mendonca, M., and P. S. R. Diniz, Data selection in neural networks. IEEE Open Journal of Signal Processing 2, pp. 1-15 (2021).Google Scholar
Souza Filho and P. S. R. Diniz, J. B., A recursive least square algorithm for online kernel principal component extraction. Neuralcomputing 237, pp. 255264 (2017).Google Scholar
Souza Filho, J. B., S. R. Diniz, P., B. Souza Filho, J., and P. S. R. Diniz, Fixed- point online kernel principal component extraction algorithm. IEEE Transactions on Signal Processing 65, pp. 6244-6259 (2017).Google Scholar
Souza Filho and P. S. R. Diniz, J. B., Improving KPCA online extraction by orthonormalization in the feature space. IEEE Transactions on Neural Networks and Learning Systems 29, pp. 2162-2388 (2018).Google Scholar