An introduction to the Hilbert-Schmidt SVD using iterated Brownian bridge kernels

Kernel-based approximation methods -- often in the form of radial basis functions -- have been used for many years now and usually involve setting up a kernel matrix which may be ill-conditioned when the shape parameter of the kernel takes on extreme values, i.e., makes the kernel "flat". In this paper we present an algorithm we refer to as the Hilbert-Schmidt SVD and use it to emphasize two important points which -- while not entirely new -- present a paradigm shift under way in the practical application of kernel-based approximation methods: (i) it is not necessary to form the kernel matrix (in fact, it might even be a bad idea to do so), and (ii) it is not necessary to know the kernel in closed form. While the Hilbert-Schmidt SVD and its two implications apply to general positive definite kernels, we introduce in this paper a class of so-called iterated Brownian bridge kernels which allow us to keep the discussion as simple and accessible as possible.