Society for Industrial and Applied Mathematics: Data Sciences

Reproducing Kernel Methods for Machine Learning, PDEs, and Statistics

Philippe G. LeFloch

Reproducing Kernel Methods for Machine Learning, PDEs, and Statistics.
Reproducing Kernel Methods for Machine Learning, PDEs, and Statistics is divided into two parts, theoretical principles and the techniques employed in their applications; contains numerous applications in engineering, finance, and machine learning; and provides a framework for designing numerically efficient, large-scale dataset strategies.

Back Matter

2026-02-27T05:30:05Z

Other Titles in Applied Mathematics, Page 291-320.
Excerpt The back matter includes Bibliography, Index, and backcover.

Chapter 1: Introduction

2026-02-27T05:30:05Z

Other Titles in Applied Mathematics, Page 3-10.
Excerpt In an inverse problem, we use data and a mathematical model to estimate unknown quantities that are not directly observable. In Section 1.1, we consider a basic example—an inverse problem governed by a heat conduction problem in one space dimension. Specifically, we focus on the problem of estimating the initial temperature field using temperature measurements at a final time. This is a classical example in the inverse problems literature [230, 232, 236] that is commonly referred to as the backward heat equation. The discussion in Section 1.1 helps illustrate the components of inverse problems governed by PDEs and a central challenge that is a recurring feature of such problems—ill-posedness. Then, we discuss the key challenges in solving infinite-dimensional inverse problems governed by PDEs in Section 1.2. Subsequently, we discuss a few additional examples of inverse problems in Section 1.3.

Chapter 10: Discretization and optimization

2026-02-27T05:30:06Z

Other Titles in Applied Mathematics, Page 141-147.
Excerpt Consider a variational inverse problem of the form(10.1)\begin{align*}&\min_{m} \; \hat{J}(m) := J(m,u(m)), \\& \text{where}\\& c(u,m) = 0.\end{align*}Here, [math] and [math] are abstract notations for the cost functional and the governing PDEs, respectively. As discussed in Section 5.4, there are two approaches to solving such problems: discretize-then-optimize (DTO) and optimize-then-discretize (OTD). In the DTO approach we first discretize the cost functional [math] and consider a finite-dimensional optimization problem. On the other hand, in the OTD approach, we first derive the necessary optimality conditions for minimization of [math], after which discretization is performed; see Figure 10.1. Figure 10.1.The OTD and DTO paradigms. In the OTD approach, we use a Lagrange multiplier formulation in a functional setting to derive the optimality system. In the DTO approach, this is accomplished using a discretized Lagrangian.

Chapter 11: Solvers for variational inverse problems

2026-02-27T05:30:07Z

Other Titles in Applied Mathematics, Page 149-158.
Excerpt Up to this point in the book, we have discussed various tools needed for developing solution methods for inverse problems governed by PDEs. These tools can be combined in different ways to develop solvers for the class of inverse problems under study. In this chapter, we consider some commonly used approaches. The primary goal here is to describe optimization methods for nonlinear inverse problems. Our focus will be on Newton (and Gauss–Newton) solvers. However, as usual, starting the discussion with linear inverse problems is instructive. For one thing, linear inverse problems constitute an important class of problems in their own right. Moreover, discussing methods for linear inverse problems is helpful for the downstream goal of developing Newton solvers for nonlinear inverse problems. We discuss Conjugate Gradient (CG) solvers for PDE-based linear inverse problems in Section 11.1. Subsequently, we consider nonlinear inverse problems in Section 11.2. There, we discuss various aspects of implementing inexact Newton–CG solvers for variational inverse problems. The discussion includes integrating adjoint-based gradient and Hessian action expressions in the Newton–CG iterations, discretization issues, preconditioned CG iterations, and computational cost. To provide some quick computational insight, that section also includes a brief numerical illustration.

Chapter 12: Post-optimality sensitivity analysis

2026-02-27T05:30:07Z

Other Titles in Applied Mathematics, Page 159-163.
Excerpt In our discussion of inverse problems so far, we have assumed full knowledge of the governing model and focused on estimating an inversion parameter of interest. However, in practice, the governing PDEs often contain additional parameters that are not known exactly, but are not being estimated. Consider, for example, the model inverse problem in Example 1.2. In that problem, we focus on estimating the log-permeability field [math] while assuming full knowledge of the boundary conditions and the volume source term. However, in practice, the boundary terms or volume source terms are typically uncertain. In principle, it is desirable to estimate all uncertain model parameters. Attempting to do so may require estimating several function-valued parameters as well as additional scalar- or vectorvalued parameters. This is often impractical due to the increased computational overhead and/or lack of measurement data that is informative for estimating all uncertain model parameters. Thus, in practice, one focuses on estimating inversion parameters of interest and fixes additional model parameters at some nominal values. We call these additional model parameters the auxiliary parameters. For example, in Example 1.2, the inversion parameter is the log-permeability field, and the functions defining the boundary conditions and the volume source term are auxiliary parameters.

Chapter 13: Motivation

2026-02-27T05:30:06Z

Other Titles in Applied Mathematics, Page 167-175.
Excerpt. What is meant by the Bayesian formulation of an inverse problem? Why is this important to study? What are the key challenges in Bayesian inversion? We discuss these questions in this chapter. We begin with an intuitive introduction to Bayesian inversion in Section 13.1. Our discussion in that section is driven by a simple example and takes a mathematical modeling perspective. Statistical concepts are introduced informally as needed. Then, in Section 13.2, we consider inverse problems governed by PDEs with function-valued parameters, which are the focus of the present book. As discussed in that section, such problems pose fundamental mathematical and computational challenges. Discussion of methods for addressing these challenges spans the remainder of Part IV. To help guide the reader, we provide a brief roadmap for the rest of Part IV in Section 13.3.

Chapter 14: Fundamental concepts from probability theory

2026-02-27T05:30:06Z

Other Titles in Applied Mathematics, Page 177-220.
Excerpt In this chapter, we cover the background concepts from probability theory needed in the remainder of the book. Our discussion includes some basic probability theory as well as more advanced topics such as random fields and probability measures on functions spaces. We begin by covering some basics regarding probability spaces and random variables in Section 14.1. In our study of infinite-dimensional Bayesian inverse problems, we need to be comfortable working with function-valued random variables. However, much insight can be gained by first considering the simpler cases of real- and vector-valued random variables, which are discussed in Section 14.2 and Section 14.3, respectively. We take on the study of function-valued random variables when discussing random fields in Section 14.4. In Section 14.5, we discuss the notion of Radon–Nikodym derivatives, which play a key role in our discussions of Bayesian inversion in a function space. We then consider measures on Hilbert spaces in Section 14.6. This is followed by a brief discussion of the notions of distance between probability measures on Hilbert spaces in Section 14.7.

Chapter 15: Bayesian inversion in a Hilbert space: Basic formulation

2026-02-27T05:30:06Z

Other Titles in Applied Mathematics, Page 221-232.
Excerpt In this chapter, we consider Bayesian inverse problems in an infinite-dimensional Hilbert space setting. We focus on key aspects of formulating such inverse problems and provide a brief discussion of the underlying theory. The study of the associated computational methods is deferred to Chapters 16–18. As seen shortly, the formulation of a Bayesian inverse problem in a function space is rather abstract. One may pose the following question: why not first “discretize the problem” and then consider a finite-dimensional Bayesian inverse problem? We can then perform Bayesian inference as done usually in finite dimensions. Would that not be more straightforward compared to the foray into function space formulations? We need to discretize the problem to facilitate numerical computations anyway. As seen shortly, things are not so simple.

Chapter 16: A primer on solving PDE-based Bayesian inverse problems

2026-02-27T05:30:06Z

Other Titles in Applied Mathematics, Page 233-243.
Excerpt The next step in our study of Bayesian inverse problems is to consider computational methods. Before delving into details of such methods, it is instructive to consider some numerical illustrations. This is the goal of this chapter, which is similar in structure to the primer in Chapter 2. We again consider the problem of estimating the initial state in a one-dimensional heat equation. Focusing on the Bayesian formulation of this inverse problem, we provide some insight into the common steps in solving infinite-dimensional Bayesian inverse problems.

Chapter 17: Computational methods for linear(ized) Bayesian inverse problems

2026-02-27T05:30:07Z

Other Titles in Applied Mathematics, Page 245-254.
Excerpt In the previous chapter, we noted several key challenges associated with the numerical solution of infinite-dimensional Bayesian linear inverse problems. In this chapter, we delve deeper into computational methods for this class of problems. Since we have already seen the process of discretizing such inverse problems in Chapter 16, here we focus on discretized problem formulations. To make matters precise, we focus on estimating [math] from the model(17.1)\(y = \text Fm + \eta\)Such models arise from discretizing an infinite-dimensional linear (or linearized) inverse problem on a spatial domain in one, two, or three space dimensions. As such, the discretized parameter dimension [math] may be very large, e.g., in the tens or hundreds of thousands in three-dimensional geometries; see, e.g., [12, 75, 137]. As before, [math] is a vector of measurement data, [math] is a discretized linear parameter-to-observable map, and [math] is a random vector modeling measurement noise. We assume [math] and endow the inversion parameter with a Gaussian prior law [math], defined on [math].

Chapter 18: Computational methods for nonlinear Bayesian inverse problems

2026-02-27T05:30:07Z

Other Titles in Applied Mathematics, Page 255-262.
Excerpt We focus on the problem setup outlined in Chapter 15. Considering a discretized version of (15.1), we focus on estimation of [math] using the model(18.1)\(y = f(m) + \eta.\)Here, [math] is a discretized inversion parameter, [math] is a discretized nonlinear parameter-to-observable mapping, and [math] is a vector of measurement data. In inverse problems governed by PDEs, evaluating [math] requires solving a governing PDE numerically and applying an observation operator. As before, we assume [math] and assume a Gaussian prior for the inversion parameter. As discussed in the previous chapters, the prior needs to be defined in a proper function space setting before discretization. Here, the discretized prior measure [math] is a Gaussian measure on [math]. Furthermore, we assume the underlying inverse problem involves estimation of a spatially distributed parameterfield on a two- or three-dimensional domain. Finally, we consider the case where the prior covariance operator is a discretization of a squared inverse elliptic differential operator as in (16.6).

Chapter 19: Inverse problems with additional model uncertainty

2026-02-27T05:30:07Z

Other Titles in Applied Mathematics, Page 263-268.
Excerpt In an inverse problem governed by a system of PDEs, we typically seek to estimate an inversion parameter while assuming full knowledge of the remaining model parameters. Let us consider an example. Recall the inverse problem from Exercise 9.5, where the goal was to estimate the boundary term [math] in(19.1)\(-\nabla\cdot (\kappa\nabla u)=f\quad \text{in}\ \mathcal{D},\) (19.2)\(u = S_D \quad \text{on}\ \Gamma_{\text D},\) (19.3)\(\kappa\nabla u\cdot \boldsymbol n = S_N \quad \text{on}\ \Gamma_{\text N},\) (19.4)\(\kappa\nabla u\cdot \boldsymbol n + e^m u = S_R \quad \text{on}\ \Gamma_{\text R}.\) The domain [math] is as depicted in Figure 9.3 and the different parts of the boundary are defined in that exercise. In this example, the focus is on estimating [math], using measurements of [math] on [math]. However, to fully specify the model, we need to specify the coefficient-field [math] as well as the source terms [math], [math], [math], and [math]. In practice, we may not know some of these auxiliary parameters exactly. For example, if we are considering heat transfer within a heterogeneous medium, [math] is a spatially varying diffusivity coefficient that is typically subject to uncertainty.

Chapter 2: A primer on solving PDE-based deterministic inverse problems

2026-02-27T05:30:02Z

Other Titles in Applied Mathematics, Page 11-24.
Excerpt In this chapter, we illustrate the typical steps in solving ill-posed inverse problems governed by PDEs. Our driving example is the backward heat equation considered in Section 1.1. The discussions that follow highlight the key features of ill-posed inverse problems, the associated challenges, and the ideas behind regularization. The present study also helps motivate the topics in the upcoming chapters, where we delve deeper into various aspects of solving ill-posed inverse problems governed by PDEs.

Chapter 20: Optimal design of experiments

2026-02-27T05:30:03Z

Other Titles in Applied Mathematics, Page 269-289.
Excerpt A key ingredient of formulating and solving an inverse problem is data. The information content of data largely determines the quality of the solution to the inverse problem. However, acquiring data can be costly—it may be time-consuming or may require use of a few expensive-to-deploy measurement devices. In such cases, it is crucial to collect data judiciously. A naive data acquisition strategy may lead to waste of resources and potentially inaccurate parameter estimates. These considerations motivate the following fundamental question: given a set of experimental constraints, how can we collect data optimally? This is a problem of optimal experimental design (OED) [27, 365].

Chapter 3: Functional analysis background

2026-02-27T05:30:02Z

Other Titles in Applied Mathematics, Page 27-43.
Excerpt In this chapter, we discuss the core background material on functional analysis and Hilbert space theory needed in the rest of the book. In Sections 3.1, 3.2, and 3.3 we present some basic concepts regarding normed vector spaces and Hilbert spaces. Then, in Section 3.4, we briefly discuss some classes of linear transformations that occur commonly in our study of infinite-dimensional deterministic and Bayesian inverse problems. The presentation here is brief. Throughout, we list a number of references for details on the topics discussed. Additional recommendations for further reading can be found in Section 3.5.

Chapter 4: Calculus of variations

2026-02-27T05:30:03Z

Other Titles in Applied Mathematics, Page 45-59.
Excerpt Infinite-dimensional deterministic inverse problems are commonly formulated in terms of minimization of suitably defined functionals on function spaces. Understanding such problems requires tools from calculus of variations. This is the topic of this chapter. We begin our discussion in Section 4.1 by discussing a basic problem from calculus of variations—the shortest path problem. This motivates the subsequent discussion in Section 4.2, where we consider some classical theory and some well-known examples. To provide further insight regarding the basic tools discussed, we also consider a traditional application from analytical mechanics in Section 4.3. In Section 4.4, we consider a few examples that commonly occur in the discussions regarding inverse problems governed by PDEs. We conclude this chapter in Section 4.5, where we discuss notions of differentiability as well as the gradient and Hessian of functionals on Hilbert spaces. Our coverage here is brief. The reader is encouraged to consult books on calculus of variations and its applications for further details. See Section 4.6 for some references.

Chapter 5: Finite element discretizations

2026-02-27T05:30:03Z

Other Titles in Applied Mathematics, Page 61-71.
Excerpt In this book, we primarily focus on the finite element method for spatial discretization of PDEs. This chapter includes a brief coverage of the continuous Galerkin finite element method. In Section 5.1, we discuss finite element discretization of a linear elliptic PDE in two space dimensions. In Section 5.2, we consider the heat equation in a two-dimensional domain and discuss its spatial and temporal discretization. This is done to elucidate the temporal discretization of timedependent PDEs when a finite element approach is used for spatial discretization. To provide some insight regarding numerical solution of nonlinear PDEs, we consider a nonlinear elliptic PDE in Section 5.3.

Chapter 6: The Conjugate Gradient method

2026-02-27T05:30:04Z

Other Titles in Applied Mathematics, Page 73-81.
Excerpt In this chapter, we discuss the Conjugate Gradient (CG) method for solving(6.1)\(\text{A}x = b,\)where [math] is a symmetric positive definite matrix. Such linear systems arise commonly when considering inverse problems. For example, when using Tikhonov regularization for linear inverse problems, we need to solve a linear system with a symmetric positive definite coefficient matrix; see, e.g., (2.17). As another example, when using the Gauss–Newton method for solving inverse problems, a system with a symmetric positive definite coefficient matrix is solved in each iteration to find a search direction; see Chapter 7.

Chapter 7: Optimization algorithms

2026-02-27T05:30:04Z

Other Titles in Applied Mathematics, Page 83-94.
Excerpt The deterministic inverse problems considered in the present book are formulated as optimization problems over function spaces. Once discretized, we need to deploy an optimization algorithm to solve such problems. We focus primarily on Newton-type methods for such optimization problems. Accordingly, the aim of this chapter is to cover the requisite background regarding line search methods, and Newton’s method along with some of its popular variants. Throughout, we consider optimization problems of the form(7.1)\begin{equation}\min_{x \in \mathbb{R}^n} J(x),\end{equation}where [math] is an objective function.