Gaussian Process Emulator

One type of MachineLearningTool we provide for emulation is a Gaussian process. Gaussian processes are a generalization of the Gaussian probability distribution, extended to functions rather than random variables. They can be used for statistical emulation, as they provide both mean and covariances. To build a Gaussian process, we first define a prior over all possible functions, by choosing the covariance function or kernel. The kernel describes how similar two outputs (y_i, y_j) are, given the similarities between their input values (x_i, x_j). Kernels encode the functional form of these relationships and are defined by hyperparameters, which are usually initially unknown to the user. To learn the posterior Gaussian process, we condition on data using Bayes theorem and optimize the hyperparameters of the kernel. Then, we can make predictions to predict a mean function and covariance for new data points.

A useful resource to learn about Gaussian processes is Rasmussen and Williams (2006).

User Interface

CalibrateEmulateSample.jl allows the Gaussian process emulator to be built using either GaussianProcesses.jl or ScikitLearn.jl. Different packages may be optimized for different settings, we recommend users give both a try, and checkout the individual package documentation to make a choice for their problem setting.

To use GaussianProcesses.jl, define the package type as

gppackage = Emulators.GPJL()

To use ScikitLearn.jl, define the package type as

gppackage = Emulators.SKLJL()

Initialize a basic Gaussian Process with

gauss_proc = GaussianProcess(gppackage)

This initializes the prior Gaussian process. We train the Gaussian process by feeding the gauss_proc alongside the data into the Emulator struct and optimizing the hyperparameters, described here.

Prediction Type

You can specify the type of prediction when initializing the Gaussian Process emulator. The default type of prediction is to predict data, YType(). You can create a latent function type prediction with

gauss_proc = GaussianProcess(
    gppackage,
    prediction_type = FType())

Kernels

The Gaussian process above assumes the default kernel: the Squared Exponential kernel, also called the Radial Basis Function (RBF). A different type of kernel can be specified when the Gaussian process is initialized. Read more about kernel options here.

GPJL

For the GaussianProcess.jl package, there are a range of kernels to choose from. For example,

using GaussianProcesses
my_kernel = GaussianProcesses.Mat32Iso(0., 0.)      # Create a Matern 3/2 kernel with lengthscale=0 and sd=0
gauss_proc = GaussianProcess(
    gppackage;
    kernel = my_kernel )

You do not need to provide useful hyperparameter values when you define the kernel, as these are learned in optimize_hyperparameters!(emulator).

You can also combine kernels together through linear operations, for example,

using GaussianProcesses
kernel_1 = GaussianProcesses.Mat32Iso(0., 0.)      # Create a Matern 3/2 kernel with lengthscale=0 and sd=0
kernel_2 = GaussianProcesses.Lin(0.)               # Create a linear kernel with lengthscale=0
my_kernel = kernel_1 + kernel_2                    # Create a new additive kernel
gauss_proc = GaussianProcess(
    gppackage;
    kernel = my_kernel )

SKLJL

Alternatively if you are using the ScikitLearn.jl package, you can find the list of kernels here. These need this preamble:

using PyCall
using ScikitLearn
const pykernels = PyNULL()
function __init__()
    copy!(pykernels, pyimport("sklearn.gaussian_process.kernels"))
end

Then they are accessible, for example, as

my_kernel = pykernels.RBF(length_scale = 1)
gauss_proc = GaussianProcess(
    gppackage;
    kernel = my_kernel )

You can also combine multiple ScikitLearn kernels via linear operations in the same way as above.

Learning additional white noise

Often it is useful to learn the discrepancy between the Gaussian process prediction and the data, by learning additional white noise. Though one often knows, and provides, the discrepancy between the true model and data with an observational noise covariance; the additional white kernel can help account for approximation error from the selected Gaussian process kernel and the true model. This is added with the Boolean keyword noise_learn when initializing the Gaussian process. The default is true.

gauss_proc = GaussianProcess(
    gppackage;
    noise_learn = true )

When noise_learn is true, an additional white noise kernel is added to the kernel. This white noise is present across all parameter values, including the training data. The scale parameters of the white noise kernel are learned in optimize_hyperparameters!(emulator).

You may not need to learn the noise if you already have a good estimate of the noise from your training data, and if the Gaussian process kernel is well specified. When noise_learn is false, a small additional regularization is added for stability. The default value is 1e-3 but this can be chosen through the optional argument alg_reg_noise:

gauss_proc = GaussianProcess(
    gppackage;
    noise_learn = false,
    alg_reg_noise = 1e-3 )