RandomFeatures

struct OneDimFactor <: CalibrateEmulateSample.Emulators.CovarianceStructureType

covariance structure for a one-dimensional space

struct DiagonalFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType

builds a diagonal covariance structure

struct CholeskyFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType

builds a general positive-definite covariance structure

struct LowRankFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType

builds a covariance structure that deviates from the identity with a low-rank perturbation. This perturbation is diagonalized in the low-rank space

struct HierarchicalLowRankFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType

builds a covariance structure that deviates from the identity with a more general low-rank perturbation

struct SeparableKernel{CST1<:CalibrateEmulateSample.Emulators.CovarianceStructureType, CST2<:CalibrateEmulateSample.Emulators.CovarianceStructureType} <: CalibrateEmulateSample.Emulators.KernelStructureType

Builds a separable kernel, i.e. one that accounts for input and output covariance structure separately

struct NonseparableKernel{CST<:CalibrateEmulateSample.Emulators.CovarianceStructureType} <: CalibrateEmulateSample.Emulators.KernelStructureType

Builds a nonseparable kernel, i.e. one that accounts for a joint input and output covariance structure

calculate_n_hyperparameters(
    d::Int64,
    odf::CalibrateEmulateSample.Emulators.OneDimFactor
) -> Int64

calculates the number of hyperparameters generated by the choice of covariance structure

hyperparameters_from_flat(
    x::AbstractVector,
    odf::CalibrateEmulateSample.Emulators.OneDimFactor
)

reshapes a list of hyperparameters into a covariance matrix based on the selected structure

build_default_prior(
    name::AbstractString,
    n_hp::Int64,
    odf::CalibrateEmulateSample.Emulators.OneDimFactor
)

builds a prior distribution for the kernel hyperparameters to initialize optimization.

struct ScalarRandomFeatureInterface{S<:AbstractString, RNG<:Random.AbstractRNG, KST<:CalibrateEmulateSample.Emulators.KernelStructureType} <: CalibrateEmulateSample.Emulators.RandomFeatureInterface

Structure holding the Scalar Random Feature models.

Fields

• rfms::Vector{RandomFeatures.Methods.RandomFeatureMethod}: vector of RandomFeatureMethods, contains the feature structure, batch-sizes and regularization

• fitted_features::Vector{RandomFeatures.Methods.Fit}: vector of Fits, containing the matrix decomposition and coefficients of RF when fitted to data

• batch_sizes::Union{Nothing, Dict{S, Int64}} where S<:AbstractString: batch sizes

• n_features::Union{Nothing, Int64}: n_features

• input_dim::Int64: input dimension

• rng::Random.AbstractRNG: choice of random number generator

• kernel_structure::CalibrateEmulateSample.Emulators.KernelStructureType: Kernel structure type (e.g. Separable or Nonseparable)

• feature_decomposition::AbstractString: Random Feature decomposition, choose from "svd" or "cholesky" (default)

• optimizer_options::Dict{S} where S<:AbstractString: dictionary of options for hyperparameter optimizer

• optimizer::Vector: diagnostics from optimizer

ScalarRandomFeatureInterface(
    n_features::Int64,
    input_dim::Int64;
    kernel_structure,
    batch_sizes,
    rng,
    feature_decomposition,
    optimizer_options
) -> CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface{String, Random.TaskLocalRNG, CalibrateEmulateSample.Emulators.SeparableKernel{CST1, CalibrateEmulateSample.Emulators.OneDimFactor}} where CST1<:CalibrateEmulateSample.Emulators.CovarianceStructureType

Constructs a ScalarRandomFeatureInterface <: MachineLearningTool interface for the RandomFeatures.jl package for multi-input and single- (or decorrelated-)output emulators.

• n_features - the number of random features
• input_dim - the dimension of the input space
• kernel_structure - - a prescribed form of kernel structure
• batch_sizes = nothing - Dictionary of batch sizes passed RandomFeatures.jl object (see definition there)
• rng = Random.GLOBAL_RNG - random number generator
• feature_decomposition = "cholesky" - choice of how to store decompositions of random features, cholesky or svd available
• optimizer_options = nothing - Dict of options to pass into EKI optimization of hyperparameters (defaults created in ScalarRandomFeatureInterface constructor):
  • "prior": the prior for the hyperparameter optimization
  • "priorinscale": use this to tune the input prior scale
  • "n_ensemble": number of ensemble members
  • "n_iteration": number of eki iterations
  • "covsamplemultiplier": increase for more samples to estimate covariance matrix in optimization (default 10.0, minimum 0.0)
  • "scheduler": Learning rate Scheduler (a.k.a. EKP timestepper) Default: DataMisfitController
  • "tikhonov": tikhonov regularization parameter if >0
  • "inflation": additive inflation ∈ [0,1] with 0 being no inflation
  • "train_fraction": e.g. 0.8 (default) means 80:20 train - test split
  • "nfeaturesopt": fix the number of features for optimization (default n_features, as used for prediction)
  • "multithread": how to multithread. "ensemble" (default) threads across ensemble members "tullio" threads random feature matrix algebra
  • "accelerator": use EKP accelerators (default is no acceleration)
  • "verbose" => false, verbose optimizer statements
  • "cov_correction" => "shrinkage", type of conditioning to improve estimated covariance (Ledoit Wolfe 03), also "nice" for (Vishny, Morzfeld et al. 2024)
  • "ncrossvalsets" => 2, train fraction creates (default 5) train-test data subsets, then use 'ncrossvalsets' of these stacked in the loss function. If set to 0, train=test on the full data provided ignoring "train_fraction".

build_models!(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
    input_output_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer{FT<:AbstractFloat}
) -> Union{Nothing, Vector}

Builds the random feature method from hyperparameters. We use cosine activation functions and a Multivariate Normal distribution (from Distributions.jl) with mean M=0, and input covariance U built with the CovarianceStructureType.

predict(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
    new_inputs::AbstractMatrix;
    multithread
) -> Tuple{Any, Any}

Prediction of data observation (not latent function) at new inputs (passed in as columns in a matrix). That is, we add the observational noise into predictions.

struct VectorRandomFeatureInterface{S<:AbstractString, RNG<:Random.AbstractRNG, KST<:CalibrateEmulateSample.Emulators.KernelStructureType} <: CalibrateEmulateSample.Emulators.RandomFeatureInterface

Structure holding the Vector Random Feature models.

Fields

• rfms::Vector{RandomFeatures.Methods.RandomFeatureMethod}: A vector of RandomFeatureMethods, contains the feature structure, batch-sizes and regularization

• fitted_features::Vector{RandomFeatures.Methods.Fit}: vector of Fits, containing the matrix decomposition and coefficients of RF when fitted to data

• batch_sizes::Union{Nothing, Dict{S, Int64}} where S<:AbstractString: batch sizes

• n_features::Union{Nothing, Int64}: number of features

• input_dim::Int64: input dimension

• output_dim::Int64: output_dimension

• rng::Random.AbstractRNG: rng

• regularization::Vector{Union{LinearAlgebra.Diagonal, LinearAlgebra.UniformScaling, Matrix}}: regularization

• kernel_structure::CalibrateEmulateSample.Emulators.KernelStructureType: Kernel structure type (e.g. Separable or Nonseparable)

• feature_decomposition::AbstractString: Random Feature decomposition, choose from "svd" or "cholesky" (default)

• optimizer_options::Dict: dictionary of options for hyperparameter optimizer

• optimizer::Vector: diagnostics from optimizer

VectorRandomFeatureInterface(
    n_features::Int64,
    input_dim::Int64,
    output_dim::Int64;
    kernel_structure,
    batch_sizes,
    rng,
    feature_decomposition,
    optimizer_options
) -> CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface{String, Random.TaskLocalRNG, CalibrateEmulateSample.Emulators.SeparableKernel{CST1, CST2}} where {CST1<:Union{CalibrateEmulateSample.Emulators.OneDimFactor, CalibrateEmulateSample.Emulators.CholeskyFactor{Float64}, CalibrateEmulateSample.Emulators.DiagonalFactor{Float64}, CalibrateEmulateSample.Emulators.HierarchicalLowRankFactor{Float64}, CalibrateEmulateSample.Emulators.LowRankFactor{Float64}}, CST2<:Union{CalibrateEmulateSample.Emulators.OneDimFactor, CalibrateEmulateSample.Emulators.CholeskyFactor{Float64}, CalibrateEmulateSample.Emulators.DiagonalFactor{Float64}, CalibrateEmulateSample.Emulators.HierarchicalLowRankFactor{Float64}, CalibrateEmulateSample.Emulators.LowRankFactor{Float64}}}

Constructs a VectorRandomFeatureInterface <: MachineLearningTool interface for the RandomFeatures.jl package for multi-input and multi-output emulators.

• n_features - the number of random features
• input_dim - the dimension of the input space
• output_dim - the dimension of the output space
• kernel_structure - - a prescribed form of kernel structure
• batch_sizes = nothing - Dictionary of batch sizes passed RandomFeatures.jl object (see definition there)
• rng = Random.GLOBAL_RNG - random number generator
• feature_decomposition = "cholesky" - choice of how to store decompositions of random features, cholesky or svd available
• optimizer_options = nothing - Dict of options to pass into EKI optimization of hyperparameters (defaults created in VectorRandomFeatureInterface constructor):
  • "prior": the prior for the hyperparameter optimization
  • "priorinscale"/"prioroutscale": use these to tune the input/output prior scale.
  • "n_ensemble": number of ensemble members
  • "n_iteration": number of eki iterations
  • "scheduler": Learning rate Scheduler (a.k.a. EKP timestepper) Default: DataMisfitController
  • "covsamplemultiplier": increase for more samples to estimate covariance matrix in optimization (default 10.0, minimum 0.0)
  • "tikhonov": tikhonov regularization parameter if > 0
  • "inflation": additive inflation ∈ [0,1] with 0 being no inflation
  • "train_fraction": e.g. 0.8 (default) means 80:20 train - test split
  • "nfeaturesopt": fix the number of features for optimization (default n_features, as used for prediction)
  • "multithread": how to multithread. "ensemble" (default) threads across ensemble members "tullio" threads random feature matrix algebra
  • "accelerator": use EKP accelerators (default is no acceleration)
  • "verbose" => false, verbose optimizer statements to check convergence, priors and optimal parameters.
  • "cov_correction" => "shrinkage", type of conditioning to improve estimated covariance (Ledoit Wolfe 03), also "nice" for (Vishny, Morzfeld et al. 2024)
  • "ncrossvalsets" => 2, train fraction creates (default 5) train-test data subsets, then use 'ncrossvalsets' of these stacked in the loss function. If set to 0, train=test on the full data provided ignoring "train_fraction".

build_models!(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
    input_output_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer{FT<:AbstractFloat};
    regularization_matrix
) -> Union{Nothing, Vector{Union{LinearAlgebra.Diagonal, LinearAlgebra.UniformScaling, Matrix}}}

Build Vector Random Feature model for the input-output pairs subject to regularization, and optimizes the hyperparameters with EKP.

predict(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
    new_inputs::AbstractMatrix
) -> Tuple{Any, Any}

Prediction of data observation (not latent function) at new inputs (passed in as columns in a matrix). That is, we add the observational noise into predictions.

get_rfms(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.RandomFeatureMethod}

gets the rfms field

get_rfms(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.RandomFeatureMethod}

Gets the rfms field

get_fitted_features(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.Fit}

gets the fitted_features field

get_fitted_features(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.Fit}

Gets the fitted_features field

get_batch_sizes(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Union{Nothing, Dict{S, Int64}} where S<:AbstractString

gets batch_sizes the field

get_batch_sizes(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Union{Nothing, Dict{S, Int64}} where S<:AbstractString

Gets the batch_sizes field

get_n_features(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Union{Nothing, Int64}

gets the n_features field

get_n_features(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Union{Nothing, Int64}

Gets the n_features field

get_input_dim(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Int64

gets the input_dim field

get_input_dim(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Int64

Gets the input_dim field

get_output_dim(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Int64

Gets the output_dim field

get_rng(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Random.AbstractRNG

gets the rng field

get_rng(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Random.AbstractRNG

Gets the rng field

get_kernel_structure(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> CalibrateEmulateSample.Emulators.KernelStructureType

Gets the kernel_structure field

get_kernel_structure(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> CalibrateEmulateSample.Emulators.KernelStructureType

Gets the kernel_structure field

get_feature_decomposition(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> AbstractString

gets the feature_decomposition field

get_feature_decomposition(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> AbstractString

Gets the feature_decomposition field

get_optimizer_options(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Dict{S} where S<:AbstractString

gets the optimizer_options field

get_optimizer_options(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Dict

Gets the optimizer_options field

optimize_hyperparameters!(
    srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
    args...;
    kwargs...
)

Empty method, as optimization takes place within the build_models stage

optimize_hyperparameters!(
    vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
    args...;
    kwargs...
)

Empty method, as optimization takes place within the build_models stage

shrinkage_cov(sample_mat::AbstractMatrix) -> Any

Calculate the empirical covariance, additionally applying a shrinkage operator (here the Ledoit Wolf 2004 shrinkage operation). Known to have better stability properties than Monte-Carlo for low sample sizes