Utilities

struct CanonicalCorrelation{VV1, VV2, VV3, FT, VV4} <: CalibrateEmulateSample.Utilities.PairedDataContainerProcessor

Uses both input and output data to learn a subspace of maximal correlation between inputs and outputs. The subspace for a pair (X,Y) will be of size minimum(rank(X),rank(Y)), computed using SVD-based method e.g. See e.g., https://numerical.recipes/whp/notes/CanonCorrBySVD.pdf

Preferred construction is with the canonical_correlation method

Fields

• data_mean::Any: storage for the input or output data mean

• encoder_mat::Any: the encoding matrix of input or output canonical correlations

• decoder_mat::Any: the decoding matrix of input or output canonical correlations

• retain_var::Any: the fraction of variance to be retained after truncating singular values (1 implies no truncation)

• apply_to::Any: Stores whether this is an input or output encoder (vector with string "in" or "out")

struct Decorrelator{VV1, VV2, VV3, FT, AS<:AbstractString} <: CalibrateEmulateSample.Utilities.DataContainerProcessor

Decorrelate the data via taking an SVD decomposition and projecting onto the singular-vectors.

Preferred construction is with the methods

• decorrelate_structure_mat
• decorrelate_sample_cov
• decorrelate

For decorrelate_structure_mat: The SVD is taken over a structure matrix (e.g., prior_cov for inputs, obs_noise_cov for outputs). The structure matrix will become exactly I after processing.

For decorrelate_sample_cov: The SVD is taken over the estimated covariance of the data. The data samples will have a Normal(0,I) distribution after processing,

For decorrelate(;decorrelate_with="combined") (default): The SVD is taken to be the sum of structure matrix and estimated covariance. This may be more robust to ill-specification of structure matrix, or poor estimation of the sample covariance.

Fields

• data_mean::Any: storage for the data mean

• encoder_mat::Any: the matrix used to perform encoding

• decoder_mat::Any: the inverse of the the matrix used to perform encoding

• retain_var::Any: the fraction of variance to be retained after truncating singular values (1 implies no truncation)

• decorrelate_with::AbstractString: Switch to choose what form of matrix to use to decorrelate the data

• structure_mat_name::Union{Nothing, Symbol}: When given, use the structure matrix by this name if decorrelate_with uses structure matrices. When nothing, try to use the only present structure matrix instead.

struct ElementwiseScaler{T, VV<:(AbstractVector)} <: CalibrateEmulateSample.Utilities.DataContainerProcessor

The ElementwiseScaler{T} will create an encoding of the data_container via elementwise affine transformations.

Different methods T will build different transformations:

• quartile_scale : creates QuartileScaling,
• minmax_scale : creates MinMaxScaling
• zscore_scale : creates ZScoreScaling

and are accessed with get_type

canonical_correlation(
;
    retain_var
) -> CalibrateEmulateSample.Utilities.CanonicalCorrelation{Vector{Any}, Vector{Any}, Vector{Any}, Float64, Vector{AbstractString}}

Constructs the CanonicalCorrelation struct. Can optionally provide the keyword

• retain_var[=1.0]: to project onto the leading singular vectors (of the input-output product) such that retain_var variance is retained.

Create size-1 encoder schedule with a tuple of (DataProcessor1(...), apply_to) with apply_to = "in", "out" or "in_and_out".

create_encoder_schedule(
    schedule_in::AbstractVector
) -> Vector{Any}

Create a flatter encoder schedule for the from the user's proposed schedule of the form:

enc_schedule = [
    (DataProcessor1(...), "in"), 
    (DataProcessor2(...), "out"), 
    (PairedDataProcessor3(...), "in"), 
    (DataProcessor4(...), "in_and_out"), 
]

This function creates the encoder scheduler that is also machine readable. E.g.,

enc_schedule = [
    (DataProcessor1(...), "in"), 
    (DataProcessor2(...), "out"), 
    (DataProcessor2(...), "out"),
    (PairedDataProcessor3(...),"in"), 
    (DataProcessor4(...), "in"),
    (DataProcessor4(...), "out"), 
]

and the decoder schedule is a copy of the encoder schedule reversed (and processors copied)

decode_data(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation,
    data::AbstractMatrix
) -> Any

Apply the CanonicalCorrelation decoder, on a columns-are-data matrix

decode_data(
    dd::CalibrateEmulateSample.Utilities.Decorrelator,
    data::AbstractMatrix
) -> Any

Apply the Decorrelator decoder, on a columns-are-data matrix

decode_data(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler,
    data::AbstractMatrix
) -> AbstractMatrix

Apply the ElementwiseScaler decoder, on a columns-are-data matrix

decode_structure_matrix(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation,
    enc_structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the CanonicalCorrelation decoder to a provided structure matrix

decode_structure_matrix(
    dd::CalibrateEmulateSample.Utilities.Decorrelator,
    enc_structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the Decorrelator decoder to a provided structure matrix

decode_structure_matrix(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler,
    enc_structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the ElementwiseScaler decoder to a provided structure matrix

decode_with_schedule(
    encoder_schedule::AbstractVector,
    data_container::EnsembleKalmanProcesses.DataContainers.DataContainer,
    in_or_out::AbstractString
) -> EnsembleKalmanProcesses.DataContainers.DataContainer

Takes in an already initialized encoder schedule, and decodes a DataContainer, the in_or_out string indicates if the data is input "in" or output "out" data (and thus decoded differently)

decode_with_schedule(
    encoder_schedule::AbstractVector,
    structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix},
    in_or_out::AbstractString
) -> Any

Takes in an already initialized encoder schedule, and decodes a structure matrix, the in_or_out string indicates if the structure matrix is for input "in" or output "out" space (and thus decoded differently)

decode_with_schedule(
    encoder_schedule::AbstractVector,
    io_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer,
    input_structure_mat::Union{LinearAlgebra.UniformScaling, AbstractMatrix},
    output_structure_mat::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Tuple{EnsembleKalmanProcesses.DataContainers.PairedDataContainer, Any, Any}

Takes in an already initialized encoder schedule, and decodes a DataContainer, and structure matrices with it, the in_or_out string indicates if the data is input "in" or output "out" data (and thus decoded differently)

decorrelate(
;
    retain_var,
    decorrelate_with,
    structure_mat_name
) -> CalibrateEmulateSample.Utilities.Decorrelator{Vector{Any}, Vector{Any}, Vector{Any}, Float64, String}

Constructs the Decorrelator struct. Users can add optional keyword arguments:

• retain_var[=1.0]: to project onto the leading singular vectors such that retain_var variance is retained
• decorrelate_with [="combined"]: from which matrix do we provide subspace directions, options are
  • "structure_mat", see decorrelate_structure_mat
  • "sample_cov", see decorrelate_sample_cov
  • "combined", sums the "sample_cov" and "structure_mat" matrices

decorrelate_sample_cov(
;
    retain_var
) -> CalibrateEmulateSample.Utilities.Decorrelator{Vector{Any}, Vector{Any}, Vector{Any}, Float64, String}

Constructs the Decorrelator struct, setting decorrelatewith = "samplecov". Encoding data with this will ensure that the distribution of data samples after encoding will be Normal(0,I). One can additionally add keywords:

• retain_var[=1.0]: to project onto the leading singular vectors such that retain_var variance is retained

decorrelate_structure_mat(
;
    retain_var,
    structure_mat_name
) -> CalibrateEmulateSample.Utilities.Decorrelator{Vector{Any}, Vector{Any}, Vector{Any}, Float64, String}

Constructs the Decorrelator struct, setting decorrelatewith = "structuremat". This encoding will transform a provided structure matrix into I. One can additionally add keywords:

• retain_var[=1.0]: to project onto the leading singular vectors such that retain_var variance is retained

encode_data(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation,
    data::AbstractMatrix
) -> Any

Apply the CanonicalCorrelation encoder, on a columns-are-data matrix

encode_data(
    dd::CalibrateEmulateSample.Utilities.Decorrelator,
    data::AbstractMatrix
) -> Any

Apply the Decorrelator encoder, on a columns-are-data matrix

encode_data(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler,
    data::AbstractMatrix
) -> AbstractMatrix

Apply the ElementwiseScaler encoder, on a columns-are-data matrix

encode_structure_matrix(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation,
    structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the CanonicalCorrelation encoder to a provided structure matrix

encode_structure_matrix(
    dd::CalibrateEmulateSample.Utilities.Decorrelator,
    structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the Decorrelator encoder to a provided structure matrix

encode_structure_matrix(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler,
    structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix}
) -> Any

Apply the ElementwiseScaler encoder to a provided structure matrix

encode_with_schedule(
    encoder_schedule::AbstractVector,
    data_container::EnsembleKalmanProcesses.DataContainers.DataContainer,
    in_or_out::AbstractString
) -> EnsembleKalmanProcesses.DataContainers.DataContainer

Takes in an already initialized encoder schedule, and encodes a DataContainer, the in_or_out string indicates if the data is input "in" or output "out" data (and thus encoded differently)

encode_with_schedule(
    encoder_schedule::AbstractVector,
    structure_matrix::Union{LinearAlgebra.UniformScaling, AbstractMatrix},
    in_or_out::AbstractString
) -> Any

Takes in an already initialized encoder schedule, and encodes a structure matrix, the in_or_out string indicates if the structure matrix is for input "in" or output "out" space (and thus encoded differently)

get_apply_to(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation
) -> Any

returns the apply_to field of the CanonicalCorrelation.

get_data_mean(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation
) -> Any

returns the data_mean field of the CanonicalCorrelation.

get_data_mean(
    dd::CalibrateEmulateSample.Utilities.Decorrelator
) -> Any

returns the data_mean field of the Decorrelator.

get_decoder_mat(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation
) -> Any

returns the decoder_mat field of the CanonicalCorrelation.

get_decoder_mat(
    dd::CalibrateEmulateSample.Utilities.Decorrelator
) -> Any

returns the decoder_mat field of the Decorrelator.

get_decorrelate_with(
    dd::CalibrateEmulateSample.Utilities.Decorrelator
) -> AbstractString

returns the decorrelate_with field of the Decorrelator.

get_encoder_mat(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation
) -> Any

returns the encoder_mat field of the CanonicalCorrelation.

get_encoder_mat(
    dd::CalibrateEmulateSample.Utilities.Decorrelator
) -> Any

returns the encoder_mat field of the Decorrelator.

get_retain_var(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation
) -> Any

returns the retain_var field of the CanonicalCorrelation.

get_retain_var(
    dd::CalibrateEmulateSample.Utilities.Decorrelator
) -> Any

returns the retain_var field of the Decorrelator.

get_scale(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler
) -> AbstractVector

Gets the scale field of the ElementwiseScaler

get_shift(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler
) -> AbstractVector

Gets the shift field of the ElementwiseScaler

get_training_points(
    ekp::EnsembleKalmanProcesses.EnsembleKalmanProcess{FT, IT, P},
    train_iterations::Union{AbstractVector{IT}, IT} where IT
) -> EnsembleKalmanProcesses.DataContainers.PairedDataContainer

Extract the training points needed to train the Gaussian process regression.

• ekp - EnsembleKalmanProcess holding the parameters and the data that were produced during the Ensemble Kalman (EK) process.
• train_iterations - Number (or indices) EK layers/iterations to train on.

get_type(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler{T}
) -> Any

Gets the UnivariateAffineScaling type T

initialize_and_encode_with_schedule!(
    encoder_schedule::AbstractVector,
    io_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer;
    input_structure_mats,
    output_structure_mats,
    input_structure_vecs,
    output_structure_vecs,
    prior_cov,
    obs_noise_cov,
    observation,
    prior_samples_in,
    prior_samples_out
) -> Tuple{EnsembleKalmanProcesses.DataContainers.PairedDataContainer, Dict{Symbol, Union{LinearAlgebra.UniformScaling, AbstractMatrix}}, Dict{Symbol, Union{LinearAlgebra.UniformScaling, AbstractMatrix}}, Dict{Symbol, Union{AbstractMatrix, AbstractVector}}, Dict{Symbol, Union{AbstractMatrix, AbstractVector}}}

Takes in the created encoder schedule (See create_encoder_schedule), and initializes it, and encodes the paired data container, and structure matrices with it.

initialize_processor!(
    cc::CalibrateEmulateSample.Utilities.CanonicalCorrelation,
    in_data::AbstractMatrix,
    out_data::AbstractMatrix,
    input_structure_matrices,
    output_structure_matrices,
    input_structure_vectors,
    output_structure_vectors,
    apply_to::AbstractString
) -> Any

Computes and populates the data_mean, encoder_mat, decoder_mat and apply_to fields for the CanonicalCorrelation

initialize_processor!(
    dd::CalibrateEmulateSample.Utilities.Decorrelator,
    data::AbstractMatrix,
    structure_matrices::Dict{Symbol, <:Union{LinearAlgebra.UniformScaling, AbstractMatrix}},
    _::Dict{Symbol, <:Union{AbstractMatrix, AbstractVector}}
) -> Any

Computes and populates the data_mean and encoder_mat and decoder_mat fields for the Decorrelator

initialize_processor!(
    es::CalibrateEmulateSample.Utilities.ElementwiseScaler,
    data::AbstractMatrix,
    structure_matrices,
    structure_vectors
) -> Any

Computes and populates the shift and scale fields for the ElementwiseScaler

minmax_scale(

) -> CalibrateEmulateSample.Utilities.ElementwiseScaler{CalibrateEmulateSample.Utilities.MinMaxScaling, Vector{Float64}}

Constructs ElementwiseScaler{MinMaxScaling} processor. As part of an encoder schedule, this will apply the transform $\frac{x - \min(x)}{\max(x) - \min(x)}$ to each data dimension.

quartile_scale(

) -> CalibrateEmulateSample.Utilities.ElementwiseScaler{CalibrateEmulateSample.Utilities.QuartileScaling, Vector{Float64}}

Constructs ElementwiseScaler{QuartileScaling} processor. As part of an encoder schedule, it will apply the transform $\frac{x - Q2(x)}{Q3(x) - Q1(x)}$ to each data dimension. Also known as "robust scaling"

zscore_scale(

) -> CalibrateEmulateSample.Utilities.ElementwiseScaler{CalibrateEmulateSample.Utilities.ZScoreScaling, Vector{Float64}}

Constructs ElementwiseScaler{ZScoreScaling} processor. As part of an encoder schedule, this will apply the transform $\frac{x-\mu}{\sigma}$, (where $x\sim N(\mu,\sigma)$), to each data dimension. For multivariate standardization, see Decorrelator