ReferenceStats

ReferenceStatistics{FT <: Real, IT <: Integer}

A structure containing statistics from the reference model used to define a well-posed inverse problem.

Fields

• y::Vector{FT} where FT<:Real: Reference data, length: nSim * nvars * nzLevels (possibly reduced by PCA)

• Γ::AbstractMatrix{FT} where FT<:Real: Data covariance matrix, dims: (y,y) (possibly reduced by PCA)

• norm_vec::Array{Vector{FT}, 1} where FT<:Real: Vector (length: nSim) of normalizing factors (length: n_vars)

• pca_vec::Array{Union{LinearAlgebra.UniformScaling, AbstractMatrix{FT}}, 1} where FT<:Real: Vector (length: nSim) of PCA projection matrices with leading eigenvectors as columns

• y_full::Vector{FT} where FT<:Real: Full reference data vector, length: nSim * n_vars * n_zLevels

• Γ_full::SparseArrays.SparseMatrixCSC{FT, Int64} where FT<:Real: Full covariance matrix, dims: (y,y)

• ndof_case::Vector{IT} where IT<:Integer: Degrees of freedom per case (possibly reduced by PCA)

• ndof_full_case::Vector{IT} where IT<:Integer: Full degrees of freedom per case: zdof * n_vars

• zdof::Vector{IT} where IT<:Integer: Vertical degrees of freedom in profiles per case

Constructors

ReferenceStatistics(
    RM::Vector{ReferenceModel};
    perform_PCA::Bool = true,
    normalize::Bool = true,
    variance_loss::FT = 0.1,
    tikhonov_noise::FT = 0.0,
    tikhonov_mode::String = "absolute",
    dim_scaling::Bool = false,
    time_shift::FT = 6 * 3600.0,
    model_errors::OptVec{T} = nothing,
    obs_var_scaling::Union{Dict, Nothing} = nothing,
) where {FT <: Real}

Constructs the ReferenceStatistics defining the inverse problem.

Inputs:

• RM :: Vector of ReferenceModels.
• perform_PCA :: Boolean specifying whether to perform PCA.
• normalize :: Boolean specifying whether to normalize the data.
• variance_loss :: Fraction of variance loss when performing PCA.
• tikhonov_noise :: Tikhonov regularization factor for covariance matrices.
• tikhonov_mode :: If "relative", tikhonov_noise is scaled by the maximum eigenvalue in the covariance matrix considered, having the interpretation of the inverse of the desired condition number. This value is enforced to be larger than the sqrt of the machine precision for stability.
• dim_scaling :: Whether to scale covariance blocks by their size.
• time_shift :: [LES last time - SCM start time (LES timeframe)] for LES_driven_SCM cases.
• model_errors :: Vector of model errors added to the internal variability noise, each containing the model error per variable normalized by the pooled variable variance.
• obs_var_scaling :: Scaling factor for observational variance estimate

get_obs(m::ReferenceModel, y_names, Σ_names, normalize; [z_scm])
get_obs(m::ReferenceModel, normalize; [z_scm])

Get observational mean y and empirical time covariance Σ for the ReferenceModel m.

Typically, the observations are fetched by specifying the ReferenceModel, which indicates if the data is generated by the SCM or LES. Alternatively, vectors of variable names, y_names, Σ_names, can be specified directly. Note: Σ_names may be different than y_names if there are LES/SCM name discrepancies.

The keyword normalize specifies whether observations are to be normalized with respect to the per-quantity pooled variance or not. See normalize_profile for details. The normalization vector is return along with y and Σ.

If z_scm is given, interpolate observations to the given levels.

Arguments

• m :: A ReferenceModel](@ref)
• y_names :: Names of observed fields from the ReferenceModel m.
• Σ_names :: Names of fields used to construct covariances, may be different than y_names if there are LES/SCM name discrepancies.
• normalize :: Whether to normalize the observations.
• obs_var_scaling :: Scaling factor for observational variance estimate

Keywords

• z_scm :: If given, interpolate LES observations to given array of vertical levels.
• model_error :: Model error per variable, added to the internal variability noise, and normalized by the pooled variance of the variable.

Returns

• y::Vector :: Mean of observations y, possibly interpolated to z_scm levels.
• Σ::Matrix :: Observational covariance matrix Σ, possibly pool-normalized.
• norm_vec::Vector :: Vertically averaged time-variance, one entry for each variable

obs_PCA(y_mean, y_var, allowed_var_loss = 1.0e-1)

Perform dimensionality reduction using principal component analysis on the variance y_var. Only eigenvectors with eigenvalues that contribute to the leading 1-allowed_var_loss variance are retained. Inputs:

• y_mean :: Mean of the observations.
• y_var :: Variance of the observations.
• allowed_var_loss :: Maximum variance loss allowed.

Outputs:

• y_pca :: Projection of y_mean onto principal subspace spanned by eigenvectors.
• y_var_pca :: Projection of y_var on principal subspace.
• P_pca :: Projection matrix onto principal subspace, with leading eigenvectors as columns.

pca(covmat::AbstractMatrix{FT}, allowed_var_loss::FT) where {FT <: Real}

Perform dimensionality reduction using principal component analysis on the variance covmat.

Inputs:

• covmat :: Variance of the observations.
• allowed_var_loss :: Maximum variance loss allowed.

Outputs:

• λ_pca :: Principal eigenvalues, ordered in increasing value order.
• P_pca :: Projection matrix onto principal subspace, with leading eigenvectors as columns.

Returns dimensionality of the ReferenceStatistics in low-dimensional latent space

Returns full dimensionality of the ReferenceStatistics, before latent space encoding

get_profile(
    filename::String,
    y_names::Vector{String};
    ti::Real = 0.0,
    tf::OptReal = nothing,
    z_scm::Union{Vector{T}, T} = nothing,
    prof_ind::Bool = false,
) where {T}
get_profile(
    m::ReferenceModel,
    filename::String;
    z_scm::Union{Vector{T}, T} = nothing,
    prof_ind::Bool = false,
) where {T}
get_profile(
    m::ReferenceModel,
    filename::String,
    y_names::Vector{String};
    z_scm::Union{Vector{T}, T} = nothing,
    prof_ind::Bool = false,
) where {T}

Get time-averaged profiles for variables y_names, interpolated to z_scm (if given), and concatenated into a single output vector.

Inputs:

• filename :: nc filename
• y_names :: Names of variables to be retrieved.
• ti :: Initial time of averaging window.
• tf :: Final time of averaging window.
• z_scm :: If given, interpolate LES observations to given levels.
• m :: ReferenceModel from which to fetch profiles, implicitly defines ti and tf.
• prof_ind :: Whether to return a boolean array indicating the variables that are profiles (i.e., not scalars).

Outputs:

• y :: Output vector used in the inverse problem, which concatenates the requested profiles.

get_time_covariance(
    m::ReferenceModel,
    y_names::Vector{String},
    z_scm::Vector{FT};
    normalize::Bool = true,
    model_error::OptVec{FT} = nothing,
    obs_var_scaling::Union{Dict, Nothing} = nothing,
) where {FT <: Real}

Obtain the covariance matrix of a group of profiles, where the covariance is obtained in time.

Inputs:

• m :: Reference model.
• y_names :: List of variable names to be included.
• z_scm :: If given, interpolates covariance matrix to this locations.
• normalize :: Whether to normalize the time series with the pooled variance before computing the covariance, or not.
• model_error :: Model error per variable, added to the internal variability noise, and normalized by the pooled variance of the variable.
• obs_var_scaling :: Scaling factor for observational variance estimate