HelperFuncs

struct ParameterMap

ParameterMap is a struct that defines relations between different parameters.

The dictionary mapping specifies a map from any parameter that is not being calibrated to either a calibrated parameter, or a number. If another parameter name is specified, the samples from that parameter's distribution is used as the given parameter's value throughout calibration, effectively defining an equivalence between the two parameters. If a number is specified, the parameter will be assigned that constant value throughout the calibration process.

These mappings are primarily useful for vector parameters, where we can choose to only calibrate specific components of a vector parameter, or setting different components of a vector equal to other components of the vector (or to other parameters). Note, vector components are specified by <name>_{<index>}, e.g. general_stochastic_ent_params_{3} for the third component of the vector parameter general_stochastic_ent_params.

Examples

Suppose we have the parameter vector param with five components. We can choose to only calibrate the first and third component, fixing the fourth component to 3.0, and also specifying that the first and second component should be equal, in the following way:

julia> param_map = ParameterMap(
    mapping = Dict(
        "param_{2}" => "param_{1}",
        "param_{4}" => 3.0,
    ),
)

The parameter map should be specified in the function get_prior_config() in config.jl,

# function get_prior_config()  
config = Dict()

config["constraints"] = Dict(
    "param_{1}" => [bounded(1.0, 2.0)],
    "param_{3} => [bounded(3.0, 4.0)],
)

config["param_map"] = CalibrateEDMF.HelperFuncs.ParameterMap(
    mapping = Dict(
        "param_{2}" => "param_{1}",
        "param_{4}" => 3.0,
    ),
)

# ...
# end

Notice that the fifth component was neither specified in the prior, nor the mapping. This will fix that component to its default value as specified in the namelist (i.e. in TurbulenceConvection.jl).

do-nothing param_map.

expand_params(u_names_calib, u_calib, param_map, namelist)

Expand a list of parameters using a param_map and a namelist.

Expand a list of parameters and its corresponding values using the ParameterMap. If u_names_calib contain vector components, fetch all unspecified components from the namelist so that for any vector component in u_names_calib, all other components of that vector are also specified.

Arguments

• u_names_calib: A list of parameter names that are being calibrated
• u_calib: A list of values associated with the parameter names
• param_map: A ParameterMap specifying a parameter mapping.
• namelist: A dictionary of default parameter values.

Returns a tuple of two vectors defining parameter names and parameter values, possibly expanded relative to the input arguments to define all components of vector parameters or from relations specified by the ParameterMap.

namelist_subdict_by_key(namelist, param_name) -> subdict

Return the subdict of namelist that has param_name as a key.

The namelist is defined in TurbulenceConvection.jl

vertical_interpolation(
    var_name::String,
    filename::String,
    z_scm::Vector{FT};
) where {FT <: AbstractFloat}

Returns the netcdf variable var_name interpolated to heights z_scm.

Inputs:

• var_name :: Name of variable in the netcdf dataset.
• filename :: nc filename
• z_scm :: Vertical coordinate vector onto which var_name is interpolated.

Output:

• The interpolated vector.

nc_fetch_interpolate(var_name::String, filename::String, z_scm::OptVec{<:Real})

Returns the netcdf variable var_name, possibly interpolated to heights z_scm.

Inputs:

• var_name :: Name of variable in the netcdf dataset.
• filename :: nc filename
• z_scm :: Vertical coordinate vector onto which var_name is interpolated.

Output:

• The interpolated vector.

fetch_interpolate_transform(var_name::String, filename::String, z_scm::OptVec{<:Real})

Returns the netcdf variable var_name, possibly interpolated to heights z_scm. If the variable needs to be transformed to be equivalent to an SCM variable, applies the transformation as well.

Inputs:

• var_name :: Name of variable in the netcdf dataset.
• filename :: nc filename
• z_scm :: Vertical coordinate vector onto which var_name is interpolated.

Output:

• The interpolated and transformed vector.

PyCLES variables that require transformations:

• PyCLES diagnostic vertical fluxes (defined in AuxiliaryStatistics.pyx) are specific quantities, not multiplied by density, and written at cell centers. These include all resolved_z_flux_(...) and sgs_z_flux_(...) diagnostics. For instance the resolved_z_flux_theta is $\langle{w^*\theta^*}\rangle$. In contrast, all massflux_(...), diffusive_flux_(...) and total_flux_(...) outputs from TC.jl are already multiplied by density and written at cell faces; e.g. total_flux_h is $\rho\langle{w^*\theta^*}\rangle$. The location mismatch is handled through is_face_variable and interpolation. Another difference is that the total_flux_(...) in TC.jl simulations includes the full flux, whereas the PyCLES resolved definitions only include the resolved flux. We must add the sgs_z_flux_(...) component here.

• PyCLES prognostic vertical fluxes (defined in ScalarAdvection.pyx, ScalarDiffusion.pyx, MomentumAdvection.pyx, MomentumDiffusion.pyx) are defined at cell centers and have already been multiplied by density. They are computed at cell faces in the low-level functions in scalar_advection.h and scalar_diffusion.h, and then interpolated in the .pyx files before they are written to file. These include all (...)_flux_z and (...)__sgs_flux_z fluxes. In contrast, flux diagnostics from TC.jl are defined at cell faces. This mismatch is handled through is_face_variable. Another difference is that the total_flux_(...) in TC.jl simulations includes the full flux, whereas the PyCLES (...)_flux_z definition only includes the resolved flux. We must add the (...)__sgs_flux_z component here. PyCLES sgs_z_flux and resolved flux fields do not include a contribution from the surface flux, so the bottom cell is set to the diagnosed surface flux.

get_height(filename::String; get_faces::Bool = false)

Returns the vertical cell centers or faces of the given configuration.

Inputs:

• filename :: nc filename.
• get_faces :: If true, returns the coordinates of cell faces. Otherwise, returns the coordinates of cell centers.

Output:

• z: Vertical level coordinates.

normalize_profile(
    y::Array{FT},
    norm_vec::Array{FT},
    prof_dof::IT,
    prof_indices::OptVec{Bool} = nothing,
) where {FT <: Real, IT <: Integer}

Perform normalization of the aggregate observation vector y using separate normalization constants for each variable, contained in norm_vec.

Inputs:

• y :: Aggregate observation vector.
• norm_vec :: Vector of squares of normalization factors.
• prof_dof :: Degrees of freedom of vertical profiles contained in y.
• prof_indices :: Vector of booleans specifying which variables are profiles, and which are timeseries.

Output:

• The normalized aggregate observation vector.

nc_fetch(filename::String, var_names::NTuple{N, Tuple}) where {N}
nc_fetch(filename::String, var_name::String)

Returns the data for a variable var_name (or tuple of strings, varnames), looping through all dataset groups.

is_face_variable(filename::String, var_name::String)

A Bool indicating whether the given variable is defined in a face, or not (cell center).

TurbulenceConvection data are consistent, meaning that variables at cell faces (resp. centers) have as dim zf (resp., zc).

PyCLES variables are inconsistent. All variables have as a dim z, the cell face locations, but most of them (except for the statistical moments of w) are actually defined at cell centers (z_half).

get_stats_path(dir)

Given directory to standard LES or SCM output, fetch path to stats file.

compute_mse(g_arr::Vector{Vector{FT}}, y::Vector{FT})::Vector{FT}
compute_mse(g_mat::Matrix{FT}, y::Vector{FT})::Vector{FT}

Computes the L2-norm error of each vector, column or row of an array with respect to a vector y.

Output:

• The mse for each ensemble member.

penalize_nan(arr::Vector{FT}; penalization::FT = 1.0e5) where {FT <: AbstractFloat}

Substitutes all NaN entries in arr by a penalization factor.

serialize_struct(s::T) where {T}

Serializes the given structure as a dictionary to allow storage in JLD2 format.

deserialize_struct(dict::Dict{String}, ::Type{T})

Deserializes the given dictionary and constructs a struct of the given type with the dictionary values.

jld2_path(root::String, identifier::Union{String, Int}, prefix::String)

Generates a JLD2 path, given a root path, an identifier and a prefix.

write_versions(versions, iteration; outdir_path = pwd())

Writes versions associated with an EnsembleKalmanProcess iteration to a text file.

get_entry(dict, key, default)

Calls get but logs whether the default is used.

change_entry!(dict, keys_and_value)

Changes the entry of a nested dictionary, given a tuple of all its keys and the new value

Inputs:

• dict :: Parent dictionary with an arbitrary number of nested dictionaries.
• keys_and_value :: Tuple of keys from the parent dictionary to the entry to be modified, and the value to use to modify it.

update_namelist!(namelist, namelist_args)

Update namelist with arguments given by namelist_args.

namelist_args is a Vector of Tuples, where each Tuple specifies the keys to traverse the namelist, and the last element of the Tuple is the value to be set.

merge_namelist_args(args, overwrite_args)

Combine two lists of namelist arguments by joining the lists, in order.

This method is intended to be used in the context of case-specific- and global namelistargs. In that case, `args = globalargsandoverwriteargs = caseargs`.

Either argument can be nothing, in which case the other argument is returned; if both are nothing, return nothing.

See also update_namelist!.