Pipeline

init_calibration(job_id::String, config::Dict{Any, Any})

Initializes a calibration process given a configuration, and a pipeline mode.

Inputs:

• job_id :: Unique job identifier for sbatch communication.
• config :: User-defined configuration dictionary.
• mode :: Whether the calibration process is parallelized through HPC resources or using Julia's pmap.

restart_calibration(
    ekobj::EnsembleKalmanProcess,
    priors::ParameterDistribution,
    last_iteration::Int64,
    config::Dict{Any, Any},
    outdir_path::String,
)

Restarts a calibration process from an EnsembleKalmanProcess, the parameter priors and the calibration process config file. If batching, it requires access to the last ReferenceModelBatch, stored in the results directory of the previous calibration, outdir_path.

Writes to file the ModelEvaluators necessary to continue the calibration process.

Inputs:

• ekobj :: EnsembleKalmanProcess to be updated.
• priors :: Priors over parameters, used for unconstrained-constrained mappings.
• last_iteration :: Last iteration of the calibration process to be restarted.
• config :: Configuration dictionary.
• outdir_path :: Output path directory of the calibration process to be restarted.
• mode :: Whether the calibration process is parallelized through HPC resources or using Julia's pmap.
• job_id :: Unique job identifier for sbatch communication.

ek_update(
    ekobj::EnsembleKalmanProcess,
    priors::ParameterDistribution,
    iteration::Int64,
    config::Dict{Any, Any},
    versions::Vector{String},
    outdir_path::String,
)

Updates an EnsembleKalmanProcess using forward model evaluations stored in output files defined by their versions, and generates the parameters for the next ensemble for forward model evaluations. The updated EnsembleKalmanProcess and new ModelEvaluators are both written to file.

Inputs:

• ekobj :: EnsembleKalmanProcess to be updated.
• priors :: Priors over parameters, used for unconstrained-constrained mappings.
• iteration :: Current iteration of the calibration process.
• config :: Process configuration dictionary.
• versions :: String versions identifying the forward model evaluations.
• outdir_path :: Output path directory.

update_validation(
    val_config::Dict{Any, Any},
    reg_config::Dict{Any, Any},
    ekp_old::EnsembleKalmanProcess,
    priors::ParameterDistribution,
    param_map::ParameterMap,
    versions::Vector{String},
    outdir_path::String,
    iteration::Integer
)

Updates the validation diagnostics and writes to file the validation ModelEvaluators for the next calibration step.

Inputs:

• val_config :: Validation model configuration.
• reg_config :: Regularization configuration.
• ekp_old :: EnsembleKalmanProcess updated using the past forward model evaluations.
• priors :: The priors over parameter space.
• param_map :: A mapping to a reduced parameter set. See ParameterMap for details.
• versions :: String versions identifying the forward model evaluations.
• outdir_path :: Output path directory.
• iteration :: EKP iteration

update_minibatch_inverse_problem(
    rm_batch::ReferenceModelBatch,
    ekp_old::EnsembleKalmanProcess,
    priors::ParameterDistribution,
    batch_size::Integer,
    outdir_path::String,
    config::Dict{Any, Any},
)

Returns the EnsembleKalmanProcess and ReferenceStatistics consistent with the new ReferenceModel minibatch, and updates the evaluation order of the ReferenceModelBatch.

Inputs:

• rm_batch :: The global ReferenceModelBatch with the current model evaluation order.
• ekp_old :: The EnsembleKalmanProcess from the previous minibatch evaluation.
• batch_size :: The batch size of the current minibatch.
• outdir_path :: The output directory.
• config :: The configuration dictionary.

Outputs:

• ekp :: The EnsembleKalmanProcess for the current minibatch.
• ref_models :: The current minibatch of ReferenceModels.
• ref_stats :: The ReferenceStatistics consistent with the current minibatch.
• refmodelbatch :: The global ReferenceModelBatch with the updated model evaluation order.

write_model_evaluators(
    ekp::EnsembleKalmanProcess,
    priors::ParameterDistribution,
    param_map::ParameterMap,
    ref_models::Vector{ReferenceModel},
    ref_stats::ReferenceStatistics,
    outdir_path::String,
    iteration::Int,
)

Creates and writes to file the ModelEvaluators for the current particle ensemble.

Inputs:

• ekp :: The EnsembleKalmanProcess with the current ensemble of parameter values.
• priors :: The parameter priors.
• param_map :: A mapping to a reduced parameter set. See ParameterMap for details.
• ref_models :: The ReferenceModels defining the new model evaluations.
• ref_stats :: The ReferenceStatistics corresponding to passed ref_models.
• outdir_path :: The output directory.
• iteration :: The current process iteration.

init_diagnostics(
    config::Dict{Any, Any},
    outdir_path::String,
    ref_stats::ReferenceStatistics,
    ekp::EnsembleKalmanProcess,
    priors::ParameterDistribution,
)

Creates a diagnostics netcdf file.

Inputs:

• config :: User-defined configuration dictionary.
• outdir_path :: Path of results directory.
• ref_stats :: ReferenceStatistics.
• ekp :: Initial EnsembleKalmanProcess, containing parameter information, but no forward model evaluations.
• priors:: Prior distributions of the parameters.

update_diagnostics(outdir_path::String, ekp::EnsembleKalmanProcess, priors::ParameterDistribution)

Appends diagnostics of the current iteration evaluations (i.e., forward model output metrics) and the next iteration state (i.e., parameters and parameter metrics) to a diagnostics netcdf file.

Inputs:

• outdir_path :: Path of results directory.
• ekp :: Current EnsembleKalmanProcess.
• priors:: Prior distributions of the parameters.
• ref_stats :: ReferenceStatistics.
• g_full :: The forward model evaluation in primitive space.
• versions :: Version identifiers of the forward model evaluations at the current iteration.
• val_config :: The validation configuration, if given.
• batch_indices :: The indices of the ReferenceModels used in the current batch.