Observations

Observation

EnsembleKalmanProcesses.ObservationType
struct Observation{AV1<:(AbstractVector), AV2<:(AbstractVector), AV3<:(AbstractVector), AV4<:(AbstractVector), AV5<:(AbstractVector), MD}

Structure that contains a (possibly stacked) observation. Defined by sample(s), noise covariance(s), and name(s)

Typical Constructors:

Observation(
    Dict(
        "samples" => [1,2,3],
        "covariances" => I(3),
        "names" => "one_two_three",
    ),
)

or

Observation([1,2,3], I(3), "one_two_three")

One can stack up multiple observations with combine_observations, (recommended), or by providing vectors of samples, covariances and names to the dictionary.

Fields

  • samples::AbstractVector: A (vector of) observation vectors

  • covs::AbstractVector: A (vector of) observation covariance matrices

  • inv_covs::AbstractVector: A (vector of) inverses of observation covariance matrices

  • names::AbstractVector: A (vector of) name strings

  • indices::AbstractVector: A (vector of) indices of the contained observation blocks

  • metadata::Any: Metadata of any type that the user can group with the Observation

source
EnsembleKalmanProcesses.get_namesFunction
get_names(o::Observation) -> AbstractVector

gets the names field from the Observation object

source
get_names(os::ObservationSeries) -> AbstractVector

gets the names field from the ObservationSeries object

source
EnsembleKalmanProcesses.combine_observationsFunction
combine_observations(
    obs_vec::AbstractVector
) -> Observation{AV1, AV2, AV3, AV4, AV5, Vector{Any}} where {AV1<:(Vector), AV2<:(Vector), AV3<:(Vector), AV4<:(Vector), AV5<:(Vector)}

combines a vector of Observation objects into a single Observation

source
EnsembleKalmanProcesses.get_obsMethod
get_obs(o::Observation; build) -> Any

if build=true, returns the stacked vector of observed samples samples(default), otherwise it calls get_samples

source

Covariance utilities

EnsembleKalmanProcesses.tsvd_matFunction
tsvd_mat(
    X,
    r::Int64;
    return_inverse,
    quiet,
    tsvd_kwargs...
) -> Any

For a given matrix X and rank r, return the truncated SVD for X as a LinearAlgebra.jl SVD object. Setting return_inverse=true also return it's psuedoinverse X⁺.

source
EnsembleKalmanProcesses.tsvd_cov_from_samplesFunction
tsvd_cov_from_samples(
    sample_mat::AbstractMatrix;
    data_are_columns,
    return_inverse,
    quiet,
    tsvd_kwargs...
) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}

For a given sample_mat, (with data_are_columns = true), rank "r" is optionally provided. Returns the SVD objects corresponding to the matrix cov(sample_mat; dims=2). Efficient representation when size(sample_mat,1) << size(sample_mat,2). Setting return_inverse=true also returns its psuedoinverse.

Example usage to make a low-rank covariance:

# "data"
n_trials = 30
output_dim = 1_000_000
Y = randn(output_dim, n_trials);

# the noise estimated from the samples (will have rank n_trials-1)
internal_cov = tsvd_cov_from_samples(Y)
internal_cov_lower_rank = tsvd_cov_from_samples(Y, n_trials-5)

If one also wishes to add a Diagonal matrix to internal cov to increase the rank in a compact fashion, use the SVDplusD object type

diag_cov = 1e-6*Diagonal(1:output_dim)
full_cov = SVDplusD(internal_cov, diag_cov)

Either can be passed in the covariances entry of an Observation

source
EnsembleKalmanProcesses.SVDplusDType
struct SVDplusD <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D + USV' for Diagonal D, and SVD decomposition USV'. Note the inverse of this type (as computed through inv_cov(...)) will be stored compactly as a DminusTall type.

  • svd_cov::LinearAlgebra.SVD: summand of covariance matrix stored with SVD decomposition

  • diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

source
EnsembleKalmanProcesses.DminusTallType
struct DminusTall{D<:LinearAlgebra.Diagonal, AM<:(AbstractMatrix)} <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D - RR' for Diagonal D, and (tall) matrix R. Primary use case for this matrix is to compactly store the inverse of the SVDplusD type.

  • diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

  • tall_cov::AbstractMatrix: summand of covariance matrix stored as an abstract matrix

source

Minibatcher

EnsembleKalmanProcesses.FixedMinibatcherType
struct FixedMinibatcher{AV1<:(AbstractVector), SS<:AbstractString, ARNG<:Random.AbstractRNG} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

  • minibatches::AbstractVector: explicit indices of the minibatched epoch

  • method::AbstractString: method of selecting minibatches from the list for each epoch ("order" select in order, "random" generate a random selector)

  • rng::Random.AbstractRNG: rng for sampling, if "random" method is selected

Example epochs

given_batches = [[1,2,3], [4,5,6], [7,8,9]]
mb = FixedMinibatcher(given_batches)
# create_new_epoch(mb) = [[1,2,3],[4,5,6],[7,8,9]]

mb2 = FixedMinibatcher(given_batches, "random")
# create_new_epoch(mb2) = [[4,5,6],[1,2,3],[7,8,9]]
source
EnsembleKalmanProcesses.no_minibatcherFunction
no_minibatcher(

) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
no_minibatcher(
    epoch_size::Int64
) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)

constructs a FixedMinibatcher of given epoch_size, that generates an epoch of 1:epoch_size and one minibatch that constitutes the whole epoch

source
EnsembleKalmanProcesses.create_new_epoch!Method
create_new_epoch!(
    m::FixedMinibatcher,
    args...;
    kwargs...
) -> Any

updates the epoch by either copying ("order") the initialization minibatches, or by randomizing ("random") their order

source
EnsembleKalmanProcesses.RandomFixedSizeMinibatcherType
struct RandomFixedSizeMinibatcher{SS<:AbstractString, ARNG<:Random.AbstractRNG, AV2<:(AbstractVector)} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

  • minibatch_size::Int64: fixed size of minibatches

  • method::AbstractString: how to deal with remainder if minibatch-size doesn't divide the epoch size ("trim" - ignore trailing samples, "extend" - have a larger final minibatch)

  • rng::Random.AbstractRNG: rng for sampling

  • minibatches::AbstractVector: explicit indices of the minibatched epoch

Example epochs

for data = 1:10
batch_size = 3
mb = RandomFixedSizeMinibatcher(batch_size)
# create_new_epoch(mb) = [[6,7,5],[4,3,10],[9,2,8]] #  1 is trimmed

mb2 = RandomFixedSizeMinibatcher(batch_size, "extend")
# create_new_epoch(mb2) = [[2,9,1],[3,4,7],[10,5,1,6]] # last batch larger
source

ObservationSeries

EnsembleKalmanProcesses.ObservationSeriesType
struct ObservationSeries{AV1<:(AbstractVector), MM<:Minibatcher, AV2<:(AbstractVector), AV3<:(AbstractVector), MD}

Structure that contains multiple Observations along with an optional Minibatcher. Stores all observations in EnsembleKalmanProcess, as well as defining the behavior of the get_obs, get_obs_noise_cov, and get_obs_noise_cov_inv methods

Typical Constructor

ObservationSeries(
    Dict(
        "observations" => vec_of_observations,
        "names" => names_of_observations,
        "minibatcher" => minibatcher,
    ),
)

Fields

  • observations::AbstractVector: A vector of Observations to be used in the experiment

  • minibatcher::Minibatcher: A Minibatcher object used to define the minibatching

  • names::AbstractVector: A vector of string identifiers for the observations

  • current_minibatch_index::Dict: The current index (epoch #, minibatch #) of the current minibatch, stored as a Dict

  • minibatches::AbstractVector: The batch history (grouped by minibatch and epoch)

  • metadata::Any: Metadata of any type that the user can group with the ObservationSeries

source
EnsembleKalmanProcesses.update_minibatch!Method
update_minibatch!(os::ObservationSeries) -> Any

Within an epoch: iterates the current minibatch index by one. At the end of an epoch: obtains a new epoch of minibatches from the Minibatcher updates the epoch index by one, and minibatch index to one.

source
EnsembleKalmanProcesses.get_minibatchFunction
get_minibatch(
    os::ObservationSeries,
    it_or_mbi::Union{Nothing, Int64, Dict}
) -> Any

get the minibatch for a given minibatch index (Dict("epoch"=> x, "minibatch" => y)), or iteration Int. If nothing is provided as an iteration then the current minibatch is returned

source
EnsembleKalmanProcesses.get_obsMethod
get_obs(os::ObservationSeries; kwargs...) -> Any

if build=true then gets the observed sample, stacked over the current minibatch. build=false lists the samples for all observations.

source
EnsembleKalmanProcesses.get_obsMethod
get_obs(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the observed sample, stacked over the minibatch at iteration. build=false lists the samples for all observations. If isnothing(iteration) or not defined then the current iteration is used.

source
EnsembleKalmanProcesses.get_obs_noise_covMethod
get_obs_noise_cov(os::ObservationSeries; kwargs...) -> Any

if build=true then gets the observation covariance matrix, blocked over the current minibatch. build=false lists the covs for all observations

source
EnsembleKalmanProcesses.get_obs_noise_covMethod
get_obs_noise_cov(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the observation covariance matrix, blocked over the minibatch at iteration. build=false lists the covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.

source
EnsembleKalmanProcesses.get_obs_noise_cov_invMethod
get_obs_noise_cov_inv(
    os::ObservationSeries;
    kwargs...
) -> Any

if build=true then gets the inverse of the observation covariance matrix, blocked over the current minibatch. build=false lists the inv_covs for all observations.

source
EnsembleKalmanProcesses.get_obs_noise_cov_invMethod
get_obs_noise_cov_inv(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the inverse of the observation covariance matrix, blocked over minibatch at iteration. build=false lists the inv_covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.

source