Observations

Observation

Structure that contains a (possibly stacked) observation. Defined by sample(s), noise covariance(s), and name(s)

Typical Constructors:

Observation(
    Dict(
        "samples" => [1,2,3],
        "covariances" => I(3),
        "names" => "one_two_three",
    ),
)

or

Observation([1,2,3], I(3), "one_two_three")

One can stack up multiple observations with combine_observations, or by providing vectors of samples, covariances and names to the dictionary.

Fields

• samples::AbstractVector

  A (vector of) observation vectors

• covs::AbstractVector

  A (vector of) observation covariance matrices

• inv_covs::AbstractVector

  A (vector of) inverses of observation covariance matrices

• names::AbstractVector

  A (vector of) name strings

• indices::AbstractVector

  A (vector of) indices of the contained observation blocks

• metadata::Any

  Metadata of any type that the user can group with the Observation

get_samples(o::Observation) -> AbstractVector

gets the samples field from the Observation object

get_covs(o::Observation) -> AbstractVector

gets the covs field from the Observation object

get_inv_covs(o::Observation) -> AbstractVector

gets the inv_covs field from the Observation object

get_names(o::Observation) -> AbstractVector

gets the names field from the Observation object

get_names(os::ObservationSeries) -> AbstractVector

gets the names field from the ObservationSeries object

get_indices(o::Observation) -> AbstractVector

gets the indices field from the Observation object

get_metadata(o::Observation) -> Any

gets the metadata field from the Observation object

combine_observations(obs_vec::AbstractVector) -> Observation{AV1, AV2, AV3, AV4, AV5, Vector{Any}} where {AV1<:(Vector), AV2<:(Vector), AV3<:(Vector), AV4<:(Vector), AV5<:(Vector)}

combines a vector of Observation objects into a single Observation

get_obs(o::Observation) -> Any

if build=true, returns the stacked vector of observed samples samples(default), otherwise it calls get_samples

get_obs_noise_cov(o::Observation) -> Any

if build=true, returns the block matrix of observation covariances covs (default), otherwise it calls get_covs

get_obs_noise_cov_inv(o::Observation) -> Any

if build=true, returns the block matrix of the inverses of the observation covariances inv_covs (default), otherwise it calls get_inv_covs

tsvd_mat(X, r::Int64) -> Any

For a given matrix X and rank r, return the truncated SVD for X as a LinearAlgebra.jl SVD object. Setting return_inverse=true also return it's psuedoinverse X⁺.

tsvd_cov_from_samples(sample_mat::AbstractMatrix) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}

For a given sample_mat, (with data_are_columns = true), rank "r" is optionally provided. Returns the SVDs corresponding to the matrix cov(sample_mat; dims=2). Efficient representation when size(samplemat,1) << size(samplemat,2). Setting return_inverse=true also returns its psuedoinverse.

SVDplusD

Storage for a covariance matrix of the form D + USV' for Diagonal D, and SVD decomposition USV'. Note the inverse of this type (as computed through inv_cov(...)) will be stored compactly as a DminusTall type.

• svd_cov::LinearAlgebra.SVD

  summand of covariance matrix stored with SVD decomposition

• diag_cov::LinearAlgebra.Diagonal

  summand of covariance matrix stored as a diagonal matrix

DminusTall

Storage for a covariance matrix of the form D - RR' for Diagonal D, and (tall) matrix R. Primary use case for this matrix is to compactly store the inverse of the SVDplusD type.

• diag_cov::LinearAlgebra.Diagonal

  summand of covariance matrix stored as a diagonal matrix

• tall_cov::AbstractMatrix

  summand of covariance matrix stored as an abstract matrix

FixedMinibatcher <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

• minibatches::AbstractVector

  explicit indices of the minibatched epoch

• method::AbstractString

  method of selecting minibatches from the list for each epoch ("order" select in order, "random" generate a random selector)

• rng::Random.AbstractRNG

  rng for sampling, if "random" method is selected

no_minibatcher() -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
no_minibatcher(epoch_size::Int64) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)

constructs a FixedMinibatcher of given epoch_size, that generates an epoch of 1:epoch_size and one minibatch that constitutes the whole epoch

create_new_epoch!(m::FixedMinibatcher, args...) -> Any

updates the epoch by either copying ("order") the initialization minibatches, or by randomizing ("random") their order

gets the minibatches field from the FixedMinibatcher object

gets the method field from the FixedMinibatcher object

get_rng(m::FixedMinibatcher) -> Random.AbstractRNG

gets the rng field from the FixedMinibatcher object

RandomFixedSizeMinibatcher <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

• minibatch_size::Int64

  fixed size of minibatches

• method::AbstractString

  how to deal with remainder if minibatch-size doesn't divide the epoch size ("trim" - ignore trailing samples, "extend" - have a larger final minibatch)

• rng::Random.AbstractRNG

  rng for sampling

• minibatches::AbstractVector

  explicit indices of the minibatched epoch

gets the minibatch_size field from the RandomFixesSizeMinibatcher object

gets the method field from the RandomFixesSizeMinibatcher object

get_rng(m::RandomFixedSizeMinibatcher) -> Random.AbstractRNG

gets the rng field from the RandomFixesSizeMinibatcher object

gets the minibatches field from the RandomFixesSizeMinibatcher object

ObservationSeries

Structure that contains multiple Observations along with an optional Minibatcher. Stores all observations in EnsembleKalmanProcess, as well as defining the behavior of the get_obs, get_obs_noise_cov, and get_obs_noise_cov_inv methods

Typical Constructor

ObservationSeries(
    Dict(
        "observations" => vec_of_observations,
        "names" => names_of_observations,
        "minibatcher" => minibatcher,
    ),
)

Fields

• observations::AbstractVector

  A vector of Observations to be used in the experiment

• minibatcher::Minibatcher

  A Minibatcher object used to define the minibatching

• names::AbstractVector

  A vector of string identifiers for the observations

• current_minibatch_index::Dict

  The current index (epoch #, minibatch #) of the current minibatch, stored as a Dict

• minibatches::AbstractVector

  The batch history (grouped by minibatch and epoch)

• metadata::Any

  Metadata of any type that the user can group with the ObservationSeries

gets the observations field from the ObservationSeries object

gets the number of minibatches in an epoch

gets the minibatches field from the ObservationSeries object

get_minibatch_index(os::ObservationSeries, iteration::Int64) -> Dict

returns the minibatch_index Dict("epoch"=> x, "minibatch" => y), for a given iteration

gets the current_minibatch_index field from the ObservationSeries object

gets the minibatcher field from the ObservationSeries object

gets the metadata field from the ObservationSeries object

Within an epoch: iterates the current minibatch index by one. At the end of an epoch: obtains a new epoch of minibatches from the Minibatcher updates the epoch index by one, and minibatch index to one.

get_minibatch(os::ObservationSeries, it_or_mbi::Union{Nothing, Int64, Dict}) -> Any

get the minibatch for a given minibatch index (Dict("epoch"=> x, "minibatch" => y)), or iteration Int. If nothing is provided as an iteration then the current minibatch is returned

get the current minibatch that is pointed to by the current_minibatch_indices field

if build=true then gets the observed sample, stacked over the current minibatch. build=false lists the samples for all observations.

if build=true then gets the observed sample, stacked over the minibatch at iteration. build=false lists the samples for all observations. If isnothing(iteration) or not defined then the current iteration is used.

if build=true then gets the observation covariance matrix, blocked over the current minibatch. build=false lists the covs for all observations

get_obs_noise_cov(os::Union{Nothing, Int64})

if build=true then gets the observation covariance matrix, blocked over the minibatch at iteration. build=false lists the covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.

if build=true then gets the inverse of the observation covariance matrix, blocked over the current minibatch. build=false lists the inv_covs for all observations.

if build=true then gets the inverse of the observation covariance matrix, blocked over minibatch at iteration. build=false lists the inv_covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.