Observations

struct Observation{AV1<:(AbstractVector), AV2<:(AbstractVector), AV3<:(AbstractVector), AV4<:(AbstractVector), AV5<:(AbstractVector), MD}

Structure that contains a (possibly stacked) observation. Defined by sample(s), noise covariance(s), and name(s)

Typical Constructors:

Observation(
    Dict(
        "samples" => [1,2,3],
        "covariances" => I(3),
        "names" => "one_two_three",
    ),
)

or

Observation([1,2,3], I(3), "one_two_three")

One can stack up multiple observations with combine_observations, (recommended), or by providing vectors of samples, covariances and names to the dictionary.

Fields

• samples::AbstractVector: A (vector of) observation vectors

• covs::AbstractVector: A (vector of) observation covariance matrices

• inv_covs::AbstractVector: A (vector of) inverses of observation covariance matrices

• names::AbstractVector: A (vector of) name strings

• indices::AbstractVector: A (vector of) indices of the contained observation blocks

• metadata::Any: Metadata of any type that the user can group with the Observation

get_samples(o::Observation) -> AbstractVector

gets the samples field from the Observation object

get_covs(o::Observation) -> AbstractVector

gets the covs field from the Observation object

get_inv_covs(o::Observation) -> AbstractVector

gets the inv_covs field from the Observation object

get_names(o::Observation) -> AbstractVector

gets the names field from the Observation object

get_names(os::ObservationSeries) -> AbstractVector

gets the names field from the ObservationSeries object

get_indices(o::Observation) -> AbstractVector

gets the indices field from the Observation object

get_metadata(o::Observation) -> Any

gets the metadata field from the Observation object

combine_observations(
    obs_vec::AbstractVector
) -> Observation{AV1, AV2, AV3, AV4, AV5, Vector{Any}} where {AV1<:(Vector), AV2<:(Vector), AV3<:(Vector), AV4<:(Vector), AV5<:(Vector)}

combines a vector of Observation objects into a single Observation

get_obs(o::Observation; build) -> Any

if build=true, returns the stacked vector of observed samples samples(default), otherwise it calls get_samples

get_obs_noise_cov(o::Observation; build) -> Any

if build=true, returns the block matrix of observation covariances covs (default), otherwise it calls get_covs

get_obs_noise_cov_inv(o::Observation; build) -> Any

if build=true, returns the block matrix of the inverses of the observation covariances inv_covs (default), otherwise it calls get_inv_covs

tsvd_mat(
    X,
    r::Int64;
    return_inverse,
    quiet,
    tsvd_kwargs...
) -> Any

For a given matrix X and rank r, return the truncated SVD for X as a LinearAlgebra.jl SVD object. Setting return_inverse=true also return it's psuedoinverse X⁺.

tsvd_cov_from_samples(
    sample_mat::AbstractMatrix;
    data_are_columns,
    return_inverse,
    quiet,
    tsvd_kwargs...
) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}

For a given sample_mat, (with data_are_columns = true), rank "r" is optionally provided. Returns the SVD objects corresponding to the matrix cov(sample_mat; dims=2). Efficient representation when size(sample_mat,1) << size(sample_mat,2). Setting return_inverse=true also returns its psuedoinverse.

Example usage to make a low-rank covariance:

# "data"
n_trials = 30
output_dim = 1_000_000
Y = randn(output_dim, n_trials);

# the noise estimated from the samples (will have rank n_trials-1)
internal_cov = tsvd_cov_from_samples(Y)
internal_cov_lower_rank = tsvd_cov_from_samples(Y, n_trials-5)

If one also wishes to add a Diagonal matrix to internal cov to increase the rank in a compact fashion, use the SVDplusD object type

diag_cov = 1e-6*Diagonal(1:output_dim)
full_cov = SVDplusD(internal_cov, diag_cov)

Either can be passed in the covariances entry of an Observation

struct SVDplusD <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D + USV' for Diagonal D, and SVD decomposition USV'. Note the inverse of this type (as computed through inv_cov(...)) will be stored compactly as a DminusTall type.

• svd_cov::LinearAlgebra.SVD: summand of covariance matrix stored with SVD decomposition

• diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

struct DminusTall{D<:LinearAlgebra.Diagonal, AM<:(AbstractMatrix)} <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D - RR' for Diagonal D, and (tall) matrix R. Primary use case for this matrix is to compactly store the inverse of the SVDplusD type.

• diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

• tall_cov::AbstractMatrix: summand of covariance matrix stored as an abstract matrix

struct FixedMinibatcher{AV1<:(AbstractVector), SS<:AbstractString, ARNG<:Random.AbstractRNG} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

• minibatches::AbstractVector: explicit indices of the minibatched epoch

• method::AbstractString: method of selecting minibatches from the list for each epoch ("order" select in order, "random" generate a random selector)

• rng::Random.AbstractRNG: rng for sampling, if "random" method is selected

Example epochs

given_batches = [[1,2,3], [4,5,6], [7,8,9]]
mb = FixedMinibatcher(given_batches)
# create_new_epoch(mb) = [[1,2,3],[4,5,6],[7,8,9]]

mb2 = FixedMinibatcher(given_batches, "random")
# create_new_epoch(mb2) = [[4,5,6],[1,2,3],[7,8,9]]

no_minibatcher(

) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
no_minibatcher(
    epoch_size::Int64
) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)

constructs a FixedMinibatcher of given epoch_size, that generates an epoch of 1:epoch_size and one minibatch that constitutes the whole epoch

create_new_epoch!(
    m::FixedMinibatcher,
    args...;
    kwargs...
) -> Any

updates the epoch by either copying ("order") the initialization minibatches, or by randomizing ("random") their order

get_minibatches(m::FixedMinibatcher) -> AbstractVector

gets the minibatches field from the FixedMinibatcher object

get_method(m::FixedMinibatcher) -> AbstractString

gets the method field from the FixedMinibatcher object

get_rng(m::FixedMinibatcher) -> Random.AbstractRNG

gets the rng field from the FixedMinibatcher object

struct RandomFixedSizeMinibatcher{SS<:AbstractString, ARNG<:Random.AbstractRNG, AV2<:(AbstractVector)} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

• minibatch_size::Int64: fixed size of minibatches

• method::AbstractString: how to deal with remainder if minibatch-size doesn't divide the epoch size ("trim" - ignore trailing samples, "extend" - have a larger final minibatch)

• rng::Random.AbstractRNG: rng for sampling

• minibatches::AbstractVector: explicit indices of the minibatched epoch

Example epochs

for data = 1:10
batch_size = 3
mb = RandomFixedSizeMinibatcher(batch_size)
# create_new_epoch(mb) = [[6,7,5],[4,3,10],[9,2,8]] #  1 is trimmed

mb2 = RandomFixedSizeMinibatcher(batch_size, "extend")
# create_new_epoch(mb2) = [[2,9,1],[3,4,7],[10,5,1,6]] # last batch larger

get_minibatch_size(m::RandomFixedSizeMinibatcher) -> Int64

gets the minibatch_size field from the RandomFixesSizeMinibatcher object

get_method(m::RandomFixedSizeMinibatcher) -> AbstractString

gets the method field from the RandomFixesSizeMinibatcher object

get_rng(m::RandomFixedSizeMinibatcher) -> Random.AbstractRNG

gets the rng field from the RandomFixesSizeMinibatcher object

get_minibatches(
    m::RandomFixedSizeMinibatcher
) -> AbstractVector

gets the minibatches field from the RandomFixesSizeMinibatcher object

struct ObservationSeries{AV1<:(AbstractVector), MM<:Minibatcher, AV2<:(AbstractVector), AV3<:(AbstractVector), MD}

Structure that contains multiple Observations along with an optional Minibatcher. Stores all observations in EnsembleKalmanProcess, as well as defining the behavior of the get_obs, get_obs_noise_cov, and get_obs_noise_cov_inv methods

Typical Constructor

ObservationSeries(
    Dict(
        "observations" => vec_of_observations,
        "names" => names_of_observations,
        "minibatcher" => minibatcher,
    ),
)

Fields

• observations::AbstractVector: A vector of Observations to be used in the experiment

• minibatcher::Minibatcher: A Minibatcher object used to define the minibatching

• names::AbstractVector: A vector of string identifiers for the observations

• current_minibatch_index::Dict: The current index (epoch #, minibatch #) of the current minibatch, stored as a Dict

• minibatches::AbstractVector: The batch history (grouped by minibatch and epoch)

• metadata::Any: Metadata of any type that the user can group with the ObservationSeries

get_observations(os::ObservationSeries) -> AbstractVector

gets the observations field from the ObservationSeries object

get_length_epoch(os::ObservationSeries) -> Any

gets the number of minibatches in an epoch

get_minibatches(os::ObservationSeries) -> AbstractVector

gets the minibatches field from the ObservationSeries object

get_minibatch_index(
    os::ObservationSeries,
    iteration::Int64
) -> Dict

returns the minibatch_index Dict("epoch"=> x, "minibatch" => y), for a given iteration

get_current_minibatch_index(os::ObservationSeries) -> Dict

gets the current_minibatch_index field from the ObservationSeries object

get_minibatcher(os::ObservationSeries) -> Minibatcher

gets the minibatcher field from the ObservationSeries object

get_metadata(os::ObservationSeries) -> Any

gets the metadata field from the ObservationSeries object

update_minibatch!(os::ObservationSeries) -> Any

Within an epoch: iterates the current minibatch index by one. At the end of an epoch: obtains a new epoch of minibatches from the Minibatcher updates the epoch index by one, and minibatch index to one.

get_minibatch(
    os::ObservationSeries,
    it_or_mbi::Union{Nothing, Int64, Dict}
) -> Any

get the minibatch for a given minibatch index (Dict("epoch"=> x, "minibatch" => y)), or iteration Int. If nothing is provided as an iteration then the current minibatch is returned

get_current_minibatch(os::ObservationSeries) -> Any

get the current minibatch that is pointed to by the current_minibatch_indices field

get_obs(os::ObservationSeries; kwargs...) -> Any

if build=true then gets the observed sample, stacked over the current minibatch. build=false lists the samples for all observations.

get_obs(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the observed sample, stacked over the minibatch at iteration. build=false lists the samples for all observations. If isnothing(iteration) or not defined then the current iteration is used.

get_obs_noise_cov(os::ObservationSeries; kwargs...) -> Any

if build=true then gets the observation covariance matrix, blocked over the current minibatch. build=false lists the covs for all observations

get_obs_noise_cov(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the observation covariance matrix, blocked over the minibatch at iteration. build=false lists the covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.

get_obs_noise_cov_inv(
    os::ObservationSeries;
    kwargs...
) -> Any

if build=true then gets the inverse of the observation covariance matrix, blocked over the current minibatch. build=false lists the inv_covs for all observations.

get_obs_noise_cov_inv(os::Union{Nothing, Int64}; kwargs...)

if build=true then gets the inverse of the observation covariance matrix, blocked over minibatch at iteration. build=false lists the inv_covs for all observations. If isnothing(iteration) or not defined then the current iteration is used.