GitHub

Observations

Observation

EnsembleKalmanProcesses.Observation — Type
struct Observation{AV1<:(AbstractVector), AV2<:(AbstractVector), AV3<:(AbstractVector), AV4<:(AbstractVector), AV5<:(AbstractVector), MD}

Structure that contains a (possibly stacked) observation. Defined by sample(s), noise covariance(s), and name(s)

Typical Constructors:

Observation(
    Dict(
        "samples" => [1,2,3],
        "covariances" => I(3),
        "names" => "one_two_three",
    ),
)

or

Observation([1,2,3], I(3), "one_two_three")

One can stack up multiple observations with combine_observations, (recommended), or by providing vectors of samples, covariances and names to the dictionary.

Fields

  • samples::AbstractVector: A (vector of) observation vectors

  • covs::AbstractVector: A (vector of) observation covariance matrices

  • inv_covs::AbstractVector: A (vector of) inverses of observation covariance matrices

  • names::AbstractVector: A (vector of) name strings

  • indices::AbstractVector: A (vector of) indices of the contained observation blocks

  • metadata::Any: Metadata of any type that the user can group with the Observation

source
EnsembleKalmanProcesses.get_names — Function
get_names(o::Observation) -> AbstractVector

Return the names field of the Observation.

source
get_names(os::ObservationSeries) -> AbstractVector

Return the names field of the ObservationSeries.

source
EnsembleKalmanProcesses.combine_observations — Function
combine_observations(
    obs_vec::AbstractVector
) -> Observation{AV1, AV2, AV3, AV4, AV5, Vector{Any}} where {AV1<:(Vector), AV2<:(Vector), AV3<:(Vector), AV4<:(Vector), AV5<:(Vector)}

combines a vector of Observation objects into a single Observation

source
EnsembleKalmanProcesses.get_obs — Method
get_obs(o::Observation; build) -> Any

Return the stacked vector of observed samples samples if build = true (default); otherwise return the output of get_samples.

source
EnsembleKalmanProcesses.get_obs_noise_cov_inv — Method
get_obs_noise_cov_inv(o::Observation; build) -> Any

Return the block matrix of the inverses of the observation covariances inv_covs if build = true (default); otherwise return the output of get_inv_covs.

source

Covariance utilities

EnsembleKalmanProcesses.tsvd_mat — Function
tsvd_mat(
    X,
    r::Int64;
    return_inverse,
    quiet,
    kwargs...
) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}

For a given matrix X and rank r, return the truncated SVD for X as a LinearAlgebra.jl SVD object. Setting return_inverse=true also return it's psuedoinverse X⁺.

source
EnsembleKalmanProcesses.tsvd_cov_from_samples — Function
tsvd_cov_from_samples(
    sample_mat::AbstractMatrix;
    data_are_columns,
    return_inverse,
    quiet,
    kwargs...
) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}

For a given sample_mat, (with data_are_columns = true), rank "r" is optionally provided. Returns the SVD objects corresponding to the matrix cov(sample_mat; dims=2). Efficient representation when size(sample_mat,1) << size(sample_mat,2). Setting return_inverse=true also returns its psuedoinverse.

Example usage to make a low-rank covariance:

# "data"
n_trials = 30
output_dim = 1_000_000
Y = randn(output_dim, n_trials);

# the noise estimated from the samples (will have rank n_trials-1)
internal_cov = tsvd_cov_from_samples(Y)
internal_cov_lower_rank = tsvd_cov_from_samples(Y, n_trials-5)

If one also wishes to add a Diagonal matrix to internal cov to increase the rank in a compact fashion, use the SVDplusD object type

diag_cov = 1e-6*Diagonal(1:output_dim)
full_cov = SVDplusD(internal_cov, diag_cov)

Either can be passed in the covariances entry of an Observation

source
EnsembleKalmanProcesses.SVDplusD — Type
struct SVDplusD <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D + USV' for Diagonal D, and SVD decomposition USV'. Note the inverse of this type (as computed through inv_cov(...)) will be stored compactly as a DminusTall type.

  • svd_cov::LinearAlgebra.SVD: summand of covariance matrix stored with SVD decomposition

  • diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

source
EnsembleKalmanProcesses.DminusTall — Type
struct DminusTall{D<:LinearAlgebra.Diagonal, AM<:(AbstractMatrix)} <: EnsembleKalmanProcesses.SumOfCovariances

Storage for a covariance matrix of the form D - RR' for Diagonal D, and (tall) matrix R. Primary use case for this matrix is to compactly store the inverse of the SVDplusD type.

  • diag_cov::LinearAlgebra.Diagonal: summand of covariance matrix stored as a diagonal matrix

  • tall_cov::AbstractMatrix: summand of covariance matrix stored as an abstract matrix

source
EnsembleKalmanProcesses.inv_cov — Method
inv_cov(a::AbstractMatrix) -> Any

Compute the inverse of the covariance representation a. For structured representations, the inverse is returned in a compact structured form (e.g., the inverse of an SVDplusD is stored as a DminusTall).

source
EnsembleKalmanProcesses.lmul_without_build — Method
lmul_without_build(A, X::AbstractVecOrMat) -> Any

Compute the matrix product A * X, where A is a (block-diagonal collection of) covariance representation(s), without explicitly building the full matrix of A.

source

Minibatcher

EnsembleKalmanProcesses.Minibatcher — Type

Abstract supertype for all minibatching strategies used with ObservationSeries.

Concrete subtypes (FixedMinibatcher, RandomFixedSizeMinibatcher) implement create_new_epoch! to produce a new ordering of observation batches at each epoch.

source
EnsembleKalmanProcesses.FixedMinibatcher — Type
struct FixedMinibatcher{AV1<:(AbstractVector), SS<:AbstractString, ARNG<:Random.AbstractRNG} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

  • minibatches::AbstractVector: explicit indices of the minibatched epoch

  • method::AbstractString: method of selecting minibatches from the list for each epoch ("order" select in order, "random" generate a random selector)

  • rng::Random.AbstractRNG: rng for sampling, if "random" method is selected

Example epochs

given_batches = [[1,2,3], [4,5,6], [7,8,9]]
mb = FixedMinibatcher(given_batches)
# create_new_epoch(mb) = [[1,2,3],[4,5,6],[7,8,9]]

mb2 = FixedMinibatcher(given_batches, "random")
# create_new_epoch(mb2) = [[4,5,6],[1,2,3],[7,8,9]]
source
EnsembleKalmanProcesses.no_minibatcher — Function
no_minibatcher(

) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
no_minibatcher(
    epoch_size::Int64
) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)

constructs a FixedMinibatcher of given epoch_size, that generates an epoch of 1:epoch_size and one minibatch that constitutes the whole epoch

source
EnsembleKalmanProcesses.create_new_epoch! — Function
create_new_epoch!(
    m::FixedMinibatcher,
    args...;
    kwargs...
) -> Any

updates the epoch by either copying ("order") the initialization minibatches, or by randomizing ("random") their order

source
create_new_epoch!(
    m::RandomFixedSizeMinibatcher,
    epoch_in::AbstractVector,
    args...;
    kwargs...
) -> Any

updates the epoch by randomizing the provided epoch indices. If the length of minibatches do not divide the length of the epoch, then the remainder is either ignored (default) or a final larger batch is created

source
EnsembleKalmanProcesses.RandomFixedSizeMinibatcher — Type
struct RandomFixedSizeMinibatcher{SS<:AbstractString, ARNG<:Random.AbstractRNG, AV2<:(AbstractVector)} <: Minibatcher

A Minibatcher that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.

Fields

  • minibatch_size::Int64: fixed size of minibatches

  • method::AbstractString: how to deal with remainder if minibatch-size doesn't divide the epoch size ("trim" - ignore trailing samples, "extend" - have a larger final minibatch)

  • rng::Random.AbstractRNG: rng for sampling

  • minibatches::AbstractVector: explicit indices of the minibatched epoch

Example epochs

for data = 1:10
batch_size = 3
mb = RandomFixedSizeMinibatcher(batch_size)
# create_new_epoch(mb) = [[6,7,5],[4,3,10],[9,2,8]] #  1 is trimmed

mb2 = RandomFixedSizeMinibatcher(batch_size, "extend")
# create_new_epoch(mb2) = [[2,9,1],[3,4,7],[10,5,1,6]] # last batch larger
source

ObservationSeries

EnsembleKalmanProcesses.ObservationSeries — Type
struct ObservationSeries{AV1<:(AbstractVector), MM<:Minibatcher, AV2<:(AbstractVector), AV3<:(AbstractVector), MD}

Structure that contains multiple Observations along with an optional Minibatcher. Stores all observations in EnsembleKalmanProcess, as well as defining the behavior of the get_obs, get_obs_noise_cov, and get_obs_noise_cov_inv methods

Typical Constructor

ObservationSeries(
    Dict(
        "observations" => vec_of_observations,
        "names" => names_of_observations,
        "minibatcher" => minibatcher,
    ),
)

Fields

  • observations::AbstractVector: A vector of Observations to be used in the experiment

  • minibatcher::Minibatcher: A Minibatcher object used to define the minibatching

  • names::AbstractVector: A vector of string identifiers for the observations

  • current_minibatch_index::Dict: The current index (epoch #, minibatch #) of the current minibatch, stored as a Dict

  • minibatches::AbstractVector: The batch history (grouped by minibatch and epoch)

  • metadata::Any: Metadata of any type that the user can group with the ObservationSeries

source
EnsembleKalmanProcesses.update_minibatch! — Method
update_minibatch!(os::ObservationSeries) -> Any

Within an epoch: iterates the current minibatch index by one. At the end of an epoch: obtains a new epoch of minibatches from the Minibatcher updates the epoch index by one, and minibatch index to one.

source
EnsembleKalmanProcesses.get_minibatch — Function
get_minibatch(
    os::ObservationSeries,
    it_or_mbi::Union{Nothing, Int64, Dict}
) -> Any

Return the minibatch for a given minibatch index (Dict("epoch" => x, "minibatch" => y)) or iteration Int. If nothing is provided as an iteration, the current minibatch is returned.

source
EnsembleKalmanProcesses.get_obs — Method
get_obs(os::ObservationSeries; kwargs...) -> Any

Return the observed sample, stacked over the current minibatch, if build = true; if build = false, list the samples for all observations.

source
EnsembleKalmanProcesses.get_obs — Method
get_obs(os::Union{Nothing, Int64}; kwargs...)

Return the observed sample, stacked over the minibatch at iteration, if build = true; if build = false, list the samples for all observations. If isnothing(iteration) or not defined, the current iteration is used.

source
EnsembleKalmanProcesses.get_obs_noise_cov — Method
get_obs_noise_cov(os::ObservationSeries; kwargs...) -> Any

Return the observation covariance matrix, blocked over the current minibatch, if build = true; if build = false, list the covs for all observations.

source
EnsembleKalmanProcesses.get_obs_noise_cov — Method
get_obs_noise_cov(os::Union{Nothing, Int64}; kwargs...)

Return the observation covariance matrix, blocked over the minibatch at iteration, if build = true; if build = false, list the covs for all observations. If isnothing(iteration) or not defined, the current iteration is used.

source
EnsembleKalmanProcesses.get_obs_noise_cov_inv — Method
get_obs_noise_cov_inv(
    os::ObservationSeries;
    kwargs...
) -> Any

Return the inverse of the observation covariance matrix, blocked over the current minibatch, if build = true; if build = false, list the inv_covs for all observations.

source
EnsembleKalmanProcesses.get_obs_noise_cov_inv — Method
get_obs_noise_cov_inv(os::Union{Nothing, Int64}; kwargs...)

Return the inverse of the observation covariance matrix, blocked over the minibatch at iteration, if build = true; if build = false, list the inv_covs for all observations. If isnothing(iteration) or not defined, the current iteration is used.

source