Observations
Observation
EnsembleKalmanProcesses.Observation
— Typestruct Observation{AV1<:(AbstractVector), AV2<:(AbstractVector), AV3<:(AbstractVector), AV4<:(AbstractVector), AV5<:(AbstractVector), MD}
Structure that contains a (possibly stacked) observation. Defined by sample(s), noise covariance(s), and name(s)
Typical Constructors:
Observation(
Dict(
"samples" => [1,2,3],
"covariances" => I(3),
"names" => "one_two_three",
),
)
or
Observation([1,2,3], I(3), "one_two_three")
One can stack up multiple observations with combine_observations
, (recommended), or by providing vectors of samples, covariances and names to the dictionary.
Fields
samples::AbstractVector
: A (vector of) observation vectorscovs::AbstractVector
: A (vector of) observation covariance matricesinv_covs::AbstractVector
: A (vector of) inverses of observation covariance matricesnames::AbstractVector
: A (vector of) name stringsindices::AbstractVector
: A (vector of) indices of the contained observation blocksmetadata::Any
: Metadata of any type that the user can group with the Observation
EnsembleKalmanProcesses.get_samples
— Functionget_samples(o::Observation) -> AbstractVector
gets the samples
field from the Observation
object
EnsembleKalmanProcesses.get_covs
— Functionget_covs(o::Observation) -> AbstractVector
gets the covs
field from the Observation
object
EnsembleKalmanProcesses.get_inv_covs
— Functionget_inv_covs(o::Observation) -> AbstractVector
gets the inv_covs
field from the Observation
object
EnsembleKalmanProcesses.get_names
— Functionget_names(o::Observation) -> AbstractVector
gets the names
field from the Observation
object
get_names(os::ObservationSeries) -> AbstractVector
gets the names
field from the ObservationSeries
object
EnsembleKalmanProcesses.get_indices
— Functionget_indices(o::Observation) -> AbstractVector
gets the indices
field from the Observation
object
EnsembleKalmanProcesses.get_metadata
— Methodget_metadata(o::Observation) -> Any
gets the metadata
field from the Observation
object
EnsembleKalmanProcesses.combine_observations
— Functioncombine_observations(
obs_vec::AbstractVector
) -> Observation{AV1, AV2, AV3, AV4, AV5, Vector{Any}} where {AV1<:(Vector), AV2<:(Vector), AV3<:(Vector), AV4<:(Vector), AV5<:(Vector)}
combines a vector of Observation
objects into a single Observation
EnsembleKalmanProcesses.get_obs
— Methodget_obs(o::Observation; build) -> Any
if build=true
, returns the stacked vector of observed samples samples
(default), otherwise it calls get_samples
EnsembleKalmanProcesses.get_obs_noise_cov
— Methodget_obs_noise_cov(o::Observation; build) -> Any
if build=true
, returns the block matrix of observation covariances covs
(default), otherwise it calls get_covs
EnsembleKalmanProcesses.get_obs_noise_cov_inv
— Methodget_obs_noise_cov_inv(o::Observation; build) -> Any
if build=true
, returns the block matrix of the inverses of the observation covariances inv_covs
(default), otherwise it calls get_inv_covs
Covariance utilities
EnsembleKalmanProcesses.tsvd_mat
— Functiontsvd_mat(
X,
r::Int64;
return_inverse,
quiet,
tsvd_kwargs...
) -> Any
For a given matrix X
and rank r
, return the truncated SVD for X as a LinearAlgebra.jl SVD
object. Setting return_inverse=true
also return it's psuedoinverse X⁺.
EnsembleKalmanProcesses.tsvd_cov_from_samples
— Functiontsvd_cov_from_samples(
sample_mat::AbstractMatrix;
data_are_columns,
return_inverse,
quiet,
tsvd_kwargs...
) -> Union{Tuple{LinearAlgebra.SVD, LinearAlgebra.SVD}, LinearAlgebra.SVD}
For a given sample_mat
, (with data_are_columns = true
), rank "r" is optionally provided. Returns the SVD objects corresponding to the matrix cov(sample_mat; dims=2)
. Efficient representation when size(sample_mat,1) << size(sample_mat,2)
. Setting return_inverse=true
also returns its psuedoinverse.
Example usage to make a low-rank covariance:
# "data"
n_trials = 30
output_dim = 1_000_000
Y = randn(output_dim, n_trials);
# the noise estimated from the samples (will have rank n_trials-1)
internal_cov = tsvd_cov_from_samples(Y)
internal_cov_lower_rank = tsvd_cov_from_samples(Y, n_trials-5)
If one also wishes to add a Diagonal matrix to internal cov
to increase the rank in a compact fashion, use the SVDplusD
object type
diag_cov = 1e-6*Diagonal(1:output_dim)
full_cov = SVDplusD(internal_cov, diag_cov)
Either can be passed in the covariances
entry of an Observation
EnsembleKalmanProcesses.SVDplusD
— Typestruct SVDplusD <: EnsembleKalmanProcesses.SumOfCovariances
Storage for a covariance matrix of the form D + USV'
for Diagonal D, and SVD decomposition USV'. Note the inverse of this type (as computed through inv_cov(...)
) will be stored compactly as a DminusTall
type.
svd_cov::LinearAlgebra.SVD
: summand of covariance matrix stored with SVD decompositiondiag_cov::LinearAlgebra.Diagonal
: summand of covariance matrix stored as a diagonal matrix
EnsembleKalmanProcesses.DminusTall
— Typestruct DminusTall{D<:LinearAlgebra.Diagonal, AM<:(AbstractMatrix)} <: EnsembleKalmanProcesses.SumOfCovariances
Storage for a covariance matrix of the form D - RR'
for Diagonal D, and (tall) matrix R. Primary use case for this matrix is to compactly store the inverse of the SVDplusD
type.
diag_cov::LinearAlgebra.Diagonal
: summand of covariance matrix stored as a diagonal matrixtall_cov::AbstractMatrix
: summand of covariance matrix stored as an abstract matrix
Minibatcher
EnsembleKalmanProcesses.FixedMinibatcher
— Typestruct FixedMinibatcher{AV1<:(AbstractVector), SS<:AbstractString, ARNG<:Random.AbstractRNG} <: Minibatcher
A Minibatcher
that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.
Fields
minibatches::AbstractVector
: explicit indices of the minibatched epochmethod::AbstractString
: method of selecting minibatches from the list for each epoch ("order" select in order, "random" generate a random selector)rng::Random.AbstractRNG
: rng for sampling, if "random" method is selected
Example epochs
given_batches = [[1,2,3], [4,5,6], [7,8,9]]
mb = FixedMinibatcher(given_batches)
# create_new_epoch(mb) = [[1,2,3],[4,5,6],[7,8,9]]
mb2 = FixedMinibatcher(given_batches, "random")
# create_new_epoch(mb2) = [[4,5,6],[1,2,3],[7,8,9]]
EnsembleKalmanProcesses.no_minibatcher
— Functionno_minibatcher(
) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
no_minibatcher(
epoch_size::Int64
) -> FixedMinibatcher{Vector{T}, String, Random.TaskLocalRNG} where T<:(AbstractVector)
constructs a FixedMinibatcher
of given epoch_size
, that generates an epoch of 1:epoch_size
and one minibatch that constitutes the whole epoch
EnsembleKalmanProcesses.create_new_epoch!
— Methodcreate_new_epoch!(
m::FixedMinibatcher,
args...;
kwargs...
) -> Any
updates the epoch by either copying ("order") the initialization minibatches, or by randomizing ("random") their order
EnsembleKalmanProcesses.get_minibatches
— Methodget_minibatches(m::FixedMinibatcher) -> AbstractVector
gets the minibatches
field from the FixedMinibatcher
object
EnsembleKalmanProcesses.get_method
— Methodget_method(m::FixedMinibatcher) -> AbstractString
gets the method
field from the FixedMinibatcher
object
EnsembleKalmanProcesses.get_rng
— Methodget_rng(m::FixedMinibatcher) -> Random.AbstractRNG
gets the rng
field from the FixedMinibatcher
object
EnsembleKalmanProcesses.RandomFixedSizeMinibatcher
— Typestruct RandomFixedSizeMinibatcher{SS<:AbstractString, ARNG<:Random.AbstractRNG, AV2<:(AbstractVector)} <: Minibatcher
A Minibatcher
that takes in a given epoch of batches. It creates a new epoch by either copying-in-order, or by shuffling, the provided batches.
Fields
minibatch_size::Int64
: fixed size of minibatchesmethod::AbstractString
: how to deal with remainder if minibatch-size doesn't divide the epoch size ("trim" - ignore trailing samples, "extend" - have a larger final minibatch)rng::Random.AbstractRNG
: rng for samplingminibatches::AbstractVector
: explicit indices of the minibatched epoch
Example epochs
for data = 1:10
batch_size = 3
mb = RandomFixedSizeMinibatcher(batch_size)
# create_new_epoch(mb) = [[6,7,5],[4,3,10],[9,2,8]] # 1 is trimmed
mb2 = RandomFixedSizeMinibatcher(batch_size, "extend")
# create_new_epoch(mb2) = [[2,9,1],[3,4,7],[10,5,1,6]] # last batch larger
EnsembleKalmanProcesses.get_minibatch_size
— Methodget_minibatch_size(m::RandomFixedSizeMinibatcher) -> Int64
gets the minibatch_size
field from the RandomFixesSizeMinibatcher
object
EnsembleKalmanProcesses.get_method
— Methodget_method(m::RandomFixedSizeMinibatcher) -> AbstractString
gets the method
field from the RandomFixesSizeMinibatcher
object
EnsembleKalmanProcesses.get_rng
— Methodget_rng(m::RandomFixedSizeMinibatcher) -> Random.AbstractRNG
gets the rng
field from the RandomFixesSizeMinibatcher
object
EnsembleKalmanProcesses.get_minibatches
— Methodget_minibatches(
m::RandomFixedSizeMinibatcher
) -> AbstractVector
gets the minibatches
field from the RandomFixesSizeMinibatcher
object
ObservationSeries
EnsembleKalmanProcesses.ObservationSeries
— Typestruct ObservationSeries{AV1<:(AbstractVector), MM<:Minibatcher, AV2<:(AbstractVector), AV3<:(AbstractVector), MD}
Structure that contains multiple Observation
s along with an optional Minibatcher
. Stores all observations in EnsembleKalmanProcess
, as well as defining the behavior of the get_obs
, get_obs_noise_cov
, and get_obs_noise_cov_inv
methods
Typical Constructor
ObservationSeries(
Dict(
"observations" => vec_of_observations,
"names" => names_of_observations,
"minibatcher" => minibatcher,
),
)
Fields
observations::AbstractVector
: A vector ofObservation
s to be used in the experimentminibatcher::Minibatcher
: AMinibatcher
object used to define the minibatchingnames::AbstractVector
: A vector of string identifiers for the observationscurrent_minibatch_index::Dict
: The current index (epoch #, minibatch #) of the current minibatch, stored as a Dictminibatches::AbstractVector
: The batch history (grouped by minibatch and epoch)metadata::Any
: Metadata of any type that the user can group with the ObservationSeries
EnsembleKalmanProcesses.get_observations
— Methodget_observations(os::ObservationSeries) -> AbstractVector
gets the observations
field from the ObservationSeries
object
EnsembleKalmanProcesses.get_length_epoch
— Methodget_length_epoch(os::ObservationSeries) -> Any
gets the number of minibatches in an epoch
EnsembleKalmanProcesses.get_minibatches
— Methodget_minibatches(os::ObservationSeries) -> AbstractVector
gets the minibatches
field from the ObservationSeries
object
EnsembleKalmanProcesses.get_minibatch_index
— Functionget_minibatch_index(
os::ObservationSeries,
iteration::Int64
) -> Dict
returns the minibatch_index Dict("epoch"=> x, "minibatch" => y)
, for a given iteration
EnsembleKalmanProcesses.get_current_minibatch_index
— Methodget_current_minibatch_index(os::ObservationSeries) -> Dict
gets the current_minibatch_index
field from the ObservationSeries
object
EnsembleKalmanProcesses.get_minibatcher
— Methodget_minibatcher(os::ObservationSeries) -> Minibatcher
gets the minibatcher
field from the ObservationSeries
object
EnsembleKalmanProcesses.get_metadata
— Methodget_metadata(os::ObservationSeries) -> Any
gets the metadata
field from the ObservationSeries
object
EnsembleKalmanProcesses.update_minibatch!
— Methodupdate_minibatch!(os::ObservationSeries) -> Any
Within an epoch: iterates the current minibatch index by one. At the end of an epoch: obtains a new epoch of minibatches from the Minibatcher
updates the epoch index by one, and minibatch index to one.
EnsembleKalmanProcesses.get_minibatch
— Functionget_minibatch(
os::ObservationSeries,
it_or_mbi::Union{Nothing, Int64, Dict}
) -> Any
get the minibatch for a given minibatch index (Dict("epoch"=> x, "minibatch" => y)
), or iteration Int
. If nothing
is provided as an iteration then the current minibatch is returned
EnsembleKalmanProcesses.get_current_minibatch
— Methodget_current_minibatch(os::ObservationSeries) -> Any
get the current minibatch that is pointed to by the current_minibatch_indices
field
EnsembleKalmanProcesses.get_obs
— Methodget_obs(os::ObservationSeries; kwargs...) -> Any
if build=true
then gets the observed sample, stacked over the current minibatch. build=false
lists the samples
for all observations.
EnsembleKalmanProcesses.get_obs
— Methodget_obs(os::Union{Nothing, Int64}; kwargs...)
if build=true
then gets the observed sample, stacked over the minibatch at iteration
. build=false
lists the samples
for all observations. If isnothing(iteration)
or not defined then the current iteration is used.
EnsembleKalmanProcesses.get_obs_noise_cov
— Methodget_obs_noise_cov(os::ObservationSeries; kwargs...) -> Any
if build=true
then gets the observation covariance matrix, blocked over the current minibatch. build=false
lists the covs
for all observations
EnsembleKalmanProcesses.get_obs_noise_cov
— Methodget_obs_noise_cov(os::Union{Nothing, Int64}; kwargs...)
if build=true
then gets the observation covariance matrix, blocked over the minibatch at iteration
. build=false
lists the covs
for all observations. If isnothing(iteration)
or not defined then the current iteration is used.
EnsembleKalmanProcesses.get_obs_noise_cov_inv
— Methodget_obs_noise_cov_inv(
os::ObservationSeries;
kwargs...
) -> Any
if build=true
then gets the inverse of the observation covariance matrix, blocked over the current minibatch. build=false
lists the inv_covs
for all observations.
EnsembleKalmanProcesses.get_obs_noise_cov_inv
— Methodget_obs_noise_cov_inv(os::Union{Nothing, Int64}; kwargs...)
if build=true
then gets the inverse of the observation covariance matrix, blocked over minibatch at iteration
. build=false
lists the inv_covs
for all observations. If isnothing(iteration)
or not defined then the current iteration is used.