Observations

The Observations object facilitates convenient storing, grouping and minibatching over observations.

The key objects

The Observation is a container for an observed variables ("samples"), their noise covariances ("covariances"), and names ("names"). They are easily stackable to help build larger heterogenous observations
The Minibatcher facilitate data streaming (minibatching), where a user can submit large group of observations, that are then batched up and looped over in epochs.
The ObservationSeries contains the list of Observations and Minibatcher and the utilities to get the current batch etc.

I usually just pass in a vector of data and a covariance to EKP

Users can indeed set up an experiment with just one data sample and covariance matrix for the noise. However internally these are still stored as an ObservationSeries with a special minibatcher that does nothing (created by no_minibatcher(size)).

Recommended constructor: A single (stacked) observation

Here the user has data for two independent variables: the five-dimensional y and the eight-dimensional z. The observational noise of y is uncorrelated in all components, while the observations of z there is a known correlation.

We recommend users build an Observation using the Dict constructor and make use of the combine_observations() utility.

using EnsembleKalmanProcesses # for `Observation`
using LinearAlgebra # for `I`, `Tridiagonal`


# specify an observation of y with diagonal noise covariance
ydim = 5
y = ones(ydim)
cov_y = 0.01*I

# specify an observation of z with tridiagonal noise covariance
zdim = 8
z = zeros(zdim)
cov_z = Tridiagonal(0.1*ones(zdim-1), ones(zdim), 0.1*ones(zdim-1))

y_obs = Observation(
    Dict(
        "samples" => y,
        "covariances" => cov_y,
        "names" => "y",
    ),
)

z_obs = Observation(
    Dict(
        "samples" => z,
        "covariances" => cov_z,
        "names" => "z",
    ),
)

full_obs = combine_observations([y_obs,z_obs]) # conveniently stacks the observations

Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.0, 1.0, 1.0, 1.0, 1.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y", "z"], UnitRange{Int64}[1:5, 6:13])

# getting information out
get_obs(full_obs) # returns [y,z]

13-element Vector{Float64}:
 1.0
 1.0
 1.0
 1.0
 1.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

get_obs_noise_cov(full_obs) # returns block-diagonal matrix with blocks [cov_y 0; 0 cov_z]

13×13 Matrix{Float64}:
 0.01  0.0   0.0   0.0   0.0   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.01  0.0   0.0   0.0   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.01  0.0   0.0   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.01  0.0   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   1.0  0.1  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.1  1.0  0.1  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.1  1.0  0.1  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  0.1  1.0  0.1  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  0.0  0.1  1.0  0.1  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  0.0  0.0  0.1  1.0  0.1  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  0.0  0.0  0.0  0.1  1.0  0.1
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  0.0  0.0  0.0  0.0  0.1  1.0

getters get_* can be used for all internals,

get_names(full_obs) # returns ["y", "z"]

2-element Vector{String}:
 "y"
 "z"

There are some other fields stored such as indices of the y and z components

get_indices(full_obs) # returns [1:ydim, ydim+1:ydim+zdim]

2-element Vector{UnitRange{Int64}}:
 1:5
 6:13

Recommended constructor: Many stacked observations

Imagine the user has 100 independent data samples for two independent variables above Rather than stacking all the data together at once (forming a full system of size 100*(8+5) to update at each step) instead the user wishes to stream the data and do updates with random batches of 5 observations at each iteration.

Why would I choose to minibatch?

The memory- and time-scaling of many EKP methods is worse-than-linear in the observation dimension, therefore there is often large computational benefit to minibatch EKP updates. Such costs must be weighed against the cost of additional forward map evaluations needed to minibatching over one full epoch.

# given a vector of 100 `Observation`s called hundred_full_obs,
using EnsembleKalmanProcesses # for `RandomFixedSizeMinibatcher`, `ObservationSeries`, `Minibatcher`

minibatcher = RandomFixedSizeMinibatcher(5) # batches the epoch of size 100, into batches of size 5

observation_series = ObservationSeries(
    Dict(
        "observations" => hundred_full_obs,
        "minibatcher" => minibatcher,
    ),
)

ObservationSeries{Vector{Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}}, RandomFixedSizeMinibatcher{String, Random.TaskLocalRNG, Vector{Vector{Int64}}}, Vector{String}, Vector{Vector{Vector{Int64}}}}(Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}[Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.924269441739825, 0.9994748351951309, 1.1646288476221354, 0.9623871414662105, 1.1472790606911694], [0.9050594516176166, 1.4807332364896455, -0.4531404088610038, 0.47460094859207574, -0.17599119306800343, 1.92129571532249, 0.2724893459740905, -1.5295502474705507]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_1", "z_1"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.017965336971198, 0.9727810808921973, 0.881179853690762, 1.1189937245667574, 0.9942812784234154], [-2.0832845006374776, 0.9469403762262455, -1.015713098038281, 0.2815783857085774, 0.4965510833518619, -0.26712339790718187, -0.7628323220424991, 1.3667781503580365]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_2", "z_2"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9784007316492808, 0.9791269412981409, 0.9369507260881303, 1.117300971310353, 1.1285206694663346], [0.0560079016086322, -0.027726665250832984, -0.13082883947746757, 2.1077590663005954, -0.9920071937010941, -1.7385929506450066, 0.2533684471318681, 0.7026828205414172]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_3", "z_3"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.0000639822547406, 1.2602490658796004, 0.9343266868723976, 0.9856454659803695, 1.1980631043429224], [1.6622945397040256, -0.27704372286248863, -0.04483067666343611, 1.2871252115889253, -0.10850099656460924, -0.4338920353234306, -0.27818269899805476, 0.9376973400616747]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_4", "z_4"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9881036026028093, 1.154993852371127, 1.2070500007391352, 0.8078000667256151, 1.0882486206461137], [2.1906050550475618, 0.060533761323479846, 0.38034315067934044, -0.5310366392182012, 0.8558966709798019, 1.357104197207177, 1.521598002560022, -1.035957182408182]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_5", "z_5"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.8587857799931959, 0.9829155988664573, 1.0103074054363523, 1.0796593215189785, 1.009189133374521], [-0.6084424257053828, -1.1435490720521222, 0.8037517694636558, -2.8378423066813343, 1.384701919831251, 0.8031374284139658, 0.8316529243105415, -0.8720271530799454]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_6", "z_6"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9862661916339934, 1.0366758336963993, 0.845080960982723, 1.0065123565913823, 0.9451793102914945], [0.9354316154305974, 0.7278339197819904, -0.08371910833475535, -2.167282238654885, -1.0370947901257994, -0.8381492612322792, 0.16065712285800368, 0.04640023900822478]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_7", "z_7"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.0608742044474695, 1.0875602475780748, 1.0484225025337204, 0.9190858581846861, 0.9723460334144294], [-0.7551984466365851, 0.19428681311945345, -0.5777529164584981, -0.5930128040240376, 1.7475947715459943, 0.22529191580944902, 2.2076141713998783, 0.5114069689778779]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_8", "z_8"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9845686514174975, 1.1567256378185813, 1.1276114638461345, 0.8816781352756572, 1.0723694074751526], [1.3781594326790647, 0.015389378277495161, -0.6141053960027436, 0.34749158434900834, -0.24459998838515024, -2.480820882214952, -1.0283112599170188, 1.3718396869399234]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_9", "z_9"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.0307486353246524, 1.2608055994842926, 0.9097028893727983, 0.9662871919985976, 0.9619607252654861], [0.4956976053685586, 0.42507094430491904, -0.5418982185742394, -0.317726736885681, -0.41050184106305465, 0.2687094534920061, -0.23357118612990543, -0.5569153237154446]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_10", "z_10"], UnitRange{Int64}[1:5, 6:13])  …  Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.990628228959984, 0.969899974294013, 1.092529189174722, 1.0075522781170554, 0.9319053754051071], [-0.06957186953537979, 1.40791455930566, 0.7924583955986539, 1.272796092023553, 1.4024235141555155, 0.00808318260325662, -0.9759026028813231, 0.925889337612231]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_91", "z_91"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9468383480461686, 1.1320708706795883, 1.0203801858813442, 0.9497683633028347, 0.9781289909358064], [-0.21872722754902083, 0.09447691824399246, 1.2324348281285924, 2.1215596201855944, -0.10641018428650567, 0.08109963708993785, 0.4903008744532721, -1.22835517302707]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_92", "z_92"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.055377024318632, 1.0609447373085141, 1.0331876493496972, 1.076220786052234, 1.033310514050847], [1.1750334077450277, -0.2726527442512461, -0.19513797093605703, 0.8150689827062736, 1.3699604113087112, -0.012460428439832999, -0.4812603694841129, -0.697865539073806]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_93", "z_93"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.9149222595728688, 0.8864075636300088, 0.9232613988708311, 0.8845975344774848, 0.9647894940637693], [1.052246034720292, -0.1413174863483201, 1.5701501544042427, 1.397051696502453, 1.8681391160454108, 0.9270429209652324, -0.39393840894001847, 0.308986648873291]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_94", "z_94"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.1306790338769794, 1.081344212313186, 1.0798391089621777, 0.8644375263752822, 0.9052589319703848], [-0.3236423627006615, -1.0172267675996407, -0.8178309802898678, 0.24517592874827282, -0.1230831988698798, 0.06410724139381468, -0.738056518571261, -0.3345962295636752]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_95", "z_95"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.018563882589364, 0.9603442463100473, 1.0120616047106819, 0.8434414127336579, 0.9876660269569281], [0.42284626777257817, 3.0318597146831943, -0.9025882417127168, -0.33451228233994784, 1.9024132945916161, 2.055637296448282, -0.3766701373330951, 0.12111181204244428]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_96", "z_96"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.1375449173717929, 1.0318296523778214, 0.910933674024934, 1.1031310020102445, 0.9048250460165492], [-1.0779564153043955, -0.6344197544326204, -0.27700032318200063, -0.45046129311446587, 0.9768831736119006, 0.1933696252087883, -1.603729216178044, 0.8229303287191068]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_97", "z_97"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.1013143015706817, 0.8695448524550822, 0.9651570025774938, 0.8990464680578264, 0.9784144621882922], [1.926730453570408, -0.9330358978285508, 0.3853323440387592, -0.09238846691197605, -1.494108561853626, -0.662734657793724, 0.9784985967676673, -0.12390978198744432]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_98", "z_98"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[0.8985416731766039, 1.013604199079614, 1.037821379717455, 1.1064780145309574, 0.9002374751070993], [-0.5614113912303987, -0.46495878756479075, -1.1187365338202204, -0.3833074035463776, 1.3438608656334263, -2.602220289934727, 0.07275334882451068, -0.32477070519552775]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_99", "z_99"], UnitRange{Int64}[1:5, 6:13]), Observation{Vector{Vector{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{AbstractMatrix{Float64}}, Vector{String}, Vector{UnitRange{Int64}}}([[1.027093941818456, 1.071643277586429, 1.0537320093494065, 0.9169999784041873, 0.909319237770267], [-0.8846281617271056, 0.3554452014923427, -0.31800655928444244, 0.821795738427613, -0.9305237687631139, -0.22053851437863078, -0.7987167301247443, 1.0828668251851552]], AbstractMatrix{Float64}[[0.01 0.0 … 0.0 0.0; 0.0 0.01 … 0.0 0.0; … ; 0.0 0.0 … 0.01 0.0; 0.0 0.0 … 0.0 0.01], [1.0 0.1 … 0.0 0.0; 0.1 1.0 … 0.0 0.0; … ; 0.0 0.0 … 1.0 0.1; 0.0 0.0 … 0.1 1.0]], AbstractMatrix{Float64}[[100.0 0.0 … 0.0 0.0; 0.0 100.0 … 0.0 0.0; … ; 0.0 0.0 … 100.0 0.0; 0.0 0.0 … 0.0 100.0], [1.010205144336438 -0.1020514433643792 … 1.0735488188434786e-6 -1.0735488188434786e-7; -0.10205144336437919 1.020514433643792 … -1.0735488188434786e-5 1.0735488188434786e-6; … ; 1.0735488188434788e-6 -1.0735488188434786e-5 … 1.020514433643792 -0.10205144336437919; -1.073548818843479e-7 1.0735488188434788e-6 … -0.10205144336437919 1.010205144336438]], ["y_100", "z_100"], UnitRange{Int64}[1:5, 6:13])], RandomFixedSizeMinibatcher{String, Random.TaskLocalRNG, Vector{Vector{Int64}}}(5, "extend", Random.TaskLocalRNG(), [[46, 70, 4, 75, 24], [38, 100, 66, 63, 80], [47, 6, 15, 91, 2], [67, 3, 31, 93, 64], [14, 72, 37, 27, 48], [30, 77, 32, 42, 60], [18, 59, 56, 88, 13], [65, 5, 25, 1, 12], [97, 55, 71, 51, 36], [33, 23, 52, 84, 10], [99, 58, 79, 43, 76], [89, 87, 39, 90, 94], [9, 54, 62, 28, 34], [57, 40, 21, 74, 85], [73, 35, 86, 61, 29], [11, 41, 20, 22, 49], [45, 95, 19, 8, 50], [83, 26, 96, 78, 16], [53, 17, 82, 98, 68], [92, 44, 7, 81, 69]]), ["series_1", "series_2", "series_3", "series_4", "series_5", "series_6", "series_7", "series_8", "series_9", "series_10"  …  "series_91", "series_92", "series_93", "series_94", "series_95", "series_96", "series_97", "series_98", "series_99", "series_100"], Dict("minibatch" => 1, "epoch" => 1), [[[46, 70, 4, 75, 24], [38, 100, 66, 63, 80], [47, 6, 15, 91, 2], [67, 3, 31, 93, 64], [14, 72, 37, 27, 48], [30, 77, 32, 42, 60], [18, 59, 56, 88, 13], [65, 5, 25, 1, 12], [97, 55, 71, 51, 36], [33, 23, 52, 84, 10], [99, 58, 79, 43, 76], [89, 87, 39, 90, 94], [9, 54, 62, 28, 34], [57, 40, 21, 74, 85], [73, 35, 86, 61, 29], [11, 41, 20, 22, 49], [45, 95, 19, 8, 50], [83, 26, 96, 78, 16], [53, 17, 82, 98, 68], [92, 44, 7, 81, 69]]])

# some example methods to get information out at the current minibatch:
get_current_minibatch(observation_series) # returns [i₁, ..., i₅],  the current minibatch subset of indices 1:100

5-element Vector{Int64}:
 46
 70
  4
 75
 24

get_obs(observation_series) # returns [yi₁, zi₁, ..., yi₅, zi₅], the data sample for the current minibatch

65-element Vector{Float64}:
  0.9216156042919288
  1.083418618771406
  1.022134688990878
  0.8179527195612566
  1.1396705462311523
  0.8746314204070936
 -0.9772716476905813
  0.4702814249907512
 -0.5071304599794773
  0.8772331669656114
  ⋮
  0.9633143017532118
 -2.1525558107277547
  0.7759060023910418
  0.42569283243202316
  0.031150855906457227
  0.4753682064455161
  0.2501918281856889
  1.2356195804354853
 -0.022952726859103864

get_obs_noise_cov(observation_series) # returns block-diagonal matrix with blocks [cov_yi₁  0 ... 0 ; 0 cov_zi₁ 0 ... 0; ... ; 0 ... 0 cov_yi₅ 0; 0 ... 0 cov_zi₅]

65×65 Matrix{Float64}:
 0.01  0.0   0.0   0.0   0.0   0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.01  0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.01  0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.01  0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.01  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   1.0  0.1  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.1  1.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.1     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 ⋮                             ⋮         ⋱            ⋮                   
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.1  0.0  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     1.0  0.1  0.0  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.1  1.0  0.1  0.0  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0  …  0.0  0.1  1.0  0.1  0.0  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.1  1.0  0.1  0.0  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.1  1.0  0.1  0.0
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.1  1.0  0.1
 0.0   0.0   0.0   0.0   0.0   0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.1  1.0

minibatches are updated internally to the update_ensemble!(ekp,...) step via a call to

update_minibatch!(observation_series)
get_current_minibatch(observation_series)

5-element Vector{Int64}:
  38
 100
  66
  63
  80

Minibatchers

Minibatchers are modular and must be derived from the Minibatcher abstract type. They contain a method create_new_epoch!(minibatcher,args...;kwargs) that creates a sampling of an epoch of data. For example, if we have 100 data observations, the epoch is 1:100, and one possible minibatching is a random partitioning of 1:100 into a batch-size (e.g., 5) leading to 20 minibatches.

Some of the implemented Minibatchers

FixedMinibatcher(given_batches, "order"), (default method = "order"), minibatches are fixed and run through in order for all epochs
FixedMinibatcher(given_batches, "random"), minibatches are fixed, but are run through in a randomly chosen order in each epoch
no_minibatcher(size), creates a FixedMinibatcher with just one batch which is the epoch (effectively no minibatching)
RandomFixedSizeMinibatcher(minibatch_size, "trim"), (default method = "trim") creates minibatches of size minibatch_size by randomly sampling the epoch, if the minibatch size does not divide into the number of samples it will ignore the remainder (and thus preserving a constant batch size)
RandomFixedSizeMinibatcher(minibatch_size, "extend"), creates minibatches of size minibatch_size by randomly sampling the epoch, if the minibatch size does not divide into the number of samples it will include the remainder in the final batch (and thus will cover the entirety of the data, with a larger final batch)

Identifiers

One can additionally provide a vector of names to name each Observation in the ObservationSeries by giving using the Dict entry "names" => names.

To think about the differences between the identifiers for Observation and ObservationSeries consider an application of observing the average state of a dynamical system over 100 time windows. The time windows will be batched over during the calibration.

The compiled information is given in the object:

yz_observation_series::ObservationSeries

As this contains many time windows, setting the names of the ObservationSeries objects to index the time window is a sensible identifier, for example,

get_names(yz_observation_series)
> ["window_1", "window_2", ..., "window_100"]

The individual Observations should refer only to the state being measured, so suitable identifiers might be, for example,

obs = get_observations(yz_observation_series)[1] # get first observation in the series
get_names(obs)
> ["y_window_average", "z_window_average"]