EnsembleKalmanProcesses

EnsembleKalmanProcesses.jl (EKP) is a library of derivative-free optimization and approximate Bayesian inference techniques based on ensemble Kalman Filters, a well known family of approximate filters used for data assimilation. The tools in this library enable fitting parameters found in expensive black-box computer codes without the need for adjoints or derivatives. This property makes them particularly useful when calibrating non-deterministic models, or when the training data are noisy.

See Installation to get started.

Our processes and quick recommendations

Here are loose recommendations and rough scalability in the current implementations

• Playground option: Inversion ($10^3$ inputs, $10^3$ outputs) - simple, handles large input spaces, and is very modifiable with all the bells-and-whistles of the package.
• Efficient option: TransformUnscented ($10^1$ inputs, $10^7$ outputs) - Very efficient, and quickly converging for large outputs. However, it strongly couples ensemble size to input dimension, and is not as robust to model failures and noise.
• Scalable and Robust option: TransformInversion ($10^2$ inputs, $10^7$ outputs) - Less efficient convergence than TransformUnscented, but only weakly couples ensemble size with input dimension, and more robust to model failures and noise.
• With uncertainty: Sampler ($10^2$ inputs, $10^3$ outputs) - generally slower to converge than inversion tools, but the final ensemble spread quantifies uncertainty.

Quick links!

• How do I build prior distributions?
• How do I access parameters/outputs from the ekp object?
• How do I plot convergence errors or parameter distributions?
• How do I build good observational noise covariances?
• How do I build my observations and encode batching?
• What ensemble size should I take? Which process should I use? What is the recommended configuration?
• What is the difference between get_u and get_ϕ? Why do the stored parameters appear to be outside their bounds?
• What can be parallelized? How do I do it in Julia?
• What is going on in my own code?
• What is this error/warning/message?
• Where can I walk through a simple example?

Learning the amplitude and vertical shift of a sine curve See full example for the code.

The library

Currently, the following processes are implemented in the library. More details given on respective pages:

• Inversion() creates Ensemble Kalman Inversion (EKI) "finite time" - The traditional optimization technique based on the (perturbed-observation-based) Ensemble Kalman Filter EnKF (Iglesias et al., 2013). This takes a transport view, initializing ensembles at the prior, and the posterior mode and (roughly approximated) uncertainty are estimated at finite algorithm time.

• Inversion(prior) creates Ensemble Kalman Inversion (EKI) "infinite time" - EKI with an augmented state that enforces the prior, (e.g., TEKI (Chada et al., 2020)). Can be initialized off-the-prior, and ensemble collapses to the posterior mode at infinite algorithm time (e.g., Section 4.5 of (Calvello et al., 2025)).

• TransformInversion() Ensemble Transform Kalman Inversion (ETKI) "finite time" - An optimization technique based on the (square-root-based) ensemble transform Kalman filter (Bishop et al., 2001; Huang et al., 2022).

• TransformInversion(prior) Ensemble Transform Kalman Inversion (ETKI) "infinite time" - ETKI with an augmented state that enforces the prior. (see EKI "infinite time")

• GaussNewtonInversion(prior) Gauss Newton Kalman Inversion (GNKI) [a.k.a. Iterative Ensemble Kalman Filter with Statistical Linearization] - An optimization technique based on the Gauss Newton optimization update and the iterative extended Kalman filter (Chada et al., 2021; Chen and Oliver, 2013),

• Sampler(prior) Ensemble Kalman Sampler (EKS) - also obtains a Gaussian Approximation of the posterior distribution, through a Monte Carlo integration (Garbuno-Inigo et al., 2020), ("ALDI" variant)

• Unscented(prior) Unscented Kalman Inversion (UKI) - also obtains a Gaussian Approximation of the posterior distribution, through a quadrature based integration approach (Huang et al., 2022),

• TransformUnscented(prior) Transform Unscented Kalman Inversion (UTKI) - An implementation of the UKI algorithm based on the linear-algebra tricks of the square-root filter (see ETKI).

• SparseInversion(γ) Sparsity-inducing Ensemble Kalman Inversion (SEKI) - Additionally adds approximate $L^0$ and $L^1$ penalization to the EKI (Schneider et al., 2022).

Authors

EnsembleKalmanProcesses.jl is being developed by the Climate Modeling Alliance. See the contributors page for the developers.