Visualization

ClimaCalibrate provides plotting utilities for plotting the mean forward map evaluation, columns of the G ensemble matrix, and the true observation via a Makie extension.

To plot the mean forward map evaluation, columns of the G ensemble matrix, the true observation, and the normalized residual, you can use Visualization.plot_g_mean, Visualization.plot_g, Visualization.plot_obs, and Visualization.plot_residual respectively. The mutating versions also exist as Visualization.plot_g_mean!, Visualization.plot_g!, Visualization.plot_obs!, and Visualization.plot_residual!. All plotting functions takes an EKP.EnsembleKalmanProcess object to plot from. Additionally, the plotting function accept an iter keyword argument for plotting from a specific iteration. If the keyword argument is not provided, then the last iteration is used for plotting. You can expect all keyword arguments that work with Makie.Lines to also work with these plotting functions and that the plotting functions behave like Makie plotting functions.

Example

Here is a complete example where we use the plotting functions to plot the ensemble members, the mean forward map evaluation, and the true observations from the second iteration.

import ClimaCalibrate
# To use this extension, one of the Makie backends should be loaded
import CairoMakie

fig = CairoMakie.Figure()
ax = CairoMakie.Axis(
    fig[1, 1],
    title = "G ensemble members, mean forward map evaluation, and observations",
    xlabel = "Index",
    ylabel = "Value",
)
g_plot = ClimaCalibrate.Visualization.plot_g!(
    ax,
    ekp;
    iter = 2,
    color = :black,
    alpha = 0.2,
)
g_mean_plot =
    ClimaCalibrate.Visualization.plot_g_mean!(ax, ekp; iter = 2, color = :black)
obs_plot =
    ClimaCalibrate.Visualization.plot_obs!(ax, ekp; iter = 2, color = :blue)

CairoMakie.Legend(
    fig[1, 2],
    [g_plot, g_mean_plot, obs_plot],
    ["G", "G mean", "Observation"],
)

fig

Residual diagnostics

Plotting the ensemble tells you whether it is approaching the observation. analyze_residual tells you something more specific: how much of the remaining residual is structured, meaning aligned with the leading directions of the observational noise covariance, rather than noise-like.

import ClimaAnalysis   # required
result = ClimaCalibrate.analyze_residual(ekp, iteration; n_eigenvectors = 3)
result.structured_energy               # about 1 under the noise model
result.structured_energy_by_variable   # how that splits between variables
result.residual_norm_by_variable       # which variable dominates the residual

It projects mean(G) - obs onto the leading eigenvectors of the noise covariance, normalized by the corresponding eigenvalues, so the projections are z-scores: a value much larger than one means the residual has structure the noise model does not account for. A high structured energy in one variable points at that variable's observation or its part of the observation map.

This requires ClimaAnalysis to be loaded, and observations built by ObservationRecipe, since it uses their metadata to attribute the residual to individual variables. It supports SVD, Diagonal, and SVDplusD covariances.

We can also plot the normalized residual (mean(G) - obs) / σ from the second iteration, where σ is the square root of the diagonal of the observation noise covariance.

fig = CairoMakie.Figure()
ClimaCalibrate.Visualization.plot_residual(fig[1, 1], ekp; iter = 2)
fig

Interpreting the residual

Each entry of the normalized residual measures the mismatch between the mean forward map evaluation and the observation in units of the observation noise standard deviation, so it can be read like a z-score.

• Sign: A positive value means the mean forward map evaluation over-predicts the observation at that index (positive bias), and a negative value means it under-predicts (negative bias).
• Magnitude: Values much larger than $\pm 2$ in magnitude indicate a mismatch that the noise model cannot explain.
• RMS as a summary: The root mean square (RMS) of the residual is a useful single-number summary. RMS much greater than 1 means learnable signal remains (or the noise covariance is too small). RMS near 1 means the calibration has fit to the noise floor and can no longer distinguish model error from the noise it was told to expect. RMS much less than 1 means the noise covariance is too large.
• Index order: The x-axis is the index into the stacked observation vector for that iteration. If you are using the ClimaAnalysis extension, you can use ObservationRecipe.reconstruct_residual to reconstruct the residual as OutputVars and plot them with the ClimaAnalysis plotting functions.
• Across iterations: As the calibration converges, the residual should shrink toward the noise level, with most entries settling within $\pm 2$ and the RMS approaching 1.