State debug statistics
This page shows how to use the StateCheck functions to get basic statistics for nodal values of fields held in ClimateMachine MPIStateArray data structures. The StateCheck functions can be used to
Generate statistics on
MPIStateArrayholding the state of a ClimateMachine experiment.and to
Compare against saved reference statistics from ClimateMachine
MPIStateArrayvariables. This can enable simple automated regression test checks for detecting unexpected changes introduced into numerical experiments by code updates.
These two cases are shown below:
1. Generating statistics for a set of MPIStateArrays
Here we create a callback that can generate statistics for an arbitrary set of the MPIStateArray type variables of the sort that hold persistent state for ClimateMachine models. We then invoke the call back to show the statistics.
In regular use the MPIStateArray variables will come from model configurations. Here we create a dummy set of MPIStateArray variables for use in stand alone examples.
Create a dummy set of MPIStateArrays
First we set up two MPIStateArray variables. This need a few packages to be in placeT, and utilizes some utility functions to create the array and add named persistent state variables. This is usually handled automatically as part of model definition in regular ClimateMachine activity. Calling ClimateMachine.init() includes initializing GPU CUDA and MPI parallel processing options that match the hardware/software system in use.
Set up a basic environment
using MPI
using StaticArrays
using Random
using ClimateMachine
using ClimateMachine.VariableTemplates
using ClimateMachine.MPIStateArrays
using ClimateMachine.GenericCallbacks
using ClimateMachine.StateCheck
ClimateMachine.init()
FT = Float64Float64Define some dummy vector and tensor abstract variables with associated types and dimensions
F1 = @vars begin
ν∇u::SMatrix{3, 2, FT, 6}
κ∇θ::SVector{3, FT}
end
F2 = @vars begin
u::SVector{2, FT}
θ::SVector{1, FT}
end
nothing # hideCreate MPIStateArray variables with arrays to hold elements of the vectors and tensors
Q1 = MPIStateArray{Float32, F1}(
MPI.COMM_WORLD,
ClimateMachine.array_type(),
4,
9,
8,
)
Q2 = MPIStateArray{Float64, F2}(
MPI.COMM_WORLD,
ClimateMachine.array_type(),
4,
3,
8,
)
nothing # hideCreate a call-back
Now we can create a StateCheck call-back, cb, tied to the MPIStateArray variables Q1 and Q2. Each MPIStateArray in the array of MPIStateArray variables tracked is paired with a label to identify it. The call-back is also given a frequency (in time step numbers) and precision for printing summary tables.
cb = ClimateMachine.StateCheck.sccreate(
[(Q1, "My gradients"), (Q2, "My fields")],
1;
prec = 15,
)
GenericCallbacks.init!(cb, nothing, nothing, nothing, nothing)
nothing # hide# SC Start: creating state check callback
# SC Creating state check callback labeled "My gradients" for symbols
# SC ν∇u
# SC κ∇θ
# SC Creating state check callback labeled "My fields" for symbols
# SC u
# SC θ
# SC Finish: creating state check callback
Invoke the call-back
The call-back is of type ClimateMachine.GenericCallbacks.EveryXSimulationSteps and in regular use is designed to be passed to a ClimateMachine timestepping solver e.g.
typeof(cb)ClimateMachine.GenericCallbacks.EveryXSimulationStepsHere, for demonstration purposes, we can invoke the call-back after simply initializing the MPIStateArray fields to a random set of values e.g.
Q1.data .= rand(MersenneTwister(0), Float32, size(Q1.data))
Q2.data .= rand(MersenneTwister(0), Float64, size(Q2.data))
GenericCallbacks.call!(cb, nothing, nothing, nothing, nothing)# SC +++++++++++ClimateMachine StateCheck call-back start+++++++++++++++++
# SC Step = 0000000
┌──────────────┬────────┬────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────┐
│ Label │ Field │ min() │ max() │ mean() │ std() │
├──────────────┼────────┼────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┤
│ My gradients │ ν∇u[1] │ 1.3434886932373047e-04 │ 9.8473286628723145e-01 │ 5.2354550361633301e-01 │ 3.0820992588996887e-01 │
│ My gradients │ ν∇u[2] │ 1.1631786823272705e-01 │ 9.9208831787109375e-01 │ 4.8380064964294434e-01 │ 2.8335043787956238e-01 │
│ My gradients │ ν∇u[3] │ 1.0584592819213867e-03 │ 9.5177590847015381e-01 │ 4.6547442674636841e-01 │ 2.7361556887626648e-01 │
│ My gradients │ ν∇u[4] │ 5.9766888618469238e-02 │ 9.6804809570312500e-01 │ 5.4261803627014160e-01 │ 2.8157085180282593e-01 │
│ My gradients │ ν∇u[5] │ 8.3103060722351074e-02 │ 9.3593192100524902e-01 │ 5.0540590286254883e-01 │ 2.4607349932193756e-01 │
│ My gradients │ ν∇u[6] │ 3.0968189239501953e-02 │ 9.9834144115447998e-01 │ 4.5437556505203247e-01 │ 3.0946105718612671e-01 │
│ My gradients │ κ∇θ[1] │ 8.4744811058044434e-02 │ 9.9418067932128906e-01 │ 5.2715736627578735e-01 │ 2.9245597124099731e-01 │
│ My gradients │ κ∇θ[2] │ 1.2051463127136230e-02 │ 9.9352765083312988e-01 │ 4.7106358408927917e-01 │ 2.9644900560379028e-01 │
│ My gradients │ κ∇θ[3] │ 8.1498026847839355e-02 │ 9.5544338226318359e-01 │ 5.0503891706466675e-01 │ 2.7720102667808533e-01 │
│ My fields │ u[1] │ 3.5344549147287685e-02 │ 9.7311877457023077e-01 │ 4.5356637667336486e-01 │ 3.2862244822350628e-01 │
│ My fields │ u[2] │ 4.2301665932029664e-02 │ 9.6779955361920011e-01 │ 5.4030723072852616e-01 │ 2.9460315332773757e-01 │
│ My fields │ θ[1] │ 6.2367558170158821e-02 │ 9.7655012304114752e-01 │ 5.1604631241833421e-01 │ 2.8198324416490389e-01 │
└──────────────┴────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┘
# SC +++++++++++ClimateMachine StateCheck call-back end+++++++++++++++++++
2. Comparing to reference values
Generate arrays of reference values
StateCheck functions can generate text that can be used to set the value of stored arrays that can be used in a reference test for subsequent regression testing. This involves 3 steps.
Step 1. First a reference array setting program code is generated from the latest state of a given callback e.g.
ClimateMachine.StateCheck.scprintref(cb)# BEGIN SCPRINT
# varr - reference values (from reference run)
# parr - digits match precision (hand edit as needed)
#
# [
# [ MPIStateArray Name, Field Name, Maximum, Minimum, Mean, Standard Deviation ],
# [ : : : : : : ],
# ]
varr = [
[ "My gradients", "ν∇u[1]", 1.34348869323730468750e-04, 9.84732866287231445313e-01, 5.23545503616333007813e-01, 3.08209925889968872070e-01 ],
[ "My gradients", "ν∇u[2]", 1.16317868232727050781e-01, 9.92088317871093750000e-01, 4.83800649642944335938e-01, 2.83350437879562377930e-01 ],
[ "My gradients", "ν∇u[3]", 1.05845928192138671875e-03, 9.51775908470153808594e-01, 4.65474426746368408203e-01, 2.73615568876266479492e-01 ],
[ "My gradients", "ν∇u[4]", 5.97668886184692382813e-02, 9.68048095703125000000e-01, 5.42618036270141601563e-01, 2.81570851802825927734e-01 ],
[ "My gradients", "ν∇u[5]", 8.31030607223510742188e-02, 9.35931921005249023438e-01, 5.05405902862548828125e-01, 2.46073499321937561035e-01 ],
[ "My gradients", "ν∇u[6]", 3.09681892395019531250e-02, 9.98341441154479980469e-01, 4.54375565052032470703e-01, 3.09461057186126708984e-01 ],
[ "My gradients", "κ∇θ[1]", 8.47448110580444335938e-02, 9.94180679321289062500e-01, 5.27157366275787353516e-01, 2.92455971240997314453e-01 ],
[ "My gradients", "κ∇θ[2]", 1.20514631271362304688e-02, 9.93527650833129882813e-01, 4.71063584089279174805e-01, 2.96449005603790283203e-01 ],
[ "My gradients", "κ∇θ[3]", 8.14980268478393554688e-02, 9.55443382263183593750e-01, 5.05038917064666748047e-01, 2.77201026678085327148e-01 ],
[ "My fields", "u[1]", 3.53445491472876849315e-02, 9.73118774570230771204e-01, 4.53566376673364857197e-01, 3.28622448223506280485e-01 ],
[ "My fields", "u[2]", 4.23016659320296639635e-02, 9.67799553619200114696e-01, 5.40307230728526155517e-01, 2.94603153327737565803e-01 ],
[ "My fields", "θ[1]", 6.23675581701588210848e-02, 9.76550123041147521974e-01, 5.16046312418334207628e-01, 2.81983244164903890105e-01 ],
]
parr = [
[ "My gradients", "ν∇u[1]", 16, 16, 16, 16 ],
[ "My gradients", "ν∇u[2]", 16, 16, 16, 16 ],
[ "My gradients", "ν∇u[3]", 16, 16, 16, 16 ],
[ "My gradients", "ν∇u[4]", 16, 16, 16, 16 ],
[ "My gradients", "ν∇u[5]", 16, 16, 16, 16 ],
[ "My gradients", "ν∇u[6]", 16, 16, 16, 16 ],
[ "My gradients", "κ∇θ[1]", 16, 16, 16, 16 ],
[ "My gradients", "κ∇θ[2]", 16, 16, 16, 16 ],
[ "My gradients", "κ∇θ[3]", 16, 16, 16, 16 ],
[ "My fields", "u[1]", 16, 16, 16, 16 ],
[ "My fields", "u[2]", 16, 16, 16, 16 ],
[ "My fields", "θ[1]", 16, 16, 16, 16 ],
]
# END SCPRINT
Step 2. Next the array setting program code is executed (see below). At this stage the parr[] array context may be hand edited. The parr[] array sets a target number of decimal places for matching against reference values in varr[]. For different experiments and different fields the degree of precision that constitutes failing a regression test may vary. Choosing the parr[] values requires some sense as to the stability of the particular numerical and physical scenario an experiment represents. In the example below some precision settings have been hand edited from the default of 16 to illustrate the process.
#! format: off
varr = [
[ "My gradients", "ν∇u[1]", 1.34348869323730468750e-04, 9.84732866287231445313e-01, 5.23545503616333007813e-01, 3.08209930764271777814e-01 ],
[ "My gradients", "ν∇u[2]", 1.16317868232727050781e-01, 9.92088317871093750000e-01, 4.83800649642944335938e-01, 2.83350456014221541157e-01 ],
[ "My gradients", "ν∇u[3]", 1.05845928192138671875e-03, 9.51775908470153808594e-01, 4.65474426746368408203e-01, 2.73615551085745090099e-01 ],
[ "My gradients", "ν∇u[4]", 5.97668886184692382813e-02, 9.68048095703125000000e-01, 5.42618036270141601563e-01, 2.81570862027933854765e-01 ],
[ "My gradients", "ν∇u[5]", 8.31030607223510742188e-02, 9.35931921005249023438e-01, 5.05405902862548828125e-01, 2.46073509972619536290e-01 ],
[ "My gradients", "ν∇u[6]", 3.09681892395019531250e-02, 9.98341441154479980469e-01, 4.54375565052032470703e-01, 3.09461067853178561915e-01 ],
[ "My gradients", "κ∇θ[1]", 8.47448110580444335938e-02, 9.94180679321289062500e-01, 5.27157366275787353516e-01, 2.92455951648181833313e-01 ],
[ "My gradients", "κ∇θ[2]", 1.20514631271362304688e-02, 9.93527650833129882813e-01, 4.71063584089279174805e-01, 2.96449027197666359346e-01 ],
[ "My gradients", "κ∇θ[3]", 8.14980268478393554688e-02, 9.55443382263183593750e-01, 5.05038917064666748047e-01, 2.77201022741208891187e-01 ],
[ "My fields", "u[1]", 4.31410233294131639781e-02, 9.97140933049696531754e-01, 4.62139750850942054861e-01, 3.23076684924287371725e-01 ],
[ "My fields", "u[2]", 1.01416659908237782872e-02, 9.14712023896926407218e-01, 4.76160523012988778913e-01, 2.71443440757963339038e-01 ],
[ "My fields", "θ[1]", 6.58965491052394547467e-02, 9.73216404386510802738e-01, 4.60007166313864512830e-01, 2.87310472114545079059e-01 ],
]
parr = [
[ "My gradients", "ν∇u[1]", 16, 7, 16, 0 ],
[ "My gradients", "ν∇u[2]", 16, 7, 16, 0 ],
[ "My gradients", "ν∇u[3]", 16, 7, 16, 0 ],
[ "My gradients", "ν∇u[4]", 16, 7, 16, 0 ],
[ "My gradients", "ν∇u[5]", 16, 7, 16, 0 ],
[ "My gradients", "ν∇u[6]", 16, 7, 16, 0 ],
[ "My gradients", "κ∇θ[1]", 16, 16, 16, 0 ],
[ "My gradients", "κ∇θ[2]", 16, 16, 16, 0 ],
[ "My gradients", "κ∇θ[3]", 16, 16, 16, 0 ],
[ "My fields", "u[1]", 16, 16, 16, 0 ],
[ "My fields", "u[2]", 16, 16, 16, 0 ],
[ "My fields", "θ[1]", 16, 16, 16, 0 ],
]
#! format: on12-element Array{Array{Any,1},1}:
["My gradients", "ν∇u[1]", 16, 7, 16, 0]
["My gradients", "ν∇u[2]", 16, 7, 16, 0]
["My gradients", "ν∇u[3]", 16, 7, 16, 0]
["My gradients", "ν∇u[4]", 16, 7, 16, 0]
["My gradients", "ν∇u[5]", 16, 7, 16, 0]
["My gradients", "ν∇u[6]", 16, 7, 16, 0]
["My gradients", "κ∇θ[1]", 16, 16, 16, 0]
["My gradients", "κ∇θ[2]", 16, 16, 16, 0]
["My gradients", "κ∇θ[3]", 16, 16, 16, 0]
["My fields", "u[1]", 16, 16, 16, 0]
["My fields", "u[2]", 16, 16, 16, 0]
["My fields", "θ[1]", 16, 16, 16, 0]Step 3. Finally a call-back stored value can be compared for consistency to with parr[] decimal places
ClimateMachine.StateCheck.scdocheck(cb, (varr, parr))
nothing # hide# SC +++++++++++ClimateMachine StateCheck ref val check start+++++++++++++++++
# SC "N( )" bracketing indicates field failed to match
┌──────────────┬────────┬───────┬───────┬────────┬───────┬───────┬───────┬─────────────┐
│ Label │ Field │ min() │ max() │ mean() │ std() │ Pass │ Fail │ Not checked │
│ │ │ │ │ │ │ count │ count │ count │
├──────────────┼────────┼───────┼───────┼────────┼───────┼───────┼───────┼─────────────┤
│ My gradients │ ν∇u[1] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[2] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[3] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[4] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[5] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[6] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[1] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[2] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[3] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My fields │ u[1] │ N(1) │ N(3) │ N(3) │ 0 │ 2 │ 3 │ 1 │
│ My fields │ u[2] │ N(1) │ N(3) │ N(1) │ 0 │ 2 │ 3 │ 1 │
│ My fields │ θ[1] │ N(3) │ N(4) │ N(1) │ 0 │ 2 │ 3 │ 1 │
└──────────────┴────────┴───────┴───────┴────────┴───────┴───────┴───────┴─────────────┘
# SC +++++++++++ClimateMachine StateCheck ref val check end+++++++++++++++++
In this trivial case the match is guaranteed. The function will return true to the calling routine and this can be passed to an @test block.
However we can modify the reference test values to see the effect of a mismatch e.g.
varr[1][3] = varr[1][3] * 10.0
ClimateMachine.StateCheck.scdocheck(cb, (varr, parr))
nothing # hide# SC +++++++++++ClimateMachine StateCheck ref val check start+++++++++++++++++
# SC "N( )" bracketing indicates field failed to match
┌──────────────┬────────┬───────┬───────┬────────┬───────┬───────┬───────┬─────────────┐
│ Label │ Field │ min() │ max() │ mean() │ std() │ Pass │ Fail │ Not checked │
│ │ │ │ │ │ │ count │ count │ count │
├──────────────┼────────┼───────┼───────┼────────┼───────┼───────┼───────┼─────────────┤
│ My gradients │ ν∇u[1] │ N(0) │ 10 │ 19 │ 0 │ 4 │ 1 │ 1 │
│ My gradients │ ν∇u[2] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[3] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[4] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[5] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ ν∇u[6] │ 19 │ 10 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[1] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[2] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My gradients │ κ∇θ[3] │ 19 │ 19 │ 19 │ 0 │ 5 │ 0 │ 1 │
│ My fields │ u[1] │ N(1) │ N(3) │ N(3) │ 0 │ 2 │ 3 │ 1 │
│ My fields │ u[2] │ N(1) │ N(3) │ N(1) │ 0 │ 2 │ 3 │ 1 │
│ My fields │ θ[1] │ N(3) │ N(4) │ N(1) │ 0 │ 2 │ 3 │ 1 │
└──────────────┴────────┴───────┴───────┴────────┴───────┴───────┴───────┴─────────────┘
# SC +++++++++++ClimateMachine StateCheck ref val check end+++++++++++++++++
Here the mis-matching field is highlighted with N(0) indicating that the precision was not met and actual match length was (in this case) 0. If any field fails the test returns false for use in any regression testing control logic.
This page was generated using Literate.jl.