Backends

ClimaCalibrate runs calibrations on different distributed computing environments, called backends. Each suits a different set of computing resources.

Available backends

JuliaBackend: The simplest backend, which runs all members serially on a single machine. Best for initial testing and small calibrations that do not require parallelization.
WorkerBackend: Uses Julia's built-in distributed computing capabilities, assigning forward model runs to separate workers using Distributed.jl. Workers can be created using SlurmManager, Distributed.addprocs, or by initializing julia with the -p option: julia -p 2. Available workers can be accessed using Distributed.workers(). On a cluster, add_workers submits each worker as an individual scheduler allocation and adds it to a pool as it connects, so a calibration can start before all workers are up and picks up workers as they join. On clusters that charge for whole nodes (e.g. Derecho), pass workers_per_node to add_workers to put several workers, one per GPU, in each allocation.
HPC Cluster Backends: These backends schedule forward model runs on HPC clusters using Slurm or PBS.
- CaltechHPCBackend: Caltech's Resnick HPC cluster,
- ClimaGPUBackend: CliMA's private GPU server,
- DerechoBackend: NSF NCAR Derecho supercomputing system,
- GCPBackend: CliMA's Google cloud platform.

What are the differences between the `WorkerBackend` and the HPC cluster backends?

The main difference between the two backends is how the work of the ensemble members is distributed. The WorkerBackend keeps a pool of long-lived Julia workers and dispatches forward-model runs to them, so each worker precompiles once and reuses that code across all iterations. The HPC cluster backends instead submit a separate scheduler job per ensemble member and iteration; each job starts a fresh Julia process and precompiles again, but the jobs are independent, so an iteration makes progress as soon as any member is scheduled. Prefer the WorkerBackend when precompilation dominates runtime and the workers fit in your allocation; prefer an HPC cluster backend when forward models are long-running, need internal parallelism, or the cluster cannot hold all workers at once.

Loading code on workers

For a WorkerBackend, load your model code with @worker_setup rather than Distributed.@everywhere. Because workers can join asynchronously, @everywhere would miss any worker that connects after it runs; @worker_setup records the setup and replays it on each worker as it joins, so workers that join later get the same setup.

Choosing the right backend for calibration

The right backend is largely determined by the computational cost of your forward model.

If your model is simple or you are debugging, use the JuliaBackend.

If your model requires one CPU core or GPU, use the WorkerBackend.

If your forward model requires parallelization across multiple cores or GPUs, choose one of the HPC cluster backends. These allow you to allocate more resources to each forward model using Slurm or PBS.

Worker registry

For a WorkerBackend, the driver keeps a record of every worker launched with add_workers: its scheduler job id, Distributed id, output file, and WorkerState. Use it to see which workers are queued, idle, or running a member, why a worker ended, and where its log is, without querying the scheduler. List the workers in the driver with worker_rows:

julia> worker_rows()
3-element Vector{ClimaCalibrate.Backend.Workers.WorkerRecord}:
 Worker 1 in group 1: busy, job 4182734, pid 2, output /scratch/logs/worker-1.out
 Worker 2 in group 1: ready, job 4182735, pid 3, output /scratch/logs/worker-2.out
 Worker 3 in group 1: ended (scheduler reported FAILED), job 4182736, output /scratch/logs/worker-3.out

Pass states to filter them, e.g. worker_rows(; states = (ClimaCalibrate.Backend.Workers.ENDED,)) for the workers that have ended. The registry lives in the driver's memory only.

Each add_workers call is a launch group with a desired worker count. Every few seconds during a calibration, the driver checks the registry against the scheduler: it asks squeue or qstat about live jobs, ends workers that are not ready within startup_timeout seconds of submission (12 hours by default), and launches replacements for any group below its desired count, up to that group's max_relaunches. The only other way to grow the pool mid-run is to call add_workers again. A worker that dies while running a member still counts as a failed member and its replacement picks up later members. A worker that fails to load its code is stopped and replaced. Once two workers of a group fail to load their code and none has succeeded, the group launches no more workers, since the setup code is likely broken.

When the driver exits, an atexit hook cancels every job the session submitted by id and ends the live rows with exit_reason = "cancel_worker_jobs".

Job status

An HPCBackend reports an ensemble member's job as one of four JobStatus values:

Status	Meaning
`PENDING`	Queued, waiting to be scheduled
`RUNNING`	Executing
`COMPLETED`	Finished successfully
`FAILED`	Finished with an error, or was cancelled, timed out, or ran out of memory

Query one with job_status, or with the predicates ispending, isrunning, issuccess, isfailed, and iscompleted, which accept either a JobInfo or a JobStatus. Each JobInfo query shells out to the scheduler, so ask for the status once and test the result rather than calling several predicates on the job.

How it is determined

On Slurm, squeue only lists jobs that are still queued or running, so it answers PENDING and RUNNING. Once a job leaves the queue, its outcome comes from sacct: COMPLETED maps to success, and FAILED, CANCELLED, TIMEOUT, OUT_OF_MEMORY, NODE_FAIL, BOOT_FAIL, DEADLINE, PREEMPTED, REVOKED, SPECIAL_EXIT, and LAUNCH_FAILED all map to FAILED. A state the package does not recognize is treated as a failure, not a success.

Slurm accounting must be enabled

Distinguishing a successful job from a failed one requires sacct. If your cluster has no accounting database, ClimaCalibrate falls back to the member's checkpoint file, which is written by the forward model itself.

On PBS, qstat reports a job_state and a substate. Q, H, W, T, and M are PENDING; R, S, B, and E are RUNNING; F and X are finished, and are FAILED rather than COMPLETED when the substate is 93, which is how PBS records a job that exited with an error.

If the scheduler cannot be reached, the job is reported as still running and a warning is logged. A member that has been running for longer than the backend's job_timeout ends the iteration, and the remaining jobs are cancelled. The default is 24 hours, and a model that runs longer than that needs a larger one:

backend = ClimaCalibrate.CaltechHPCBackend(;
    directives = [:time => 60 * 48, :ntasks => 1],
    job_timeout = 60 * 60 * 48,
)

The clock starts when a job leaves the queue, so a long wait for an allocation does not count against it.

The checkpoint is the final word

Whatever the scheduler says, a member that did not write a completed checkpoint is counted as failed. A batch script can exit successfully while the model inside it did not, and a scheduler can lose the record of a job, so the checkpoint the forward model writes itself is the more reliable signal.