Infrastructure API

These small support modules make physical field semantics explicit, expose opt-in host/GPU timing instrumentation, check configuration tables, and publish files only when they are complete.

Quantity traits

AtmosTransport.Quantities Module

Quantities

Dispatch traits that classify a field by its regrid-and-conservation semantics. Regridding.apply_regridder!(dst, r, src, kind) implements the rule of each kind for every regridder, and the Preprocessing CS regrid helpers (regrid_2d_to_cs_panels!, regrid_3d_to_cs_panels!) take a kind argument and pass it on, so extensive/intensive distinctions stay explicit at the type level.

The four kinds are not interchangeable. Confusing one for another is the exact mistake that produced ~12× polar mass deficits in C180 ERA5 binaries (the regrid path treated extensive m as intensive, distorting the spatial distribution proportional to source-cell area variation).

Kind	Examples	Regrid handling
IntensiveCellField	ps, qv, T, mixing ratios, cell-center winds	Pass-through to apply_regridder!.
ExtensiveCellField	m (kg/cell), tracer mass, accumulated emissions mass	Convert via density: src ./ src_areas → regrid → × dst_areas.
HorizontalVectorField	cell-center (u, v) viewed as a tangent vector	Regrid components as intensive, then rotate into the target panel-local basis at every CS panel.
HorizontalFluxField	window-summed am, bm, accumulated face-flux integrals	Directed integrals across faces; correct handling needs face-aware regridding (MAPL CONSERVE_HFLUX-style) or reconstruction from regridded winds + pressure. The pipeline reconstructs from winds (reconstruct_cs_fluxes!).

IntensiveCellField and ExtensiveCellField cover all scalar cell fields and are the only kinds apply_regridder! accepts. Passing HorizontalVectorField or HorizontalFluxField makes them throw an ArgumentError, because those fields need richer handling than scalar regridding can give.

The trait types are small singletons. Use them as a final positional argument on regrid helpers, not as wrapper types around arrays — staying out of the array data type keeps the runtime hot loops untouched and lets us introduce these dispatch tags incrementally.

source

AtmosTransport.Quantities.ExtensiveCellField Type

ExtensiveCellField <: QuantityKind

Per-cell total: m in kg/cell, tracer mass, accumulated emissions mass. Conservative regridding requires conversion to density (intensive) on the source grid, then re-multiplication by destination cell area to recover the extensive total. Without this conversion, the regridded field is distorted in proportion to source-grid area variation — at the LL pole, cells are 230× smaller than at the equator, producing severe spatial errors in CS polar cells.

source

AtmosTransport.Quantities.HorizontalFluxField Type

HorizontalFluxField <: QuantityKind

Directed face-flux integral: window-summed am, bm, accumulated mass flux across a face over a time window. Correct regridding requires face-aware operators (MAPL CONSERVE_HFLUX) or reconstruction from regridded winds + pressure. The pipeline reconstructs face fluxes from regridded cell-center winds and a target-grid pressure thickness (reconstruct_cs_fluxes!); the scalar regrid helpers reject this kind with an ArgumentError.

source

AtmosTransport.Quantities.HorizontalVectorField Type

HorizontalVectorField <: QuantityKind

Tangent-plane vector field defined at cell centers: cell-center (u, v) winds. The components are individually intensive, but at every CS panel the vector itself must be re-expressed in the target panel-local basis. This requires a rotation step on top of component-wise regridding: rotate_winds_to_panel_local! (Preprocessing) projects the regridded east/north components onto panel-local face normals built from panel_cell_local_tangent_basis (Grids). The scalar regrid helpers reject this kind with an ArgumentError.

source

AtmosTransport.Quantities.IntensiveCellField Type

IntensiveCellField <: QuantityKind

Per-area or point-value cell field: ps, qv, T, mixing ratios, cell-center wind components, etc. Conservative area-weighted regridding gives a correct destination-cell average without unit conversion.

source

AtmosTransport.Quantities.QuantityKind Type

QuantityKind

Marker supertype for the four field-classification traits.

source

Section timing

AtmosTransport.SectionTimer.disable! Method

disable!()

Stop accumulating; existing samples remain in _TIMINGS until enable! is called again.

source

AtmosTransport.SectionTimer.enable! Method

enable!(; timing=true, allocations=false, nvtx=false)

Reset and turn on. timing controls host-side wall-clock accumulation; nvtx emits NVTX ranges for nsys/Nsight Compute timelines (off by default — under ATMOSTR_NVTX=1 it is set independently of timing).

source

AtmosTransport.SectionTimer.maybe_enable_from_env! Method

maybe_enable_from_env!()

Inspect ENV["ATMOSTR_TIMERS"], ENV["ATMOSTR_NVTX"], ENV["ATMOSTR_ALLOC_TIMERS"] at call time. "1" / "true" / "on" / "yes" switches each axis on independently; anything else (or unset) is a no-op for that axis. Returns true if anything was enabled.

source

AtmosTransport.SectionTimer.report Function

report(io = stderr)

Print a per-section summary table. Columns: section, n_calls, total_s, mean_ms, p50_ms, p95_ms, max_ms, frac%, plus alloc_MiB and mean_KiB when allocation samples exist. frac% is over the sum of section totals, not over wall-clock; the header line reports wall-clock time and the time covered by sections separately.

source

AtmosTransport.SectionTimer.time_section Method

time_section(f, name::Symbol)

Function-form equivalent of @section. Use when the timed region is a do-block or already a closure.

source

AtmosTransport.SectionTimer.write_csv Method

write_csv(path)

Emit the same summary as report to a CSV at path. Header: section,n_calls,total_s,mean_ms,p50_ms,p95_ms,max_ms,fraction_of_total,allocated_bytes,mean_alloc_bytes. fraction_of_total is a fraction (0–1), not a percentage; the allocation columns are 0 unless allocation timing was enabled. Returns the path on success, or nothing if there are no samples.

source

AtmosTransport.SectionTimer.@section Macro

@section name expr

Time expr and accumulate the elapsed nanoseconds under name (a Symbol). When ATMOSTR_NVTX=1, also emits an NVTX range labeled with name. When everything is off the macro just executes expr with zero overhead beyond a single atomic flag load.

source

Configuration checks

AtmosTransport.ConfigChecks Module

ConfigChecks

Shared helpers for reading TOML configuration tables: strict Boolean values and known-key checks that suggest the intended key for a misspelled one. Output, Preprocessing and Models use them, so this module is loaded right after Architectures.

source

AtmosTransport.ConfigChecks.check_known_keys Method

check_known_keys(cfg, allowed, label)

Throw an ArgumentError listing every key of cfg that is not in allowed, for tables where an unknown key is an error.

source

AtmosTransport.ConfigChecks.config_bool Method

config_bool(value, path) -> Bool
config_bool(cfg, key, default, path) -> Bool

Return a TOML Boolean, rejecting anything else ("true", 1, ...) with an ArgumentError that names the setting path.

source

AtmosTransport.ConfigChecks.key_suggestion Method

key_suggestion(key, allowed) -> Union{String, Nothing}

The allowed key a misspelled key most likely means: the closest one by case-insensitive edit distance, if at most 2 (1 for keys of up to four characters), else an allowed key that contains key (at least four characters long) or is contained in it. nothing if none qualifies.

source

AtmosTransport.ConfigChecks.unknown_key_messages Method

unknown_key_messages(cfg, allowed) -> Vector{String}

One entry per key of the table cfg that is not in allowed, sorted, each with a suggestion when one exists, e.g. $k{z}_{m}ax(didyoumean‘K{z}_{m}ax‘?)$.

source

Log levels

AtmosTransport.LogLevels Module

LogLevels

The [logging] level of run and preprocessing configs: parse it and run code at that level. Preprocessing and Models both use it, so this module is loaded right after ConfigChecks.

source

AtmosTransport.LogLevels.LevelFilterLogger Type

LevelFilterLogger(parent, min_level)

A logger that sets the minimum level of parent: it passes the messages at or above min_level to parent (also below the parent's own minimum, so debug messages reach a console logger) and drops the others.

source

AtmosTransport.LogLevels.log_level Method

log_level(cfg) -> Union{Nothing, Logging.LogLevel}

The [logging] level of a config: "debug", "info", "warn" or "error", or nothing when it is not set (the caller's logger is then left as it is). Anything else is an ArgumentError.

source

AtmosTransport.LogLevels.with_log_level Method

with_log_level(f, level)

Run f() with the current logger at minimum level level (a Logging.LogLevel); level = nothing leaves the logger as it is.

source

Atomic file publication

Every transport-binary writer, the spectral day cache and the regridder cache write to a staging file next to the target and rename it onto the target only when the file is complete; a failed write removes the staging file and leaves a previous target untouched. Publication is one replacing rename (rename(2) on POSIX systems) on every Julia version, so readers see the previous file until the new one replaces it, and a failed rename never deletes the target. The staging file is created with O_EXCL and mode 0o666, so published files get 0o666 & ~umask (never the 0o600 of mktemp).

AtmosTransport.AtomicFiles Module

AtomicFiles

Publish a file only when it is complete. A writer fills a staging file next to the target and renames it onto the target when it has finished; on an error the staging file is removed and a previous target is left untouched. A reader of the target therefore sees the previous file or the complete new one, never a partial file.

Staging file contract (one implementation for every writer in the package):

• Name. "<abspath(path)>.tmp-<pid>-<n>", with n a random UInt32 from libc's generator (Libc.rand), so the task-local Random stream of a seeded run does not depend on whether a file was written. The file is a sibling of the target, so the final rename stays on one filesystem and never copies.

• Creation. open(2) with O_CREAT | O_EXCL: no other process can own the name (an existing name is retried with a new n), and nothing is truncated or followed.

• Mode. Created with 0o666, so the published file gets 0o666 & ~umask, exactly like a file created with open(path, "w"). mktemp is never used on POSIX systems: it creates mode 0o600, and the rename would publish an owner-only file. (On Windows, where POSIX modes do not apply, the staging file comes from mktemp in the target's directory.)

• Publication. One replacing rename of the staging file onto the target, on every Julia version: libuv's uv_fs_rename (through the public Base.rename on Julia ≥ 1.12, through jl_fs_rename, the call it wraps, on 1.10 and 1.11), which is rename(2) on POSIX systems and MoveFileExW with MOVEFILE_REPLACE_EXISTING on Windows. On POSIX systems the target is replaced atomically: the path never disappears, and a reader opens either the previous file or the complete new one. If the rename fails, the staging file is removed and the previous target is left as it was. Base's mv(tmp, path; force = true) is not used: it deletes the target and renames again when a rename fails (before renaming at all on Julia 1.10). Nor is Julia 1.10's Base.Filesystem.rename, which copies and deletes when the rename fails. A cross-device rename (EXDEV) is therefore an error, never a copy; since the staging file is a sibling of the target it does not occur in practice. An existing directory at path is refused.

• Durability. No fsync: the guarantee is towards concurrent readers, not across a power failure.

Entry points: atomic_publish for writers that take a path (JLD2, NetCDF, open(path, "w")), and open_atomic_staging, commit_atomic_staging and abandon_atomic_staging for writers that keep the stream open across calls (the streaming transport-binary writers).

Uses Base only, so it loads before Quantities; MetDrivers (transport binaries), Regridding (regridder caches) and Preprocessing (spectral caches) use it.

source

AtmosTransport.AtomicFiles.abandon_atomic_staging Method

abandon_atomic_staging(tmp) -> Nothing

Remove the staging file tmp after a failed write; a missing file is not an error. The caller closes its stream first. The target is not touched.

source

AtmosTransport.AtomicFiles.atomic_publish Method

atomic_publish(f, path) -> String

Write path atomically: f(tmp) writes the complete file at the staging path tmp (any writer that takes a path, e.g. open(tmp, "w"), JLD2.jldsave or NCDataset(tmp, "c")), which is then renamed onto path. If f throws, tmp is removed, path is left as it was, and the error is rethrown. The staging file is created by open_atomic_staging (O_EXCL, mode 0o666 & ~umask) and closed before f runs; f may truncate or replace it. Returns path.

atomic_publish(path) do tmp
    open(io -> write(io, payload), tmp, "w")
end

source

AtmosTransport.AtomicFiles.commit_atomic_staging Method

commit_atomic_staging(tmp, path) -> String

Publish the closed staging file tmp (from open_atomic_staging) as path by one replacing rename (see AtomicFiles), so a previous file at path stays readable until the new one takes its place. If path has become a directory (ArgumentError) or the rename fails (Base.IOError, also for a cross-device rename), tmp is removed, a previous file at path is left as it was, and the error is thrown. Returns path.

source

AtmosTransport.AtomicFiles.open_atomic_staging Method

open_atomic_staging(path) -> (tmp::String, io::IOStream)

Create the staging file for path (see AtomicFiles for its name, O_EXCL creation and mode 0o666 & ~umask) and return its name with an IOStream open for reading and writing. The descriptor returned by open(2) becomes io; the staging path is never reopened.

The caller writes through io, closes it, and then calls commit_atomic_staging(tmp, path), or abandon_atomic_staging(tmp) if anything failed. Throws an ArgumentError if the parent directory of path does not exist or path is a directory.

source