Preprocessing API
For narrative coverage of the source × target dispatch model and per-source pipelines, see Preprocessing overview and the per-source pages ERA5 spectral path and GEOS native cubed-sphere.
AtmosTransport.Preprocessing Module
PreprocessingTransport binary generation from meteorological input.
Converts raw meteorology into mass-balanced transport binaries ready for the runtime TransportBinaryDriver. Supported sources and targets:
ERA5 spectral GRIB (VO, D, LNSP) → lat-lon, reduced-Gaussian, or cubed sphere
GEOS native cubed-sphere NetCDF (GEOS-IT, GEOS-FP) → cubed sphere (same resolution or nested block coarsening)
ERA5 N320 native GRIB → cubed sphere (conservative regrid)
MERRA-2 lat-lon NetCDF → cubed sphere (conservative regrid)
an existing lat-lon transport binary → cubed sphere (
regrid_transport_binary)
Architecture
The pipeline dispatches on AbstractTargetGeometry subtypes:
AbstractTargetGeometry
├── LatLonTargetGeometry — regular lat-lon (any Nx × Ny)
├── ReducedGaussianTargetGeometry — native ERA5 RG (O90, O160, N320, …)
└── CubedSphereTargetGeometry — convention-aware CS (C24, C90, C180, …)and on the source settings type. Each supported pair has a process_day method:
process_day(date, grid::LatLonTargetGeometry, settings::ERA5SpectralSettings, vertical; …)
process_day(date, grid::ReducedGaussianTargetGeometry, settings::ERA5SpectralSettings, vertical; …)
process_day(date, grid::CubedSphereTargetGeometry, settings::ERA5SpectralSettings, vertical; …)
process_day(date, grid::CubedSphereTargetGeometry, settings::AbstractGEOSSettings, vertical; …)
process_day(date, grid::CubedSphereTargetGeometry, settings::AbstractERA5GRIBSettings, vertical; …)
process_day(date, grid::CubedSphereTargetGeometry, settings::MERRA2Settings, vertical; …)ERA5 spectral pipeline (per day)
Read ERA5 spectral GRIB (VO, D, LNSP) —
spectral_io.jlSpectral synthesis (Legendre + FFT → gridpoint) —
spectral_synthesis.jlMerge native 137L → transport levels —
vertical_coordinates.jlPin global mean ps (mass fix) —
pin_global_mean_ps!/pin_global_mean_ps_using_qv!inmass_support.jlColumn Poisson balance of the horizontal mass fluxes (header key
horizontal_balance = "column"): one Poisson solve for the column-integrated divergence against the column mass tendency, each face's correction spread over the layers by the column weights (spread_column_correction!; air mass in hybrid layers by default), socmkeeps the vertical motion:
LL: FFT Poisson solve —
balance_column_mass_fluxes!,mass_support.jlRG: compressed-Laplacian CG —
balance_compressed_column_fluxes!,ring_poisson_balance.jlCS: global 6-panel graph-Laplacian CG —
balance_cs_column_mass_fluxes!,cs_poisson_balance.jl
Diagnose
cmfrom explicit endpoint mass targets — replay continuityWrite transport binary with declared payload semantics and replay checks
The GEOS, ERA5 N320, MERRA-2, and binary-regrid paths implement their own mass construction or regridding, flux balancing, and cm diagnosis (with an optional dry-mass pin where the source has one); every path ends in the same write-time replay and positivity gates.
The high-level transport-binary workflows are split under transport_binary/. Each meteorological source/target pair owns a process_day method (binary regridding has its own entry point) while shared binary metadata, endpoint-delta, and replay helpers stay in common files.
Usage
julia --project=. scripts/preprocessing/preprocess_transport_binary.jl config.toml --day 2021-12-01Advanced users can call process_day directly from Julia.
AtmosTransport.Preprocessing.AbstractBinaryWriter Type
AbstractBinaryWriter{G <: AbstractTargetGeometry, FT,
Basis <: AbstractMassBasis}Typed nominal for the topology's streaming binary writer. The third type parameter encodes the on-disk mass-basis convention (reusing State.AbstractMassBasis so the same nominal flows through the runtime reader path) so a writer↔reader pairing mismatch is a compile-time MethodError.
Concrete subtypes (writer_adapters.jl): LatLonBinaryWriter, ReducedGaussianBinaryWriter, CubedSphereBinaryWriter.
AtmosTransport.Preprocessing.AbstractChainPolicy Type
AbstractChainPolicyType-system tag for cross-day mass-state carry. Concrete subtypes are NoChain (no carry; each day starts from the raw source endpoint) and ChainedMass{T} where T is the array shape of the seed (e.g. NTuple{6, Array{Float32, 3}} for GEOS cubed-sphere).
AtmosTransport.Preprocessing.AbstractERA5GRIBSettings Type
AbstractERA5GRIBSettings <: AbstractMetSettingsAbstract supertype for ERA5 native-GRIB sources. Concrete subtypes pick the source mesh via the flavor parameter on ERA5GRIBSettings.
AtmosTransport.Preprocessing.AbstractFieldKind Type
AbstractFieldKindSingleton-type tag selecting the vertical-reduction rule apply_vertical! uses for one payload field. Concrete subtypes (all zero-size singletons) below.
AtmosTransport.Preprocessing.AbstractMetReader Type
AbstractMetReader{FT, S, CP}Typed met-reader nominal. Bundles a met-source's settings, per-day handle context, and chained-mass-policy carry into one struct that the unified preprocessor driver dispatches on. Type parameters:
FT <: AbstractFloat— preprocessing float type (Float32for GPU,Float64for diagnostic runs).S <: AbstractMetSettings— concrete settings type (GEOSITSettings,GEOSFPSettings,ERA5SpectralSettings, …).CP <: AbstractChainPolicy—NoChainorChainedMass{T}for some seed array typeT.
Concrete subtypes implement the seven trait functions below.
sourceAtmosTransport.Preprocessing.AbstractMetSettings Type
AbstractMetSettingsTop-level supertype for typed met-data source descriptors used by the preprocessor. Concrete subtypes (GEOSITSettings, GEOSFPSettings, ERA5N320Settings, MERRA2Settings, ERA5SpectralSettings) carry source-specific paths and parameters and implement the parts of the read_window! / source_grid / windows_per_day interface that their process_day path uses (GEOS implements all three).
process_day(date, grid::AbstractTargetGeometry, settings::AbstractMetSettings, vertical; ...) dispatches on settings to pick the reader.
AtmosTransport.Preprocessing.AbstractVerticalTransform Type
AbstractVerticalTransformTyped nominal selecting how native source levels are mapped to output levels. Concrete subtypes:
IdentityVertical— no merge.MergeByIndex— explicit native-level groups.MergeLayersThinnerThan— automatic local coarsening.MergeAbovePressure— upper-atmosphere coarsening.LevelSelection— echlevs-style level selection.PressureOverlap— pressure-thickness overlap onto a different hybrid coordinate.
A concrete transform T is consumed by plan_vertical(transform, native_vc) to produce a VerticalPlan{FT, T}. apply_vertical! dispatches on (plan, ::FieldKind) to select the right per-field rule.
AtmosTransport.Preprocessing.AbstractWindowContract Type
AbstractWindowContract{G <: AbstractTargetGeometry, FT}Typed nominal owning a topology's per-window gate policy and the worst-window accumulator state. Concrete subtypes:
CubedSphereContract{FT}(cubed_sphere_contracts.jl)LatLonContract{FT}(latlon_contracts.jl)ReducedGaussianContract{FT}(reduced_gaussian_contracts.jl)
A concrete contract validates its policy at construction time (so e.g. positivity_cfl_limit = 0.0 errors before any window is preprocessed) and exposes the four-method trait surface:
verify_window!(window, contract, win_idx) -> (replay, positivity)
update_accumulator!(contract, positivity_diag, win_idx) -> nothing
summarize_status!(contract; quarantine_path) -> nothingwindow is the topology-specific window payload: a NamedTuple of typed buffers or a ReadyWindow{G, FT} wrapping one.
Keeps contract knobs from being drift-prone kwargs — each topology constructs its own contract once from config, with whatever fields IT needs (see DESIGN.md).
sourceAtmosTransport.Preprocessing.AbstractWindowWorkspace Type
AbstractWindowWorkspace{G <: AbstractTargetGeometry, FT}Typed nominal for the per-day target-shape workspace buffers. Concrete subtypes: LatLonSpectralWindowWorkspace, CubedSphereSpectralWindowWorkspace, and GEOSCubedSphereWindowWorkspace.
AtmosTransport.Preprocessing.ChainedMass Type
ChainedMass{T} <: AbstractChainPolicyThe reader carries an end-of-day mass field of shape T into the next day's open. The shape T is the seed array type (e.g. NTuple{6, Array{Float32, 3}}); the actual seed VALUE lives in the reader struct, but its STATIC type is encoded here so end_of_day_seed return-type is known at compile time.
AtmosTransport.Preprocessing.ConvectionInterfaceFlux Type
Convective interface flux (e.g. cmfmc). Same selection rule as PressureFluxField.
AtmosTransport.Preprocessing.ConvectionTendencyField Type
Convective center tendency (e.g. dtrain). Extensive at the layer; sum within merged group.
AtmosTransport.Preprocessing.CubedSphereContract Type
CubedSphereContract{FT} <: AbstractWindowContract{CubedSphereTargetGeometry, FT}Typed nominal owning a CS preprocessor's per-window gate policy and worst-window positivity accumulator.
Fields:
replay_tol::Float64— relative replay tolerance.positivity_cfl_limit::Float64— per-substep positivity CFL gate. Must satisfy0 < limit ≤ 1; validated at construction.require_substep_positivity::Bool— whethersummarize_status!errors (true) or warns (false) on a positivity violation.steps_per_window::Int— for the recommended-steps message in the summary's escape-hatch detail. Must be ≥ 1.halo_width::Int— passed through toverify_substep_positivity_cs!.worst::NamedTuple— mutable accumulator (initially zero).
Construct with explicit kwargs; defaults match the CS round-2/round-3 production policy.
sourceAtmosTransport.Preprocessing.ERA5C180RawConvectionFields Type
Raw ECMWF convection diagnostics after conservative N320 → C180 mapping. Conversion to TM5 entrainment/detrainment deliberately happens only after this step, matching tmm_Read_Convec_EC in the vendored TM5 procedure.
AtmosTransport.Preprocessing.ERA5C180RegridFields Type
ERA5C180RegridFields{FT}Per-window output container for fields regridded onto the C180 cubed-sphere target. Holds:
ps— 2D(Nc, Nc)matrix per panel (Pa, moist surface pressure),u,v,t,qv— 3D(Nc, Nc, Nz)arrays per panel (U/V in geographic east/north frame; rotation to panel-local axes happens in the breakpoint-F glue where the panel basis is known).
Mass fields (m_dry, delp_dry, ps_dry) are not regridded directly; they are re-derived on the C180 mesh from the regridded PS + Q so that Σ_k DELP_dry == PS_dry to roundoff on the target side as well.
AtmosTransport.Preprocessing.ERA5C180RegridWorkspace Type
ERA5C180RegridWorkspace{FT, R}Owns the conservative regridder + flat scratch buffers used by regrid_n320_to_c180!. Allocated once per (source_grid, target_grid) pair and reused across every window. coverage is the fraction of each target cell covered by the source mesh (see _regrid_intensive!).
AtmosTransport.Preprocessing.ERA5C180TM5ConvectionFields Type
ERA5C180TM5ConvectionFields{FT}TM5 convection fields derived on the C180 cubed-sphere target after the raw ECMWF diagnostics have been conservatively mapped there. Each field is a 6-tuple of (Nc, Nc, Nz) panel arrays, layer-centered, kg / m² / s.
AtmosTransport.Preprocessing.ERA5GRIBDayHandles Type
ERA5GRIBDayHandles{S<:AbstractERA5GRIBSettings}Per-day source-file context. Carries the resolved on-disk paths to the day's GRIB streams plus the optional next-day core path that supplies the right endpoint of the last window.
convection_path is nothing unless settings.include_convection is set; likewise surface_path is nothing unless settings.include_surface is set. next_core_path is nothing either when the caller passed next_day_handle=false or when the next-day file is not on disk (last day of the available archive). core_index holds the ecCodes index of core_path once the first window has been read.
AtmosTransport.Preprocessing.ERA5GRIBSettings Type
ERA5GRIBSettings{flavor} <: AbstractERA5GRIBSettingsTyped settings for one ERA5 native-GRIB flavor:
flavor = :n320— reduced linear-Gaussian N320, the default MARS native grid for ERA5 analyses (640 longitudes at the equator, 137 hybrid levels).
ERA5 archives and the runtime both use top-down hybrid levels (k = 1 is the TOA-side layer). Only the GRIB horizontal ring order is reversed on ingestion.
AtmosTransport.Preprocessing.ERA5HumidityRemap Type
ERA5HumidityRemapSpecific humidity of one ERA5 N320 day conservatively remapped to another horizontal mesh, for converting convective fluxes that live on that mesh to dry air. Open with open_era5_humidity_remap, fill with remap_era5_humidity!, release with close.
AtmosTransport.Preprocessing.ERA5N320ConvectionFields Type
ERA5N320ConvectionFields{FT}Per-window convection forecast fields on the N320 source mesh. All four fields are (n_cells, Nz):
udmf— updraft convective mass flux (kg m⁻² s⁻¹)ddmf— downdraft convective mass flux (kg m⁻² s⁻¹)udrf— updraft detrainment rate (kg m⁻³ s⁻¹)ddrf— downdraft detrainment rate (kg m⁻³ s⁻¹)
Downstream conversion to GEOS-style CMFMC + DTRAIN (or TM5-style entu/entd) happens in the breakpoint-F glue once both source and target geometries are known.
sourceAtmosTransport.Preprocessing.ERA5N320DryMassFields Type
ERA5N320DryMassFields{FT}Per-window dry-basis output container. m_dry is dry-air mass per cell per layer (kg), delp_dry is dry pressure thickness per cell per layer (Pa), ps_dry is the column-integrated dry surface pressure (Pa). ps_dry_acc is a Float64 accumulator that backs ps_dry so Σ_k DELP_dry stays accurate even when FT = Float32 (137-layer summation with ~10 hPa values per layer would otherwise lose ~100 Pa to single-precision rounding).
AtmosTransport.Preprocessing.ERA5N320SpectralWorkspace Type
ERA5N320SpectralWorkspace{FT, G}Per-window workspace for ERA5 N320 spectral synthesis. Owns:
the spectral coefficient cubes for T, VO, D, LNSP for the current hour (sized
(T+1) × (T+1) × NzinComplexF64),per-level scratch matrices
u_spec/v_specreused inside the synthesis loop,a
ReducedSpectralThreadCachewith Legendre column buffer plus FFT and real ring buffers sized to every unique ring length in the source mesh,a
read_bufscratch used byread_spectral_coeffs!for the raw ecCodescodes_get_double_arraypayload,completion bookkeeping (
have_t/have_vo/have_d/have_lnsp).
The cubes dominate memory: at T = 639 and Nz = 137 each cube is ≈ 0.9 GB. Workspaces are intended to be allocated once per day-handle and reused across the 24 hours.
Spectral buffers (vo_spec, d_spec, t_spec, lnsp_spec, u_spec, v_spec, synth_cache.P_buf) are unconditionally ComplexF64 / Float64 regardless of FT — read_spectral_coeffs! and vod2uv! only operate at that precision. FT only controls the eltype of the downstream gridpoint fields written via read_era5_n320_window_fields!.
AtmosTransport.Preprocessing.ERA5N320ToC180Pipeline Type
ERA5N320ToC180Pipeline{FT, RW, CSGrid, SrcGrid, W}All-in-one container for the per-day ERA5 N320 → C180 preprocessing workspace. Holds the source-grid descriptor, the hybrid coordinate, the shared per-cell area vector, every read/derive/regrid workspace from breakpoints B-D, the convection workspace from E, and the per-window output fields on both the source mesh and the C180 target.
One pipeline allocated per day-handle, reused across the 24 hourly windows.
sourceAtmosTransport.Preprocessing.ERA5N320WindowFields Type
ERA5N320WindowFields{FT}Per-window output container. u / v (m/s, geographic frame), t (K), qv (kg/kg specific humidity) are (n_cells, Nz); ps (Pa) is (n_cells,). Dry-basis layer mass derivation, regridding, and convection conversion are downstream of this struct.
AtmosTransport.Preprocessing.ERA5SpectralReader Type
ERA5SpectralReader{FT, S} <: AbstractMetReader{FT, S, NoChain}Typed reader nominal for the ERA5 spectral path. ChainPolicy is fixed at NoChain because today's spectral path does not carry cross-day mass state (it pins global-mean ps per window instead). The per-window read still fuses with merge in today's process_window!; splitting it out is deferred.
AtmosTransport.Preprocessing.ERA5SpectralSettings Type
ERA5SpectralSettings <: AbstractMetSettingsTyped wrapper for the historical ERA5 spectral settings NamedTuple. The spectral preprocessor still performs spectral synthesis inside the topology workspaces, but the source axis is now explicit: TOML parsing constructs this settings type and topology methods dispatch on it.
AtmosTransport.Preprocessing.FidelitySampleMap Type
Canonical sampled face identifiers, stable across processes and Julia versions.
sourceAtmosTransport.Preprocessing.FluxFidelitySample Type
One canonical face/level comparison; mirrors are represented by one face id.
sourceAtmosTransport.Preprocessing.GEOSDayHandles Type
GEOSDayHandlesOpen NCDataset handles for one UTC day's GEOS collections plus the resolved level orientation and the hybrid coordinate (loaded once for endpoint-DELP reconstruction). The orchestrator opens these once at the start of process_day and closes them at the end.
AtmosTransport.Preprocessing.GEOSNativeReader Type
GEOSNativeReader{FT, S, CP, V, H} <: AbstractMetReader{FT, S, CP}Typed reader wrapping (GEOSSettings, handles) for one day, plus optional chained-mass state. Type parameters:
FT— preprocessing float type.S <: AbstractGEOSSettings— concrete settings (GEOSITSettings / GEOSFPSettings / …).CP <: AbstractChainPolicy—NoChainorChainedMass{T}.V— seed-field slot type.NothingforNoChain;Union{Nothing, NTuple{6, Array{FT, 3}}}forChainedMass. The V parameter is fixed at construction (Julia-style review round-1: value-dependent V was the type-instability foot-gun; we pin V to the unconditional Union on the chained path so the inner constructor specializes once).H— concrete handles type. Pinned totypeof(open_day(...))at construction soreader.handlesaccess is type-stable (replaces the previous:: Anyfield). Different GEOS flavors produce different handle structs (GEOSDayHandlesfor GEOS-IT,GEOSFPNativeDayHandlesfor GEOS-FP); each becomes its own concrete reader type via this parameter.
Constructor: open_reader(settings::AbstractGEOSSettings, date, FT; seed, next_day_handle). See the function docstring.
AtmosTransport.Preprocessing.GEOSSettings Type
GEOSSettings{flavor} <: AbstractGEOSSettingsSettings for one of the two supported GEOS flavors:
flavor = :geosit— GEOS-IT (file patternGEOSIT.{date}.{collection}.C{Nc}.nc).flavor = :geosfp— GEOS-FP (file patternGEOSFP.{date}.{collection}.C{Nc}.nc).
Auto-detection of level orientation runs at open_geos_day time when level_orientation = :auto. Set explicitly to :bottom_up or :top_down to skip the heuristic.
AtmosTransport.Preprocessing.IdentityVertical Type
No-op identity vertical transform. Nz_output = Nz_native, merge_map[k] = k.
AtmosTransport.Preprocessing.IntensiveCenterField Type
Center-level intensive field (e.g. T, Q). Mass-weighted mean within merged group; weights provided positionally.
AtmosTransport.Preprocessing.LatLonContract Type
LatLonContract{FT} <: AbstractWindowContract{LatLonTargetGeometry, FT}Typed nominal owning an LL preprocessor's per-window gate policy and worst-window positivity accumulator. Construction validates positivity_cfl_limit ∈ (0, 1] and steps_per_window ≥ 1 so an invalid TOML value fails before any window runs.
AtmosTransport.Preprocessing.LevelSelection Type
LevelSelection(echlevs)Typed wrapper for the existing select_levels_echlevs algorithm. echlevs is a vector of native level INTERFACE indices (0-based, numbered from the model top as in TM5); levels between selected interfaces are summed. See vertical_coordinates.jl:64 and the ECHLEVS_ML137_* constants.
AtmosTransport.Preprocessing.MERRA2Archive Type
NASAArchive()
GEOSChemArchive()MERRA-2 file layouts. NASAArchive: the GES DISC collections under {root_dir}/M2{I,T}3NVASM/YYYY/MM/ (PS, QV, U, V only). GEOSChemArchive: the GEOS-Chem-processed files {root_dir}/YYYY/MM/MERRA2.YYYYMMDD.{tag}.05x0625.nc4, which also carry the convection and boundary-layer fields.
AtmosTransport.Preprocessing.MERRA2DayHandles Type
MERRA2DayHandlesOpen NCDataset handles for one UTC day:
inst3— PS/QV instantaneous endpoints (always open).tavg3— U/V time-averaged winds;nothingwhensettings.winds_collection == :inst3(winds then come frominst3).next_inst3— next day's inst3 dataset, for the final window's right endpoint look-ahead;nothingon the archive boundary or whennext_day_handle = false.a3mste,a3mstc,a1— GEOS-Chem A3mstE (CMFMC), A3mstC (DQRCU) and A1 (PBL surface fields); open only when the corresponding output is requested.level_order,next_level_order—TopDown()orSurfaceFirst(), detected from the day's and the next day's inst3 QV; the day's order applies to all of that day's collections (one archive, one convention).
AtmosTransport.Preprocessing.MERRA2LevelOrder Type
TopDown()
SurfaceFirst()Level order of a MERRA-2 file: the GES DISC files are top-down (k = 1 at the model top, like the L72 coefficient table), the GEOS-Chem files surface-first.
sourceAtmosTransport.Preprocessing.MERRA2Settings Type
MERRA2Settings <: AbstractMetSettingsSettings for the MERRA-2 wind-derived cubed-sphere preprocessor.
root_dir— directory holding the MERRA-2 collection trees ({root_dir}/M2{I,T}3NVASM/{YYYY}/{MM}/...nc4).coefficients_file— hybrid σ-pressure coefficient TOML (the GEOS L72 table; MERRA-2 shares the GEOS-5 L72 coordinate).winds_collection—:tavg3(time-averaged U/V, the default and the GEOS-Chem-faithful choice) or:inst3(instantaneous U/V from the inst3 collection, no separate tavg3 file needed;NASAArchiveonly).archive—NASAArchive()orGEOSChemArchive()(seeMERRA2Archive).include_surface— write PBLH, USTAR, HFLUX, T2M (GEOSChemArchiveonly).include_convection— write dry CMFMC and DTRAIN (GEOSChemArchiveonly).include_vdiff_fields— write the Holtslag-Boville VDIFF fields U, V, T, QV and the latent heat flux EFLUX (implies the surface fields;GEOSChemArchiveonly).include_convective_cloud_base— write GCHP's convective cloud base, the lowest layer with DQRCU > 0 (A3mstC; needsinclude_convection).
AtmosTransport.Preprocessing.MERRA2ToC180Pipeline Type
MERRA2ToC180Pipeline{FT, R, P, E, W}Per-day MERRA-2 → CS preprocessing workspace. Owns the conservative LL→CS regridder, the CS preprocess scratch, and the per-window regridded target-grid scalar fields (c180_fields.{ps, qv, u, v}, any Nc), laid out as NTuple{6, …} panels so the shared CS helpers (derive_c180_dry_mass!, rotate_winds_to_panel_local!, reconstruct_cs_fluxes!) work unchanged.
phys holds the regridded optional physics panels (cmfmc, dtrain, t, dqrcu and its cloud_base, and surface_hours, the three hourly A1 records of the block as (pblh, ustar, hflux, t2m[, eflux]) panels — only those requested; empty without physics); edge_src/edge_dst are the regridder buffers for the Nz+1-edge CMFMC; wind selects how U/V are regridded (ScalarWindRegrid or CartesianWindRegrid, which owns its buffers).
Two pipelines per day (block start and end), reused across the 8 blocks.
sourceAtmosTransport.Preprocessing.MassField Type
Center-level extensive mass (e.g. delp, m). Sum native layers within each merged group.
AtmosTransport.Preprocessing.MassFluxField Type
Horizontal face mass flux (e.g. am, bm). Already integrated over the layer thickness; sum within merged group.
AtmosTransport.Preprocessing.MergeAbovePressure Type
MergeAbovePressure(pressure_Pa; target_min_thickness_Pa = Inf,
reference_surface_pressure_Pa = STANDARD_PRESSURE)Upper-atmosphere coarsening: native layers whose midpoint pressure is LOWER than pressure_Pa (physically ABOVE the cutoff altitude) get greedily merged until each merged layer exceeds target_min_thickness_Pa. Below the cutoff, the native grid is preserved verbatim.
target_min_thickness_Pa = Inf merges every above-cutoff native layer into one top cap. The GEOS-IT L72 use case is pressure_Pa = 100.0 + target_min_thickness_Pa = 50.0 — merges the ~14 Pa mesospheric layers into ~50 Pa output layers while keeping the troposphere and stratosphere at native resolution.
AtmosTransport.Preprocessing.MergeByIndex Type
MergeByIndex(groups)Explicit native center-level groups. groups[l] is the UnitRange{Int} of native center levels merged into output level l. Validation at plan_vertical:
groups[1]starts at 1;groups[end]ends atNz_native;groups are contiguous (
groups[l+1].start == groups[l].stop + 1);each range is non-empty.
This is the most auditable transform for production reruns — the group list lives in the run TOML and is version-controlled.
sourceAtmosTransport.Preprocessing.MergeLayersThinnerThan Type
MergeLayersThinnerThan(min_thickness_Pa; reference_surface_pressure_Pa = STANDARD_PRESSURE)Typed wrapper for the merge_thin_levels algorithm: merge adjacent native layers from the surface up, closing each group as soon as its thickness at the reference surface pressure reaches min_thickness_Pa. The native layers left above the last group become the top output layer, which can be thinner than min_thickness_Pa (ERA5 L137 at 100 Pa: 117 layers, top layer 6.8 Pa from native layers 1–4); every other output layer is at least that thick.
AtmosTransport.Preprocessing.NativeFluxEvidence Type
Native-flux windows (GEOS): no independent reconstruction exists, so the evidence records only how far the column balance moved the sampled native amounts; the certificate status is not_applicable.
AtmosTransport.Preprocessing.NoChain Type
NoChain <: AbstractChainPolicyThe reader does not carry mass state across days. Day N starts from the raw source endpoint. end_of_day_seed(::reader{…, NoChain}) returns nothing (statically inferred).
AtmosTransport.Preprocessing.PreprocessorRunCache Type
PreprocessorRunCache{G, FT}()Small typed cache for artifacts that should be built once per preprocessing run instead of once per day/window (for example spectral LL->CS regridders or RG compressed Laplacians). The ERA5 spectral entry point builds one per run and passes it to every day's process_day; each driver owns its own keys.
AtmosTransport.Preprocessing.PressureFluxField Type
Vertical interface mass flux (e.g. cm). Interfaces are selected (not summed); top/bottom zeros preserved.
AtmosTransport.Preprocessing.PressureOverlap Type
PressureOverlap(target_coeff_path)Remap native layer integrals onto an independent target hybrid coordinate by pressure-thickness overlap. The target half-level coefficients are loaded from target_coeff_path. The full apply_vertical! implementation lands alongside the spectral driver cutover; plan_vertical constructs the plan today.
AtmosTransport.Preprocessing.PreverifiedWindow Type
PreverifiedWindow(ready::ReadyWindow, contract_diag; accumulated=false)Typed event wrapper for topology hooks that have already run verify_window! before handing a window to the generic driver. When accumulated=true, the hook also guarantees it has already folded the positivity diagnostic into the contract accumulator.
AtmosTransport.Preprocessing.RawWindow Type
RawWindow{FT, A2, A3}Per-window source-grid intermediate carrying both window endpoints (t_n and t_{n+1}) and the window-integrated horizontal mass fluxes between them.
The right endpoint of window n is the left endpoint of window n+1, so the reader caches it across calls. For the last window of the day, the right endpoint comes from the next day's first instantaneous file (the existing next_day_hour0 plumbing in the orchestrator).
Cell-center winds u, v (geographic frame) are filled only when the target grid differs from the source grid and fluxes must be reconstructed downstream. For native passthrough (source mesh == target mesh) u/v stay nothing and am/bm are written through directly after vertical merging.
cmfmc/dtrain are filled only when the source supports convection and the user has enabled it via settings.include_convection. vdiff carries optional Holtslag-Boville VDIFF source fields (u, v, t, qv) when the source can archive them into the transport binary for runtime diffusion.
AtmosTransport.Preprocessing.ReadyWindow Type
ReadyWindow{G, FT}(index, payload::NamedTuple)Typed wrapper for a topology-specific window payload that is ready to be verified and written. G is the target geometry, FT is the on-disk float type, and payload is the existing topology NamedTuple (m_cur/am/... for LL/CS, m_cur/hflux/... for RG).
Unknown property access is forwarded to payload, so existing verify_window!(window, contract, win_idx) methods can accept either a raw NamedTuple or a ReadyWindow.
AtmosTransport.Preprocessing.ReducedGaussianContract Type
ReducedGaussianContract{FT} <: AbstractWindowContract{ReducedGaussianTargetGeometry, FT}Typed nominal owning an RG preprocessor's per-window gate policy and worst-window positivity accumulator. Holds the face connectivity (face_left / face_right) so the per-window call site doesn't need to thread it through every call.
Construction validates replay_tol, positivity_cfl_limit ∈ (0, 1], steps_per_window ≥ 1, boundary_stub_tol ≥ 0, and length(face_left) == length(face_right).
boundary_stub_tol (default 0.0) is the absolute tolerance for the boundary-stub flux gate (verify_boundary_stub_flux_rg). Default is strict; tighten only with caution.
AtmosTransport.Preprocessing.SurfaceField Type
2D surface field (e.g. ps, pblh). No vertical reduction — identity passthrough.
AtmosTransport.Preprocessing.TracerMassField Type
Center-level extensive tracer mass (e.g. qv-mass). Sum native layers within each merged group.
AtmosTransport.Preprocessing.UnifiedPreprocessorDay Type
UnifiedPreprocessorDay(reader, workspace, contract, writer; context=nothing)Bundle the four typed axes a unified preprocessing day needs. context is an opaque topology/source adapter payload used by hook methods during migration.
AtmosTransport.Preprocessing.VerticalPlan Type
VerticalPlan{FT, T <: AbstractVerticalTransform}Result of plan_vertical. Type-parameterized by the originating transform so the apply_vertical! dispatch is statically resolved.
Fields:
transform: the originatingAbstractVerticalTransformvalue.native_vc: the input native hybrid coordinate.merged_vc: the output hybrid coordinate.merge_map:Vector{Int}such thatmerge_map[k_native]is the output-level index for merge-map flavors.Int[]forPressureOverlap(uses overlap coefficients instead).groups:Vector{UnitRange{Int}}of native center-level ranges that map to each output level (derived frommerge_mapfor the merge-map flavors).Nz_output: the output level count.Nz_native: the input level count (cached for cheap access).
AtmosTransport.MetDrivers.has_pbl_eflux Method
has_pbl_eflux(settings) -> Bool
has_cmfmc_cloud_base(settings) -> BoolWhether this source writes the surface latent heat flux (GCHP non-local PBL scheme) or the convective cloud-base layer (GCHP cloud convection).
sourceAtmosTransport.MetDrivers.has_surface Method
has_surface(settings::AbstractMetSettings) -> BoolWhether this source can populate raw PBL surface fields in RawWindow.surface. Sources gate actual reads behind their settings flags.
AtmosTransport.MetDrivers.has_vdiff_fields Method
has_vdiff_fields(settings::AbstractMetSettings) -> BoolWhether this source can populate RawWindow.vdiff with layer-center fields needed by the GCHP Holtslag-Boville vertical-diffusion data contract.
AtmosTransport.Preprocessing.TM5CleanupStats Method
TM5CleanupStats() -> NamedTuple of Ref{Int} countersDiagnostic counters bumped by ec2tm_from_rates!. Nothing-overhead when the function runs without stats (pass nothing); when stats are passed, each counter increments once per level/column that hit the corresponding cleanup branch.
columns_processed— total columns the function was called on.no_updraft— columns with no level satisfyingudmf > 0(after small-value clipping). entu/detu zeroed out.no_downdraft— columns with no level satisfyingddmf < 0.levels_udmf_clipped,levels_ddmf_clipped— levels where half-level mass flux magnitude was below 1e-6 kg/m²/s and got zeroed.levels_udrf_clipped,levels_ddrf_clipped— full-level detrainment rates zeroed (|rate| < 1e-10 kg/m³/s).levels_entu_neg,levels_detu_neg,levels_entd_neg,levels_detd_neg— levels where the indicated output went negative and got fixed via symmetric redistribution with its complementary rate.columns_closure_entu,columns_closure_entd— columns where the column closure moved a non-zero residualΣdet − Σent. For ERA5 input that residual is rounding (or a flux through the model top); seeec2tm_from_rates!.
Interpretation
On clean data we expect ~0% clipped levels and ~0 no-updraft columns (outside pure stratospheric columns). O(1%) redistribution firings are normal TM5 behaviour. O(50%) firings indicate a data pathology (wrong param IDs, wrong stream, bad units).
sourceAtmosTransport.Preprocessing.allocate_era5_n320_spectral_workspace Method
allocate_era5_n320_spectral_workspace(source_grid, T, Nz)Allocate a fresh workspace sized to source_grid (build via discover_era5_n320_source_grid), spectral truncation T (from discover_era5_spectral_truncation) and vertical level count Nz (callers default to ERA5_NATIVE_LEVEL_COUNT = 137, but the workspace itself stays generic).
AtmosTransport.Preprocessing.allocate_era5_n320_to_c180_pipeline Method
allocate_era5_n320_to_c180_pipeline(handles, target_grid;
Nz=ERA5_NATIVE_LEVEL_COUNT,
cache_dir=nothing,
include_convection=true)Build the full per-window pipeline for the ERA5 source described by handles (resolved via open_era5_day) and the C-tier target target_grid. Discovers the source mesh and spectral truncation from the day's core GRIB, loads the hybrid coordinate file declared in the settings, builds (or JLD2-loads from cache_dir) the conservative regridder, and allocates every B/C/D/E workspace.
Convection workspace allocation is gated on include_convection so the caller can opt out for a scalar-only smoke.
AtmosTransport.Preprocessing.allocate_merra2_to_c180_pipeline Method
allocate_merra2_to_c180_pipeline(target_grid; Nz, cache_dir, settings) -> MERRA2ToC180PipelineBuild (or JLD2-load from cache_dir) the MERRA-2 LL → CS conservative regridder and allocate every per-window buffer, including the physics panels settings requests. The source LL mesh is built with the TARGET mesh radius so the two manifolds match (build_regridder rejects a radius mismatch).
AtmosTransport.Preprocessing.allocate_merra2_window_physics Method
allocate_merra2_window_physics(settings, Nc, Nz, FT) -> NamedTupleOutput buffers for one window's physics sections (see merra2_window_physics!); only the requested ones are allocated.
AtmosTransport.Preprocessing.allocate_raw_window Function
allocate_raw_window(settings::AbstractMetSettings;
FT::Type, Nc=nothing, Nz=nothing) -> RawWindowAllocate a pre-zeroed RawWindow sized for one window of settings's source data. The orchestrator calls this once before the per-window loop and reuses the same buffer across all windows in a day. The shape parameters (Nc, Nz, …) are source-specific — concrete subtypes pick the keys they need.
AtmosTransport.Preprocessing.allocate_raw_window Method
allocate_raw_window(settings::GEOSSettings; FT, Nz) -> RawWindowPre-allocate a per-window workspace for the GEOS reader: 6 zero-filled panel arrays each for m, m_next, qv, qv_next, am, bm (shape (Nc, Nc, Nz)) and ps, ps_next (shape (Nc, Nc)).
When settings.include_convection, also allocates cmfmc (interfaces, shape (Nc, Nc, Nz + 1) per panel) and dtrain (centers, shape (Nc, Nc, Nz) per panel). Cross-topology winds (u, v) stay nothing here — they are produced by the orchestrator only when the target grid differs from the source.
AtmosTransport.Preprocessing.allocate_window_workspace Function
allocate_window_workspace(args...; kwargs...)Construct the topology-specific window workspace for one preprocessing day. Methods exist for the LL spectral, RG spectral, CS spectral, and GEOS CS paths.
sourceAtmosTransport.Preprocessing.apply_vertical! Function
apply_vertical!(buf_out, buf_in, plan::VerticalPlan, kind::AbstractFieldKind, args...)Apply the vertical transform to one window of buf_in, writing the result into buf_out. Dispatches on the (plan.transform, kind) combination:
Extensive center fields (
MassField,TracerMassField,MassFluxField,ConvectionTendencyField) sum native layers within each output group.Interface fields (
PressureFluxField,ConvectionInterfaceFlux) select the kept half-level interfaces.IntensiveCenterFieldtakes an additional positionalweightsargument (native mass-per-layer); produces the mass-weighted mean within each output group.SurfaceFieldis a passthrough copy (no vertical reduction).
buf_out and buf_in must be 3D arrays with the vertical axis on dim 3 (or 2D for SurfaceField).
AtmosTransport.Preprocessing.build_target_geometry Method
build_target_geometry(cfg_grid, FT=Float64) -> AbstractTargetGeometryDispatch configuration-driven target-geometry construction from the user-facing grid.type string.
AtmosTransport.Preprocessing.build_target_geometry Method
build_target_geometry(::Val{:cubed_sphere}, cfg_grid, FT)Build a cubed-sphere target geometry from the [grid] config section.
Required keys:
Nc :: Int— cells per panel edge
Optional keys:
definition—"equiangular_gnomonic"(synthetic) or"gmao_equal_distance"(GEOS-IT/GEOS-FP equal-distance gnomonic)panel_convention—"gnomonic"(default) or"geos_native". Ifdefinitionis omitted,"geos_native"selects"gmao_equal_distance"and"gnomonic"selects"equiangular_gnomonic".regridder_cache_dir— directory for CR.jl weight cache (defaultRegridding.default_regrid_cache_dir(), the cache the runtime uses too)staging_nlon,staging_nlat— override the internal LL staging grid size (defaults:max(4Nc, 360)×max(2Nc+1, 181))
AtmosTransport.Preprocessing.build_target_geometry Method
build_target_geometry(::Val{:era5_native_reduced_gaussian}, cfg_grid, FT)Build the reduced-Gaussian geometry metadata from a native ERA5 GRIB file. This is currently intended for geometry discovery and future native-grid preprocessing work rather than the active lat-lon synthesis path.
sourceAtmosTransport.Preprocessing.build_target_geometry Method
build_target_geometry(::Val{:latlon}, cfg_grid, FT) -> LatLonTargetGeometryBuild the regular lat-lon target geometry used by the current v4 spectral preprocessor.
sourceAtmosTransport.Preprocessing.build_target_geometry Method
build_target_geometry(::Val{:synthetic_reduced_gaussian}, cfg_grid, FT)Build a reduced-Gaussian target geometry from scratch without needing a source GRIB file. cfg_grid["gaussian_number"] controls the truncation N so the mesh has 2N Gauss-Legendre latitude rings. cfg_grid["nlon_mode"] selects the longitude layout:
"regular"(default): every ring has4Ncells — the classical "regular reduced" Gaussian grid used e.g. in the TL159/N80 family."octahedral": ECMWF octahedral layoutnlon = 4k + 16per hemisphere, mirrored between the two hemispheres withk = 1at the pole-adjacent ring.
Mesh latitudes are produced from FastGaussQuadrature.gausslegendre(2N), which returns ascending Gauss-Legendre nodes on (-1, 1). geometry_source_grib is left as an empty string to mark the grid as synthetic.
AtmosTransport.Preprocessing.close_day! Function
close_day!(ctx)Close all resources held by a day context. Must be idempotent; safe to call from a finally block.
AtmosTransport.Preprocessing.close_era5_day! Method
close_era5_day!(handles::ERA5GRIBDayHandles)Release the ecCodes index of the day's core GRIB file, if one was built. Idempotent — safe to call from a finally block.
AtmosTransport.Preprocessing.close_merra2_day! Method
close_merra2_day!(handles)Close every open NCDataset. Idempotent — safe to call from a finally.
AtmosTransport.Preprocessing.close_reader! Function
close_reader!(reader::AbstractMetReader) → nothingClose all per-day file handles and release scratch held by the reader. Idempotent — safe to call from a finally block. Concrete readers implement.
AtmosTransport.Preprocessing.close_streaming_binary! Function
close_streaming_binary!(writer)Close a streaming binary writer. Implementations should be idempotent.
sourceAtmosTransport.Preprocessing.coarsen_nested_cs_transport_binary Method
coarsen_nested_cs_transport_binary(input_path, output_path; target_Nc=30, ...)Experimentally restrict a nested cubed-sphere transport binary (for example C90 to C30) one met window at a time. Cell-extensive quantities and dkg are block-summed, coarse boundary faces are summed, intensive surface fields are area weighted, and a new adaptive per-window transport schedule is selected.
This is a testing-only operator coarsener, not yet a validated substitute for direct preprocessing at target_Nc. The output header records that caveat. Only the payload used by the ERA5 N320 C90 L66 no-convection campaign is currently accepted; unfamiliar physics sections are rejected explicitly.
AtmosTransport.Preprocessing.contract_cfl_limit Function
contract_cfl_limit(contract::AbstractWindowContract) -> Float64Per-substep positivity CFL gate the contract enforces.
sourceAtmosTransport.Preprocessing.contract_replay_tolerance Function
contract_replay_tolerance(contract::AbstractWindowContract) -> Float64Relative replay tolerance the contract's verify_window! uses. Concrete contracts return their stored policy value.
AtmosTransport.Preprocessing.contract_require_positivity Function
contract_require_positivity(contract::AbstractWindowContract) -> BoolWhether summarize_status! errors (true) or warns (false) on a positivity violation. Closes-the-escape-hatch toggle from CS round-2.
AtmosTransport.Preprocessing.convective_cloud_base! Method
convective_cloud_base!(cloud_base, dqrcu) -> cloud_baseGCHP's convective cloud base (convection_mod.F90, DO_RAS_CLOUD_CONVECTION): the lowest layer with convective rain production DQRCU > 0, or the surface layer when there is none. Levels are top-down, so this is the largest such k, stored as a float layer index.
AtmosTransport.Preprocessing.derive_c180_dry_mass! Method
derive_c180_dry_mass!(m_dry, delp_dry, ps_dry, ps_dry_acc,
ps_panels, qv_panels, vc, cell_areas; grav=STANDARD_GRAVITY) -> nothingCubed-sphere variant of derive_n320_dry_mass!. Builds the dry-air layer mass, dry pressure thickness, and dry surface pressure for each of the 6 C-tier panels from the regridded moist PS + Q on C180. Same formula as the GEOS endpoint dry-mass derivation so Σ_k DELP_dry = PS_dry to roundoff on the target mesh too.
Inputs and outputs are NTuple{6, ...} panel tuples; ps_dry_acc is a panel-tuple Float64 accumulator (so a Float32 output preserves the multi-level summation precision). cell_areas[i, j] is shared across all 6 panels (the CS mesh is isotropic per panel).
AtmosTransport.Preprocessing.derive_c180_tm5_convection! Method
derive_c180_tm5_convection!(tm5_fields, conv_fields, window_fields, vc, scratches;
basis, stats=nothing) -> tm5_fieldsTM5 entrainment and detrainment on every target column from the raw ECMWF diagnostics already mapped to the target (conv_fields), following TM5's tmm_Read_Convec_EC: UDMF/DDMF level k becomes interface k (the top of layer k) and the ground interface Nz + 1 is zero, layer thickness from the target T, q and ps (dz_hydrostatic_virtual!), the fluxes put on the air-mass basis of the binary with ecmwf_convection_to_basis! (DryBasis() multiplies them by 1 − q), then ec2tm_from_rates!.
AtmosTransport.Preprocessing.derive_n320_dry_mass! Method
derive_n320_dry_mass!(dry, window, vc, cell_areas; grav=STANDARD_GRAVITY) -> dryReconstruct dry-air mass per layer, dry pressure thickness, and dry surface pressure from a populated window::ERA5N320WindowFields (moist PS, Q) using the hybrid coordinate vc (length Nz+1 A and B arrays in Pa and 1 respectively, top-down) and the per-cell areas (m²). Matches the GEOS endpoint_dry_mass! formula so the runtime sees a nominally bit-identical dry-basis contract regardless of source.
Asserts shapes and length(vc.A) == length(vc.B) == Nz + 1 so a coefficient table that does not match the workspace Nz fails immediately with a clear DimensionMismatch.
AtmosTransport.Preprocessing.detect_level_orientation Method
detect_level_orientation(ctm_a1::NCDataset) -> SymbolReturn :bottom_up if k=1 is the surface (mass-thicker) and :top_down if k=1 is TOA. Heuristic is unambiguous: surface DELP is O(1000 Pa), TOA DELP is O(1 Pa).
AtmosTransport.Preprocessing.detect_merra2_level_order Method
detect_merra2_level_order(inst3, path) -> TopDown() or SurfaceFirst()Level order of an inst3/I3 file from its first-slice QV: the global-mean specific humidity is ≈7e-3 kg/kg in the surface layer and ≈2e-6 at the top, so the moist end is the surface. Errors unless one end is >100× the other. Neither archive carries DELP or pressure metadata to decide from.
sourceAtmosTransport.Preprocessing.discover_era5_n320_source_grid Method
discover_era5_n320_source_grid(core_path; FT=Float64) -> ReducedGaussianTargetGeometryBuild the N320 source-grid descriptor from the first gridType=reduced_gg message in core_path (the q field is always present). The resulting mesh ring order is south→north — the GRIB stores rings north→south and the read_era5_reduced_gaussian_* helpers flip into the project convention.
AtmosTransport.Preprocessing.discover_era5_spectral_truncation Method
discover_era5_spectral_truncation(core_path) -> IntRead the first spectral (gridType=sh) message in core_path and return its triangular truncation J = K = M. ERA5 native model-level analyses are T639 in the current archive; the helper avoids hard-coding that value in case a future archive convention shifts.
AtmosTransport.Preprocessing.drain_ready_windows! Function
drain_ready_windows!(workspace) -> iterator of `ReadyWindow`Return the windows that became write-ready after the last ingest.
sourceAtmosTransport.Preprocessing.driver_after_write_window! Method
driver_after_write_window!(workspace, reader, ready, context)Post-write migration hook. GEOS-native CS uses this to advance its chained endpoint state; most topologies do nothing.
sourceAtmosTransport.Preprocessing.driver_drain_ready_windows! Method
driver_drain_ready_windows!(workspace, contract, win, context)Return ready windows produced by the most recent ingest. A hook may return a single ReadyWindow, a single PreverifiedWindow, or any iterator of either shape.
AtmosTransport.Preprocessing.driver_flush_final_windows! Method
driver_flush_final_windows!(workspace, reader, contract, context)Return final ready windows after all source windows have been ingested.
sourceAtmosTransport.Preprocessing.driver_ingest_window! Method
driver_ingest_window!(workspace, reader, win, context)Migration hook for ingesting one source window into the target workspace. Topology/source adapters may override while legacy workspace signatures are being collapsed.
sourceAtmosTransport.Preprocessing.driver_windows_per_day Method
driver_windows_per_day(reader, context) -> IntMigration hook for source readers whose window count needs adapter context.
sourceAtmosTransport.Preprocessing.dz_hydrostatic_constT! Method
dz_hydrostatic_constT!(dz, ps, ak, bk, Nz; T_ref=HYDROSTATIC_REFERENCE_TEMPERATURE) -> dzConstant-temperature hydrostatic layer thickness — fallback for use when T and Q are unavailable. Matches main's Julia port's shortcut (T_ref = HYDROSTATIC_REFERENCE_TEMPERATURE = 260 K). Biases entu/detu magnitudes by ~10-20% vs the virtual-temperature version; dz_hydrostatic_virtual! is preferred when T + Q are downloaded together with the convection fields.
AtmosTransport.Preprocessing.dz_hydrostatic_virtual! Method
dz_hydrostatic_virtual!(dz, T_col, Q_col, ps, ak, bk, Nz) -> dzCompute layer thickness dz[1:Nz] (m) at layer centers from a single column's temperature T_col[1:Nz] (K) and specific humidity Q_col[1:Nz] (kg/kg), plus surface pressure ps (Pa) and the hybrid-sigma coefficients ak, bk (length Nz+1).
Uses the hydrostatic approximation with virtual temperature:
p_top[k] = ak[k] + bk[k] * ps (Pa, higher-altitude side)
p_bot[k] = ak[k+1] + bk[k+1] * ps (Pa, lower-altitude side)
dp[k] = p_bot[k] - p_top[k] (> 0 in AtmosTransport orientation)
p_mid[k] = 0.5 · (p_top[k] + p_bot[k])
T_v[k] = T_col[k] · (1 + 0.61 · Q_col[k])
dz[k] = R · T_v[k] / g · dp[k] / p_mid[k]Orientation: AtmosTransport (k=1=TOA, k=Nz=surface). ak/bk are the full (Nz+1)-length ERA5 L137 hybrid coefficients.
For the TOA half-level we fall back to an ordinary scale-height estimate (T_v=T_top, p_mid=p_bot) when p_top → 0. In practice the top-level dz is never in the convection window so the approximation is irrelevant; it's a guard against divide-by-zero.
AtmosTransport.Preprocessing.ec2tm! Method
ec2tm!(entu, detu, entd, detd,
mflu_ec, mfld_ec, detu_ec, detd_ec) -> (entu, detu, entd, detd)Convert ECMWF convective mass-flux fields into TM5 (entu, detu, entd, detd) layer-center fields. All inputs / outputs in AtmosTransport orientation (k=1=TOA, k=Nz=surface). Operates in place on the four output arrays (no allocations).
Shapes:
entu, detu, entd, detd:(..., Nz)— layer centers.mflu_ec, mfld_ec:(..., Nz+1)— half levels (interfaces). Interfacekis the TOP of layerk; interfaceNz+1is the surface boundary.detu_ec, detd_ec:(..., Nz)— layer centers.
The leading dimensions are arbitrary (scalar, (Nx, Ny), (ncells,), or per-panel (Nc, Nc)) as long as the arrays all have consistent shape. This function is backend-agnostic pure Julia; call from the preprocessor (CPU) ahead of binary writeout.
Negative small values (<= 0) in detu_ec / detd_ec are clipped to zero, following the ECMWF-rounding-artifact clean-up documented in phys_convec_ec2tm.F90 (ECMWF diagnostics can produce ~-1e-19 values from rounding).
For every location it computes, in AtmosTransport orientation (k=1=TOA):
Layer k has interface
kat its TOP (higher altitude side) and interfacek+1at its BOTTOM (lower altitude side).Updraft flows upward: from interface
k+1into layerkvia entrainmententu[k], out of layerkvia interfacek. Continuity:mflu_out − mflu_in + detu − entu = 0⟹entu[k] = detu[k] + mflu_ec[k] − mflu_ec[k+1]wheremflu_ec[k]is the flux through interface k (above layer k) andmflu_ec[k+1]is the flux through interface k+1 (below layer k). Convention:mflu_ec >= 0(positive upward).Downdraft flows downward: from interface
kinto layerkvia entrainmententd[k], out through interfacek+1via detrainment. Sign convention in ECMWF:mfdo <= 0(negative). We definemfdo_abs = -mfdo_ec >= 0, thenentd[k] = detd[k] + mfdo_abs[k+1] − mfdo_abs[k].
Boundary conditions:
mflu_ec[1] = 0(no updraft above TOA).mflu_ec[Nz+1] = 0(no updraft into ground from below — the surface is the sink).mfld_ec[1] = 0(no downdraft above TOA).mfld_ec[Nz+1] = 0(no downdraft escapes the surface).
These boundaries are typically already zero in ECMWF output; we enforce explicitly by construction below.
sourceAtmosTransport.Preprocessing.ec2tm_from_rates! Method
ec2tm_from_rates!(entu, detu, entd, detd,
udmf, ddmf, udrf_rate, ddrf_rate,
dz, Nz; stats=nothing) -> nothingColumn-level port of TM5's ECconv_to_TMconv (see deps/tm5/base/src/phys_convec_ec2tm.F90:87-237). Fills the output arrays entu, detu, entd, detd (kg/m²/s at layer centers) in place from raw ERA5 physics inputs. All arrays use AtmosTransport orientation (k=1=TOA, k=Nz=surface).
Inputs
udmf::AbstractVector{FT}, lengthNz+1— updraft mass flux at half levels (kg/m²/s). Half levelkis the interface at the TOP of layerk;udmf[Nz+1]is the ground and must be zero (after clipping; anArgumentErrorotherwise). ERA5 archives UDMF levelk(1 = model top) as the flux through the top of layerk, so callers pass[GRIB levels 1..Nz; 0], as TM5'stmm.F90does (udmf_gg(:,0:nlev-1) = buf; udmf_gg(:,nlev) = 0). Must be ≥ 0.ddmf::AbstractVector{FT}, lengthNz+1— downdraft mass flux at half levels, ECMWF convention (≤ 0), packed likeudmf.udrf_rate::AbstractVector{FT}, lengthNz— updraft detrainment RATE at layer centers (kg/m³/s).ddrf_rate::AbstractVector{FT}, lengthNz— downdraft detrainment RATE at layer centers (kg/m³/s).dz::AbstractVector{FT}, lengthNz— layer thickness (m), positive. Compute fromdz_hydrostatic_virtual!.Nz::Int— number of full levels.stats::Union{Nothing, NamedTuple}— cleanup-stats counters fromTM5CleanupStats(). Whennothing, no counter work.
Algorithm (line-for-line match to F90 lines 132-236)
Copy-with-clipping (F90 L132-144).
|udmf| < 1e-6 → 0,|ddmf| < 1e-6 → 0(applied asddmf > -1e-6 → 0),udrf_rate < 1e-10 → 0,ddrf_rate < 1e-10 → 0.dz integration (F90 L146-151):
detu = udrf × dz(kg/m³/s → kg/m²/s). Same for detd.Cloud-top search (F90 L153-173,
do l = 1, lm; if (mflu(l) > 0)): the cloud-top layeruptopis the highest layer whose BOTTOM interface carries updraft, i.e.uptop = k - 1for the first interfacek ∈ 2:Nz+1withudmf[k] > 0;dotoplikewise withddmf[k] < 0. The model-top interface 1 is not searched, as in TM5. If none, zero out everything in that direction.Layer budgets (F90 L175-212): for each active direction,
entu[k] = udmf[k] - udmf[k+1] + detu[k](out through the top, minus in through the bottom, plus detrainment) fork = uptop:Nz; layers aboveuptopstay zero.Symmetric negative redistribution (F90 L214-232): if
entu[k] < 0, add-entu[k]todetu[k]and zeroentu[k]. Same fordetu<0(adds to entu), and the same two for downdraft.Column closure (TM5
meteo.F90:2023-2052): the residualΣdetu − Σentuis added to the layer of largest entrainment (on the detrainment side when it is negative, so nothing turns negative).
Mass conservation
In every layer entu[k] - detu[k] = udmf[k] - udmf[k+1], the flux convergence (step 4; the redistribution of step 5 keeps entu - detu). Summed over the column this telescopes to udmf[uptop] - udmf[Nz+1] = 0: no flux crosses the top of the cloud-top layer or the ground, so the TM5 fields reproduce the input fluxes at every interface and the closure of step 6 moves only rounding (or a flux through the model top, which TM5 does not exclude either).
AtmosTransport.Preprocessing.ecmwf_convection_to_basis! Method
ecmwf_convection_to_basis!(basis, udmf, ddmf, udrf, ddrf, q) -> nothingPut one column of ECMWF convective fields (top-first, Nz = length(q) layers) on the air-mass basis of the transport binary before ec2tm_from_rates! turns them into TM5 entrainment and detrainment.
udmf,ddmf(lengthNz + 1): updraft and downdraft mass fluxes at the interfaces, kg m⁻² s⁻¹; interfacekis the top of layerk.udrf,ddrf(lengthNz): updraft and downdraft detrainment rates at the layer centres, kg m⁻³ s⁻¹.q(lengthNz): specific humidity at the layer centres, kg kg⁻¹.
ECMWF fluxes are per unit of total (moist) air. For DryBasis() they become dry-air fluxes,
for the interface fluxes M (udmf, ddmf) and the centre rates D (udrf, ddrf), with the interface humidity q̂_k of interface_specific_humidity, so the TM5 rates entu / m_dry equal TM5's own entu / m_moist. Converting the inputs keeps the ec2tm budget entu − detu = udmf[k] − udmf[k+1] exact. MoistBasis() leaves the fields unchanged. ec2tm's noise thresholds then act on the converted values.
AtmosTransport.Preprocessing.end_of_day_seed Function
end_of_day_seed(reader::AbstractMetReader) → seed_or_nothingReturn the seed value to thread into the next day's open_reader(..., seed = ...). Type-system guarantees:
end_of_day_seed(::AbstractMetReader{FT, S, NoChain})returnsnothing(statically inferred).end_of_day_seed(::AbstractMetReader{FT, S, ChainedMass{T}})returns a value of typeT(or throws if the reader has not yet produced the end-of-day endpoint).
Closes foot-gun (D).
sourceAtmosTransport.Preprocessing.endpoint_dry_mass! Method
endpoint_dry_mass!(delp_dry, ps_dry, ps_total, qv, vc) -> (delp_dry, ps_dry)Reconstruct dry DELP and dry PS at one endpoint hour from the moist PS and QV provided by GEOS, using the hybrid coordinate vc. The output is on top-down level convention (k=1 = TOA).
Algorithm:
DELP_full[k] = ΔA[k] + ΔB[k] * PS_total
DELP_dry[k] = (1 - QV[k]) * DELP_full[k]
PS_dry = Σ DELP_dry[k]In-place: writes into delp_dry and ps_dry.
AtmosTransport.Preprocessing.endpoint_dry_mass Method
Allocate (delp_dry, ps_dry) for one panel and run endpoint_dry_mass!.
AtmosTransport.Preprocessing.era5_convection_hour_address Method
era5_convection_hour_address(hour) -> (use_prev_day, data_time_hhmm, step_range)Map a UTC hour h ∈ 0..23 to the GRIB header tuple that addresses the matching ECMWF convection forecast sample (see read_era5_n320_convection_window! docstring for the cycle layout). Returns (::Bool, ::Int, ::String).
AtmosTransport.Preprocessing.era5_grib_path Method
era5_grib_path(settings, date, stream) -> StringResolve the on-disk GRIB path for stream on date. stream must be one of :core, :convection, or :surface. Existence is not checked here — the caller (typically open_era5_day) decides whether a missing file is fatal or merely "no next-day endpoint available".
AtmosTransport.Preprocessing.flush_final_windows! Function
flush_final_windows!(workspace, args...; kwargs...)Emit any final cross-day/zero-tendency windows once the source stream is exhausted.
sourceAtmosTransport.Preprocessing.flux_fidelity_header! Method
flux_fidelity_header!(header, acc) -> headerReplace every certificate key of header (FLUX_CERTIFICATE_KEY_PREFIXES) by the aggregate of acc. Consistency is enforced at write time: one kind of flux evidence for all windows, sample counts that cover nwindow windows and nlevel levels when the header has them, OMEGA evidence for all windows or none, and column-closure records for all windows or none. A failed aggregate OMEGA gate writes its failed certificate and then throws, so the staged binary is quarantined.
AtmosTransport.Preprocessing.geos_collection_path Method
geos_collection_path(settings::GEOSITSettings, date::Date, collection) -> StringResolve the on-disk path of one GEOS-IT collection for date. Searches a flat root_dir and the per-day root_dir/YYYYMMDD/ layout.
AtmosTransport.Preprocessing.has_convection Method
has_convection(settings::AbstractMetSettings) -> BoolWhether this source can populate RawWindow.cmfmc and RawWindow.dtrain. Defaults to false; sources that support convection override and gate the actual output behind a user flag (e.g. settings.include_convection).
AtmosTransport.Preprocessing.ingest_window! Function
ingest_window!(workspace, args...; kwargs...) -> nothingConsume one source/met window into the topology workspace. Ready windows are exposed by drain_ready_windows!.
AtmosTransport.Preprocessing.init_cs_positivity_accumulator Method
init_cs_positivity_accumulator() -> NamedTupleZero-valued state for accumulating the worst per-window positivity diagnostic across a preprocessing loop. Pair with update_cs_positivity_accumulator!.
AtmosTransport.Preprocessing.init_ll_positivity_accumulator Method
init_ll_positivity_accumulator() -> NamedTupleZero-valued state for accumulating the worst per-window LL positivity diagnostic across a preprocessing loop. Pair with update_ll_positivity_accumulator.
AtmosTransport.Preprocessing.init_rg_positivity_accumulator Method
init_rg_positivity_accumulator() -> NamedTupleZero-valued state for accumulating the worst per-window RG positivity diagnostic across a preprocessing loop.
sourceAtmosTransport.Preprocessing.interface_specific_humidity Method
interface_specific_humidity(k, Nz, q...) -> q̂_k
interface_specific_humidity(k, Nz, i, j, q...) -> q̂_kSpecific humidity at interface k of a top-first column of Nz layers (interface k is the top of layer k; k = 1 is the model top and k = Nz + 1 the surface), the mean of the adjacent layer centres:
where q_k is the value at the centre of layer k and q̂_k the value at interface k.
Each q is a column of layer-centre values indexed by level; with several columns (time samples, e.g. the start and end of a window) the result is also averaged over them, in the order q[1][k-1], q[1][k], q[2][k-1], ..., which is the four-corner mean of GEOS CMFMC (_moist_to_dry_cmfmc!).
The (i, j, q...) method applies the identical rule directly to columns of three-dimensional arrays without constructing a view in the GEOS grid loop.
AtmosTransport.Preprocessing.load_hybrid_coefficients Method
load_hybrid_coefficients(coeff_path::String) -> HybridSigmaPressureLoad all hybrid sigma-pressure interface coefficients from a TOML file. Unlike load_era5_vertical_coordinate, this does not slice — useful for sources whose level count comes from the file rather than a config knob (e.g. GEOS-72, MERRA-2, native ERA5 L137 without sub-tropo selection).
AtmosTransport.Preprocessing.load_met_settings Method
load_met_settings(toml_path::String; root_dir, kwargs...) -> AbstractMetSettingsConstruct a typed met-source descriptor from toml_path. The TOML's [source].name key picks the concrete settings type, and per-type _build_met_settings methods consume the source-specific TOML tables.
root_dir is the on-disk directory holding the source's daily files. Additional kwargs are forwarded to the constructor and override any values supplied by the TOML.
AtmosTransport.Preprocessing.mass_basis_from_symbol Method
mass_basis_from_symbol(s::Symbol) -> AbstractMassBasisConstruct the matching basis singleton from a header Symbol. Throws ArgumentError for unknown values.
AtmosTransport.Preprocessing.mass_basis_symbol Method
mass_basis_symbol(::AbstractMassBasis) -> SymbolMap a basis singleton (State.DryBasis / State.MoistBasis) to the on-disk binary-header value (:dry / :moist). Inverse of mass_basis_from_symbol.
AtmosTransport.Preprocessing.merra2_archive Method
merra2_archive(name) -> MERRA2ArchiveThe archive for the [preprocessing] layout key: "nasa" or "geoschem".
AtmosTransport.Preprocessing.merra2_path Method
merra2_path(settings, date, collection) -> StringResolve the on-disk path for one MERRA-2 collection on date:
NASAArchive(:inst3,:tavg3):{root_dir}/{dir}/{YYYY}/{MM}/MERRA2_{stream}.{name}.{YYYYMMDD}.nc4GEOSChemArchive(:inst3→ I3,:tavg3→ A3dyn,:a3mste,:a3mstc,:a1):{root_dir}/{YYYY}/{MM}/MERRA2.{YYYYMMDD}.{tag}.05x0625.nc4
Existence is not checked here — open_day decides whether a missing file is fatal or merely "no next-day endpoint available".
AtmosTransport.Preprocessing.merra2_stream_code Method
merra2_stream_code(date) -> StringMERRA-2 production stream code for date (same year→stream map the download path uses): 100 (1980-91), 200 (92-00), 300 (01-10), 400 (2011 onward).
AtmosTransport.Preprocessing.merra2_window_physics! Method
merra2_window_physics!(out, settings, cur, nxt, h, nsub)
-> (; surface, cmfmc, dtrain, vdiff, cmfmc_cloud_base)Physics sections of output window h of the nsub equal windows that split the 3-hour block held in cur (nxt holds the block's right-endpoint QV and T). Sections not requested are nothing.
The window's start and end QV, and the VDIFF temperature, are linear in time between the block endpoints (
curexactly whennsub == 1).CMFMC and DTRAIN (3-hour averages) are copied into
outand converted to dry air there with the window's start and end QV, socur.physstays moist and can serve every window of the block.The PBL surface fields (and EFLUX) average the A1 hours inside the window.
The convective cloud base is the block's (from 3-hour DQRCU).
AtmosTransport.Preprocessing.n320_cell_areas Method
n320_cell_areas(source_grid) -> Vector{Float64}Materialise per-cell areas (m²) for the N320 source mesh. Always returns Vector{Float64} regardless of source_grid's element type — downstream dry-mass arithmetic runs in Float64 for precision and the per-cell mesh quadrature is itself Float64. Cached by callers that derive dry mass for many windows of the same day.
AtmosTransport.Preprocessing.native_vertical Function
native_vertical(reader::AbstractMetReader) → HybridSigmaPressure{FT_v}Native vertical coordinate of the source data (NOT the merged/output coordinate). The vertical-transform axis (P0b) consumes this to plan its mapping.
sourceAtmosTransport.Preprocessing.open_day Function
open_day(settings::AbstractMetSettings, date::Date; ...) -> ctxOpen per-day source-specific context (file handles, caches, vertical coefficients, …). The orchestrator calls this once per day at the start of process_day, threads ctx through every read_window! call, and calls close_day!(ctx) in a finally block.
ctx is opaque to the orchestrator — only the source knows its layout.
AtmosTransport.Preprocessing.open_day Method
open_day(settings::GEOSSettings, date::Date; next_day_handle=true) -> GEOSDayHandlesCanonical-contract alias for open_geos_day. The orchestrator calls this once per day and threads the returned handles through every per-window read_window!.
AtmosTransport.Preprocessing.open_era5_day Method
open_era5_day(settings, date; next_day_handle=true) -> ERA5GRIBDayHandlesResolve the GRIB stream paths for date and assert that today's required files are on disk. When next_day_handle=true and <date+1> has a core GRIB available, records its path so the last window's right endpoint can be read from the next day's hour-0 fields.
Errors are explicit about which file is missing so a misconfigured root_dir or an incomplete download is immediately visible.
AtmosTransport.Preprocessing.open_era5_humidity_remap Method
open_era5_humidity_remap(root_dir, date, target_mesh; Nz=ERA5_NATIVE_LEVEL_COUNT,
cache_dir=nothing) -> ERA5HumidityRemapPrepare to read the ERA5 N320 specific humidity of date from the native GRIB archive under root_dir (ml_an_native_core/era5_core_YYYYMMDD.grib) and remap it conservatively, with the coverage normalisation of _regrid_intensive!, to target_mesh. The regridder weights are cached in cache_dir.
AtmosTransport.Preprocessing.open_geos_day Method
open_geos_day(settings, date; next_day_handle=true, adjacent_omega=false)
-> GEOSDayHandlesOpen per-collection NCDataset handles for date. When next_day_handle is true and the next-day CTM_I1 file exists, opens it too so the last window of date has its right endpoint available.
adjacent_omega (default false) opens the prev/next-day A3dyn (OMEGA) + I3 (QV) handles that the :omega_consistent cm closure needs to PCHIP-interpolate across midnight. Every other closure (incl. the validated :endpoint_balanced default and the GCHP VDIFF path) never reads them, so the default leaves all four handles nothing and opens nothing extra.
AtmosTransport.Preprocessing.open_merra2_day Method
open_merra2_day(settings, date; next_day_handle=true) -> MERRA2DayHandlesOpen the day's inst3 (and tavg3, when winds_collection==:tavg3) datasets, asserting both are on disk. When next_day_handle=true and the next-day inst3 file exists, opens it for the final-window endpoint look-ahead.
AtmosTransport.Preprocessing.open_reader Function
open_reader(settings::AbstractMetSettings, date::Date, ::Type{FT};
seed = nothing, next_day_handle::Bool = true)
→ reader::AbstractMetReader{FT, typeof(settings), CP}Construct a typed met-reader for one calendar day. The reader owns the underlying day-handle context (file handles, vertical coefficients, chained-mass seed) and exposes the per-window trait surface.
seed carries cross-day mass state for chained-mass readers; pass the return value of end_of_day_seed(prev_day_reader). For NoChain readers, seed must be nothing.
next_day_handle controls whether the reader opens a handle into the next day's first-hour instantaneous file. Required when the day produces endpoint mass for the last window (the standard case for hourly sources).
Dispatch by settings type to select the concrete reader; concrete readers register their own method.
AtmosTransport.Preprocessing.open_reader Method
open_reader(settings::ERA5SpectralSettings, date::Date, ::Type{FT};
seed = nothing, next_day_handle::Bool = true)Opens the spectral day's GRIB and caches the typed nominal. seed/next_day_handle accepted for signature parity with the GEOS constructor; the spectral path's "next-day endpoint" is handled by the existing next_day_hour0 helper in configuration.jl:349.
AtmosTransport.Preprocessing.plan_vertical Function
plan_vertical(transform::AbstractVerticalTransform,
native_vc::HybridSigmaPressure{FT}) → VerticalPlan{FT, typeof(transform)}Materialize the planned vertical-coordinate mapping for one day's run. Called once per day (or once per run if native_vc is invariant); the returned plan is reused across all windows.
AtmosTransport.Preprocessing.process_day Method
process_day(date::Date, grid::AbstractTargetGeometry, settings, vertical; next_day_hour0=nothing)Topology-specific daily transport-binary preprocessor extension point.
Concrete target geometries implement this method with ordinary Julia multiple dispatch:
LatLonTargetGeometrywrites structured directional LL binaries.ReducedGaussianTargetGeometrywrites face-indexed RG binaries.CubedSphereTargetGeometrywrites panel-local CS binaries.
Every implementation must satisfy the same transport contract:
use explicit forward endpoint mass targets for every window, including the final cross-day window when
next_day_hour0is available;write declared payload semantics, including
delta_semantics;run a write-time replay check unless explicitly disabled for diagnostics;
produce binaries that the runtime driver can load with replay validation.
This fallback rejects unsupported source/target pairs after config parsing has already produced an AbstractTargetGeometry.
AtmosTransport.Preprocessing.process_day Method
process_day(date, grid::CubedSphereTargetGeometry, settings::AbstractERA5GRIBSettings,
vertical; out_path, FT, mass_basis, dt_met_seconds, positivity_cfl_limit,
kwargs...)Adapter that the unified preprocessor CLI calls into. Forwards to process_era5_n320_to_cs_day the substep policy (min_steps_per_window as the floor, adaptive_substeps, substep_cfl_target, max_steps_per_window), the positivity settings, write_replay_check, and the global dry-mass pin. The remaining unified-CLI day-kwargs (e.g. chain_mass, seed_m) are absorbed by the trailing kwargs... and ignored — ERA5 N320 has no day-to-day mass-chain state. The removed horizontal_balance writer keyword is rejected rather than silently absorbed.
Returns (; final_m = nothing) so the unified CLI's seed_m = get(result, :final_m, nothing) chain remains a no-op.
AtmosTransport.Preprocessing.process_day Method
process_day(date, grid::CubedSphereTargetGeometry,
settings::AbstractGEOSSettings, vertical;
out_path,
dt_met_seconds = GEOS_MET_INTERVAL_SECONDS,
FT = Float64,
mass_basis = :dry,
replay_tol = replay_tolerance(FT),
write_replay_check = true,
seed_m = nothing,
next_day_hour0 = nothing,
chain_mass = true) -> NamedTupleBuild a v4 cubed-sphere transport binary at out_path from one UTC day of native GEOS data. The target mesh is either the native mesh (CS passthrough) or a nested block coarsening of it (native Nc an integer multiple of the target Nc); coarsening sums masses and face fluxes over each block and area-weights the physics fields.
Stored mass targets the raw GEOS dry endpoint (DELP_dry) transformed to the output vertical grid. With cm_closure = :endpoint_balanced, the horizontal fluxes are column-balanced to that endpoint, then cm is diagnosed so the replay and positivity contracts are checked against the same endpoint the runtime will see.
For multi-day preprocessing with chain_mass = true, seed_m carries the raw endpoint from the previous day so adjacent daily binaries share a boundary mass: pass nothing (default) on day 1 to seed from raw GEOS DELP_dry, and on day N+1 pass the final_mreturned by day N'sprocess_day. Withchain_mass = false,seed_m is ignored and every window reinitializes from raw GEOS mass.
When chain_mass = true, the returned NamedTuple includes final_m::NTuple{6, Array{FT, 3}}, the raw-endpoint state at the END of the last window. With chain_mass = false, final_m is nothing.
next_day_hour0 is part of the inherited topology-dispatch contract but unused — the GEOS reader handles next-day endpoints internally via next_ctm_i1.
AtmosTransport.Preprocessing.process_day Method
process_day(date, grid::CubedSphereTargetGeometry, settings, vertical; ...)Spectral→CS transport binary: spectral synthesis to an internal LL staging grid, conservative regridding to CS panels, endpoint continuity closure, and streaming binary write. No on-disk LL intermediate.
sourceAtmosTransport.Preprocessing.process_day Method
process_day(date, grid::CubedSphereTargetGeometry, settings::MERRA2Settings,
vertical; out_path, mass_basis, dt_met_seconds, …)Adapter that the unified preprocessor CLI calls into. Forwards to process_merra2_to_cs_day with the kwargs the underlying function accepts; the rest of the unified-CLI day-kwargs (e.g. chain_mass, seed_m, cm_closure) are absorbed by the trailing kwargs... and ignored — MERRA-2 has no day-to-day mass-chain state. The removed horizontal_balance writer keyword is rejected rather than silently absorbed.
Returns (; final_m = nothing, global_mass_target_kg) so the unified CLI's seed_m/global_mass_target_kg chain remains a no-op.
AtmosTransport.Preprocessing.process_day Method
process_day(date, grid::LatLonTargetGeometry, settings, vertical;
next_day_hour0=nothing, positivity_cfl_limit=DEFAULT_POSITIVITY_CFL_LIMIT,
require_substep_positivity=true, write_replay_check=true)Run the full one-day preprocessing workflow for the structured lat-lon target: read spectral input, process all windows, close continuity against forward mass endpoints, and write the final binary.
sourceAtmosTransport.Preprocessing.process_day Method
process_day(date, grid::ReducedGaussianTargetGeometry, settings, vertical;
next_day_hour0=nothing, positivity_cfl_limit=DEFAULT_POSITIVITY_CFL_LIMIT,
require_substep_positivity=true, write_replay_check=true)Streaming one-day preprocessing for reduced-Gaussian targets.
Uses a 2-window sliding buffer: at any time only two windows' worth of (m, hflux, cm, ps) are held in memory. Each window is Poisson-balanced and written to disk before the next pair is computed. This reduces peak memory from O(Nt) to O(1) and enables O160/O320 binary generation.
Pipeline per window: spectral synthesis → mass fix → level merge → (wait for next window) → column Poisson balance using (m_cur, m_next) → cm recomputation → stream-write
sourceAtmosTransport.Preprocessing.process_day Method
process_day(cfg::Dict; day_override=nothing, start_date=nothing, end_date=nothing)Top-level TOML-driven preprocessor entry. Detects source type from cfg:
[source].toml = "config/met_sources/<source>.toml"→ typedAbstractMetSettingspath, supports cross-day state carry (e.g. GEOS pressure-fixer chained mass).otherwise → typed ERA5 spectral config path (
[input].spectral_dir).
Both paths accept a single day_override or a start_date/end_date range.
Both paths converge on process_day(date, grid::AbstractTargetGeometry, settings, vertical; ...) for the per-day work. There is no parallel GEOS-only or per-source CLI — new sources plug in via AbstractMetSettings + load_met_settings.
AtmosTransport.Preprocessing.process_era5_n320_to_cs_day Method
process_era5_n320_to_cs_day(date, settings, target_grid;
out_path,
Nz = ERA5_NATIVE_LEVEL_COUNT,
vertical_plan = nothing,
mass_basis = :dry,
dt_met_seconds = ERA5_MET_INTERVAL_SECONDS,
steps_per_window = 8,
adaptive_substeps = true,
substep_cfl_target = ERA5_N320_SUBSTEP_CFL_TARGET,
max_steps_per_window = typemax(Int),
cs_balance_tol = DEFAULT_POISSON_BALANCE_TOL,
cs_balance_project_every = DEFAULT_POISSON_PROJECT_EVERY,
positivity_cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
require_substep_positivity = true,
write_replay_check = true,
cache_dir = nothing,
include_convection = false,
global_mass_pin = false,
global_mass_target_kg = NaN,
mass_fix_target_ps_dry_pa = NaN)The float type is the target grid's (target_grid::CubedSphereTargetGeometry{FT}).
Generate a v4 cubed-sphere transport binary for one UTC date from the ERA5 native-GRIB source described by settings, written to out_path.
mass_basis is fixed to :dry here — the writer pulls dry-basis layer mass and dry surface pressure (re-derived on the target from regridded PS + Q). Dry basis is the runtime default; a :moist request errors.
steps_per_window is the number of Strang substeps per met window. With adaptive_substeps = true (default) it is the floor: each window's count is raised until the per-substep CFL is under substep_cfl_target (at most max_steps_per_window), and the per-window schedule is written into the header. Each am / bm per-face slot stores the substep-mass amount. Per face, the Courant number of a substep is |F| / m_donor (the donor cell depends on the flux sign); the runtime subcycling gate sums each cell's outgoing x, y and z face fluxes over the palindrome and divides by its mass (cs_subcycling.jl).
The function writes through a unique sibling staging file and atomically publishes out_path on success — a partial run leaves no usable file at the requested path.
AtmosTransport.Preprocessing.process_era5_n320_window! Method
process_era5_n320_window!(pipeline, handles, date, hour) -> pipelineDrive the per-window pipeline for (date, hour):
Synthesise PS / U / V / T / Q on the N320 source mesh (breakpoint B).
Derive dry-air mass + DELP_dry + PS_dry on the source mesh (breakpoint C).
Conservatively regrid PS / U / V / T / Q to the C-tier target.
Optionally read and regrid raw UDMF / DDMF / UDRF / DDRF, put them on the dry basis, and perform the nonlinear TM5 conversion and closure on the target columns.
After the call, pipeline.window_fields, pipeline.dry_fields, pipeline.convection_fields, and pipeline.c180_fields carry the window's data on their respective grids.
AtmosTransport.Preprocessing.process_merra2_to_cs_day Method
process_merra2_to_cs_day(date, settings, target_grid; out_path, dt_met_seconds, …)Write a v4 cubed-sphere transport binary for one UTC date from MERRA-2 to out_path (written through a unique sibling staging file and atomically published when the gates pass).
Horizontal mass fluxes come from the regridded 3-hour mean winds and are Poisson-balanced (a column pressure fix) against the dry-mass change between the 3-hourly inst3 endpoints. dt_met_seconds = 10800 (the default, GEOS_3HOURLY_INTERVAL_SECONDS) writes one window per 3-hour block (8 per day); 3600 splits each block into three hourly windows (24 per day) with endpoint mass, PS, QV and T linear in time, so the hourly A1 boundary-layer fields reach the runtime unaveraged. Optional convection, PBL, VDIFF and cloud-base sections follow settings. Only mass_basis = :dry.
AtmosTransport.Preprocessing.process_merra2_window! Method
process_merra2_window!(pipe, handles, win; FT) -> pipeRead native MERRA-2 LL fields for window win (PS/QV from inst3 slice win, U/V from tavg3 slice win) and conservatively regrid PS (2D intensive) and QV/U/V (3D intensive) onto the CS target panels. U/V/QV are intensive → default field type, as in the ERA5 path. The readers return top-down levels whatever the file order; requested physics fields are regridded into pipe.phys.
AtmosTransport.Preprocessing.promote_streaming_binary! Function
promote_streaming_binary!(writer)Close and promote a staged binary file into its final path.
sourceAtmosTransport.Preprocessing.quarantine_streaming_binary! Function
quarantine_streaming_binary!(writer)Close and remove a staged binary file after a failed validation pass.
sourceAtmosTransport.Preprocessing.read_era5_n320_convection_window! Method
read_era5_n320_convection_window!(fields, handles, mesh, date, hour) -> fieldsFill fields with one hour's worth of N320 convection forecast fields for (date, hour). The reader picks handles.convection_path or handles.prev_convection_path based on the hour-address mapping and forward-iterates that file, matching messages whose (dataTime, stepRange, shortName) triple falls within the requested sample. Each matching message is decoded directly into the appropriate (n_cells, Nz) output slot via _reorder_grib_reduced_gg_to_mesh!.
Completeness gates name the missing field in any error so a corrupt or partial download is immediately visible. All four fields × 137 levels must be present for the call to succeed.
sourceAtmosTransport.Preprocessing.read_era5_n320_humidity! Method
read_era5_n320_humidity!(qv, handles, mesh, date, hour) -> qvSpecific humidity (kg kg⁻¹) of (date, hour) from the day's ERA5 core GRIB, decoded into qv (n_cells × Nz, top-first levels) on the ReducedGaussianMesh cell centres exactly as read_era5_n320_window_fields! does, without the spectral fields. Errors if a level is missing.
AtmosTransport.Preprocessing.read_era5_n320_window_fields! Method
read_era5_n320_window_fields!(fields, workspace, handles, date, hour) -> fieldsFill fields with one window's worth of N320 source-grid fields for (date, hour). Performs one forward pass over handles.core_path, decoding every message whose (dataDate, dataTime) matches into the workspace's level-indexed spectral cubes (for gridType=sh) or directly into the output qv array (for gridType=reduced_gg). After the pass the function synthesizes spectral T per level, applies vod2uv! per level and synthesizes U/V. PS comes from LNSP synthesis, or — when settings.arco_surface_pressure is set — from bilinear interpolation of the ARCO single_level sp netCDF (_fill_ps_from_arco_sp!). Errors loudly if any required level/field is absent so a partial download is immediately visible.
The function does not allocate beyond read_buf resizing inside read_spectral_coeffs!. Reusing (fields, workspace) across the 24 windows of a day is the intended call pattern.
AtmosTransport.Preprocessing.read_merra2_next_day_endpoint Method
read_merra2_next_day_endpoint(handles, Nz; FT) -> (; ps, qv[, t])Read the next day's inst3 slice-1 PS/QV (and T when VDIFF fields are requested) — the right endpoint for the final window of handles.date. Returns nothing when no next-day inst3 handle is open (archive boundary).
AtmosTransport.Preprocessing.read_merra2_physics_window Method
read_merra2_physics_window(handles, win, Nz; FT) -> NamedTupleNative moist-basis physics fields for window win, top-down, from the GEOS-Chem archive. Keys present only for the requested outputs:
cmfmc(Nx, Ny, Nz+1) kg m⁻² s⁻¹ anddtrain(Nx, Ny, Nz) kg m⁻² s⁻¹: A3mstE / A3dyn slicewin(A3mstE labels CMFMC "kg m-2 s-2", a typo).dqrcu(Nx, Ny, Nz) kg kg⁻¹ s⁻¹: A3mstC slicewin, convective rain production (only its sign is used, to locate the cloud base).pblhm,ustarm s⁻¹,hfluxandefluxW m⁻² (upward positive),t2mK, each (Nx, Ny, 3): the hourly A1 records 3(win-1)+1 .. 3win (hour averages centred 30 min into each hour of the window);efluxonly with VDIFF fields.t(Nx, Ny, Nz) K: I3 slicewin, the window-start state (as QV).
AtmosTransport.Preprocessing.read_merra2_window_fields Method
read_merra2_window_fields(handles, win, Nz; FT)
-> (; ps, qv, u, v, omega)Read one window's worth of native MERRA-2 LL fields for 1-based window index win (1..8):
ps(Pa) andqv(kg/kg) from the inst3 handle slicewin(instantaneous mass endpoint),u,v(m/s) from the tavg3 handle slicewin(3-hr time-average advecting winds), or from inst3 whenwinds_collection==:inst3.
All arrays are (lon, lat[, lev]), top-down, no reorientation.
AtmosTransport.Preprocessing.read_window! Function
read_window!(raw::RawWindow, settings::AbstractMetSettings, ctx,
date::Date, win_idx::Int) -> rawFill raw in place with one window of source data on the source's native grid. ctx is the day context returned by open_day. Implementations must NOT allocate per call beyond bounded scratch — raw is a pre-allocated workspace owned by the orchestrator — and should be idempotent in (date, win_idx).
AtmosTransport.Preprocessing.read_window! Method
read_window!(raw, settings, handles, date, win_idx) -> raw
Fill `raw` in place with one window of GEOS data on the source CS grid.
Both endpoints (t_n, t_{n+1}) carry dry DELP + dry PS reconstructed from
PS_total via the hybrid coordinate, plus the original QV. Dynamics-step
`am`/`bm` are MFXC/MFYC scaled by `1/mass_flux_dt`.The signature matches the canonical AbstractMetSettings contract declared in met_sources.jl::read_window!.
AtmosTransport.Preprocessing.regrid_ll_binary_to_cs Method
regrid_ll_binary_to_cs(ll_binary_path, cs_grid, out_path; FT=Float64, mass_basis=nothing)Regrid an existing LL transport binary to a cubed-sphere binary.
Reads each window from the LL binary, recovers cell-center winds from am/bm, conservatively regrids m/ps/winds to CS panels, rotates winds to panel-local coordinates, reconstructs CS face fluxes, closes continuity against forward mass endpoints, and stream-writes the CS binary.
This reuses the entire CS regrid/continuity/write infrastructure from the spectral→CS path — the only difference is the data source (binary reader instead of spectral synthesis).
The met interval comes from the source header. The LL winds of window w are recovered with that window's own source scaling, flux_application_seconds(dt_met, steps_w, flux_kind) with steps_w from steps_per_window_by_window (the stored am, bm are amounts of each window's own substep count). The CS output substep count is a property of the CS grid, not of the source, whose count the narrow lat-lon pole-row cells set: by default every CS window gets the smallest count whose per-substep Courant ratio meets positivity_cfl_limit (the fluxes are reconstructed for one substep, balanced, and rescaled), and the output carries that per-window schedule. Passing steps_per_window fixes one count for all windows instead.
The pole rows of the source are ordinary cells whose zonal faces carry flux, so the pole-row u is recovered from am like every other row. ERA5 spectral lat-lon binaries written before the fix of issue #58 (2026-10-14) closed those faces before the column balance; the stored pole-row am is then the balance correction, not the wind. They are recognised by their header (an ERA5 spectral lat-lon binary without pole_row_zonal_flux = "open") and refused before any output is opened. Regenerate them with the current preprocessor.
Keyword arguments
FT::Type = Float64— on-disk float type for the output CS binary.mass_basis::Union{Nothing, Symbol} = nothing— output mass-basis label.nothing(default) = match the source. Setting this to a value that differs from the source'smass_basiscurrently errors: actual dry↔moist conversion requires loading the source'sqvand applyingapply_dry_basis_native!, which this function does not do. Invariant 14 mandates:dryend-to-end; use a dry source.steps_per_window::Union{Nothing, Integer} = nothing— output substep count.nothing= per window from the CS Courant number (see above); an integer fixes the count for every window (stored per-substep fluxes scale with1/steps_per_window, the window-integrated transport does not).allow_terminal_zero_tendency::Bool = false— diagnostic-only escape hatch for legacy LL sources that do not carrydm. Production-safe regrids should leave this atfalseso the final CS window is closed against an explicit endpoint target instead of an inferred zero-tendency fallback.column_balance_weights::Symbol = :hybrid_mass— how the column Poisson balance spreads its correction over the levels (:mass,:hybrid_bor:hybrid_mass; seeAbstractColumnWeights), from the source binary's hybridBinterfaces. Other weights than:massare recorded in the header.write_replay_check::Bool = true— run the write-time replay-continuity gate.falseis a diagnostic escape hatch; the output header then recordswrite_replay_check = false.run_cache = nothing— optionalPreprocessorRunCacheused to reuse the LL→CS conservative regridder across calls in the same preprocessing run.
AtmosTransport.Preprocessing.regrid_n320_to_c180! Method
regrid_n320_to_c180!(c180_fields, n320_window, workspace, target_grid,
wind = ScalarWindRegrid()) -> c180_fieldsConservatively regrid PS (2D) and U, V, T, Q (3D) from the N320 source mesh to the C180 cubed-sphere target. PS, T and Q are intensive scalars; U and V go through wind, as two scalars (ScalarWindRegrid, the default) or as a vector (CartesianWindRegrid). The flat scratch buffers in workspace hold the intermediate Float64 arrays.
AtmosTransport.Preprocessing.regrid_transport_binary Method
regrid_transport_binary(input_path, target_grid, out_path; kwargs...)Generic transport-binary regrid extension point. Concrete target/source combinations implement methods here instead of adding new topology-specific scripts. The currently implemented production pair is LL transport binary to cubed sphere.
sourceAtmosTransport.Preprocessing.remap_era5_humidity! Method
remap_era5_humidity!(remap::ERA5HumidityRemap, hour) -> q_targetRead the specific humidity of hour (0–23 UTC) on the N320 mesh (read_era5_n320_humidity!) and remap it to the target mesh. Returns remap.q_target, (target cells, Nz) with top-first levels.
AtmosTransport.Preprocessing.run_unified_preprocessor_day! Method
run_unified_preprocessor_day!(day::UnifiedPreprocessorDay; close_reader=true)Execute the additive unified-driver lifecycle for one day. The function is generic over concrete reader/workspace/contract/writer types and depends on hook methods for topology-specific ingest/drain/flush behavior.
The writer is closed before summarize_status!, so fatal positivity summaries can quarantine a closed staging file. Any exception before promotion closes and quarantines the writer, then closes the reader.
AtmosTransport.Preprocessing.set_end_of_day_seed! Method
set_end_of_day_seed!(reader::GEOSNativeReader, seed) → readerSet the end-of-day mass seed produced by the orchestrator's last window. Called once per day at the end of process_day. For NoChain readers this is a no-op.
AtmosTransport.Preprocessing.source_grid Function
source_grid(settings::AbstractMetSettings)Return the source grid mesh that read_window! produces data on. Used by the orchestrator to build the appropriate regridder for the target grid (or to detect the source==target passthrough case).
AtmosTransport.Preprocessing.source_grid Method
source_grid(settings::GEOSSettings) -> CubedSphereMeshThe native source mesh GEOS data is archived on (Nc × Nc per panel, GEOS-native panel convention).
AtmosTransport.Preprocessing.source_grid Method
source_grid(settings::MERRA2Settings; FT=Float64) -> LatLonMeshThe native MERRA-2 source mesh (576 × 361, -180..180 lon, -90..90 lat). The preprocessor builds its own regridder against the target mesh radius; this canonical descriptor uses the same EARTH_RADIUS as preprocessing targets.
AtmosTransport.Preprocessing.summarize_cs_positivity_status Method
summarize_cs_positivity_status(worst; cfl_limit, steps_per_window,
require_substep_positivity = true,
quarantine_path = nothing)Post-loop summary helper. Logs the worst-window outcome, and if it exceeds cfl_limit:
deletes
quarantine_path(if given) so a downstream consumer cannot pick up the half-written binary;errors when
require_substep_positivity = true, otherwise warns.
The error message includes a recommended steps_per_window value that would satisfy the gate, computed from the observed worst ratio.
AtmosTransport.Preprocessing.summarize_ll_positivity_status Method
summarize_ll_positivity_status(worst; cfl_limit, steps_per_window,
require_substep_positivity = true,
quarantine_path = nothing)Post-loop summary helper for the LL positivity accumulator. Logs the worst-window outcome, and if it exceeds cfl_limit:
deletes
quarantine_path(if given) so a downstream consumer cannot pick up the half-written binary;errors when
require_substep_positivity = true, otherwise warns.
The error/warn message includes a recommended steps_per_window value that would satisfy the gate, computed from the observed worst ratio. The "no representable rescue" branch from CS round-3 is mirrored here.
AtmosTransport.Preprocessing.summarize_rg_positivity_status Method
summarize_rg_positivity_status(worst; cfl_limit, steps_per_window,
require_substep_positivity = true,
quarantine_path = nothing)Post-loop summary helper for the RG positivity accumulator. Mirrors summarize_cs_positivity_status (CS round-2 + round-3 escape-hatch semantics).
AtmosTransport.Preprocessing.summarize_status! Function
summarize_status!(contract::AbstractWindowContract;
quarantine_path::Union{Nothing, AbstractString} = nothing)
-> nothingPost-loop summary helper. Logs the worst-window outcome; if the contract's policy demands and the worst exceeds the gate, deletes quarantine_path (if given) and errors. Otherwise warns when the gate is violated but the policy is set to record-and-continue.
AtmosTransport.Preprocessing.summarize_status! Method
summarize_status!(contract::CubedSphereContract;
quarantine_path::Union{Nothing, AbstractString} = nothing)Run the CS positivity post-loop summary using the contract's worst accumulator and policy fields. May log, warn, or error depending on the accumulator state and require_substep_positivity.
AtmosTransport.Preprocessing.target_summary Method
target_summary(grid) -> StringReturn a short human-readable summary of the configured target grid.
sourceAtmosTransport.Preprocessing.tm5_convection_to_basis! Method
tm5_convection_to_basis!(basis, entu, detu, entd, detd, q) -> nothingPut one top-first column of TM5 layer-centre convection fields (kg m⁻² s⁻¹, per unit of total air, as in the TM5 meteo archive) on the air-mass basis. For MoistBasis() nothing changes.
For DryBasis() the draft mass fluxes are rebuilt from the TM5 budgets, converted where they live, and the entrainment re-derived from the converted budgets, as ecmwf_convection_to_basis! followed by ec2tm would do:
with the interface humidity q̂_k of interface_specific_humidity. Scaling every field by 1 − q_k instead would break the column closure Σ e = Σ d, because a draft entrains moist air low down and detrains drier air aloft. Each draft is converted only over its active span (the levels where its entrainment or detrainment is non-zero), so no rounding-level values appear above the cloud top, where TM5Convection diagnoses the cloud dimensions. A negative re-derived rate is moved to its partner rate as in TM5's ECconv_to_TMconv, which keeps e − d and so the fluxes.
AtmosTransport.Preprocessing.update_accumulator! Function
update_accumulator!(contract::AbstractWindowContract,
positivity_diag, win_idx::Int) -> nothingFold one window's positivity diagnostic into the contract's worst- window accumulator. Concrete contracts mutate their internal state and return nothing. Idempotent if positivity_diag.ratio does not exceed the current worst.
AtmosTransport.Preprocessing.update_accumulator! Method
update_accumulator!(contract::CubedSphereContract, positivity_diag, win_idx::Int)Fold one window's positivity diagnostic into the CS contract's worst-window accumulator. Mutates contract.worst.
AtmosTransport.Preprocessing.update_cs_positivity_accumulator Method
update_cs_positivity_accumulator(worst, diag, win_idx) -> NamedTupleReturn an updated accumulator from a fresh per-window diagnostic.
sourceAtmosTransport.Preprocessing.update_ll_positivity_accumulator Method
update_ll_positivity_accumulator(worst, diag, win_idx) -> NamedTupleReturn an updated LL accumulator from a fresh per-window diagnostic.
sourceAtmosTransport.Preprocessing.update_rg_positivity_accumulator Method
update_rg_positivity_accumulator(worst, diag, win_idx) -> NamedTupleReturn an updated RG accumulator from a fresh per-window diagnostic.
sourceAtmosTransport.Preprocessing.validate_merra2_settings Method
validate_merra2_settings(settings) -> settingsReject unknown wind collections and requests the archive cannot serve (only the GEOS-Chem files carry CMFMC, DTRAIN, DQRCU and A1).
sourceAtmosTransport.Preprocessing.verify_boundary_stub_flux_rg Method
verify_boundary_stub_flux_rg(hflux, face_left, face_right;
tol = 0.0) -> NamedTupleExplicit-invariant scan: any non-zero hflux value on a boundary stub (face_left ≤ 0 or face_right ≤ 0) is a contract violation. The runtime advection silently discards such fluxes (Advection/sweeps.jl), so a writer that produces them is emitting data the runtime cannot apply — almost always a sign-flip or boundary- masking bug in preprocessing.
Returns (violated, worst_flux, worst_face, worst_level):
violated :: Bool—trueiff any |flux| > tol on a boundary stub.worst_flux :: Float64— signed value of the worst-magnitude violation, or0.0if none.worst_face :: Int— face index of the worst violation, or0.worst_level :: Int— k-index of the worst violation, or0.
tol is the absolute tolerance below which a "near-zero" stub flux is permitted. Default 0.0 (strict) — RG writers should explicitly zero boundary stubs.
AtmosTransport.Preprocessing.verify_cs_window_contract! Method
verify_cs_window_contract!(m_cur, am, bm, cm, m_next, steps_per_window, win_idx;
replay_tol, positivity_cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
halo_width = 0, replay_scratch = nothing)Single canonical per-window CS binary contract check. Runs the replay gate (verify_write_replay_cs!, errors on failure) followed by the per-substep positivity scan (verify_substep_positivity_cs, returns a diagnostic). Every CS-producing preprocessor (spectral, regrid, GEOS-native) should call this so no path can silently skip a gate.
Returns (; replay, positivity) with both diagnostics. Positivity is non-fatal here — callers aggregate the worst window and pass it to summarize_cs_positivity_status after the loop, where the run-level require_substep_positivity policy decides whether to error or warn.
AtmosTransport.Preprocessing.verify_flux_fidelity! Method
verify_flux_fidelity!(acc, bundle::FluxFidelityEvidenceBundle, win)Verify one window's evidence before its payload is written: sampled wind fluxes against their independent reconstruction (throws on a failed sample), non-empty and finite OMEGA samples, and a column-closure record within its per-cell tolerances. Aggregates are serialized by flux_fidelity_header!.
AtmosTransport.Preprocessing.verify_ll_window_contract! Method
verify_ll_window_contract!(m_cur, am, bm, cm, m_next, steps_per_window, win_idx;
replay_tol, positivity_cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
replay_scratch = nothing)Single canonical per-window LL binary contract check. Runs the replay gate (errors on failure) followed by the per-substep positivity scan (verify_substep_positivity_ll!, returns a diagnostic).
replay_scratch (a ReplayScratch of the mass shape) may be pre-allocated by the caller (workspace-owned scratch) to suppress the per-window allocation. Default nothing → allocate locally.
Returns (; replay, positivity). Positivity is non-fatal here — callers aggregate the worst window across the loop and pass it to summarize_ll_positivity_status where the run-level require_substep_positivity policy decides whether to error or warn.
AtmosTransport.Preprocessing.verify_rg_window_contract! Method
verify_rg_window_contract!(m_cur, hflux, cm, m_next, face_left, face_right,
steps_per_window, win_idx;
replay_tol, positivity_cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
replay_scratch = nothing,
outgoing_h = nothing, bad_h = nothing,
boundary_stub_tol = 0.0)Single canonical per-window RG binary contract check. Runs three gates in order:
Boundary-stub flux gate — errors hard if any boundary stub (
face_left ≤ 0/face_right ≤ 0) carries non-zerohfluxaboveboundary_stub_tol. Norequire_*escape hatch: such fluxes are silently discarded by the runtime (Advection/sweeps.jl), so emitting them is always a writer bug.Replay gate —
verify_window_continuity_rg; errors on failure.Per-substep positivity scan —
verify_substep_positivity_rg!, returns a non-fatal diagnostic; the run-level accumulator +summarize_rg_positivity_statusdecides fatal-vs-warn.
replay_scratch (a ReplayScratch of the mass shape), outgoing_h, bad_h may be pre-allocated by the caller (workspace-owned scratch) to suppress per-window allocation. Default nothing → allocate locally.
Returns (; replay, positivity). Boundary-stub failure does not return; it errors out before the replay gate so a broken writer cannot silently emit a binary the runtime would partially evaluate.
AtmosTransport.Preprocessing.verify_substep_positivity_cs! Method
verify_substep_positivity_cs!(m, am, bm, cm; cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
halo_width = 0, m_next = nothing)Verify the per-substep horizontal+vertical positivity contract that the runtime's _cs_static_palindrome_subcycle_count depends on. For every interior cell on every panel (any rectangular block; the lat-lon gate verify_substep_positivity_ll! passes its window as one block):
The cell air mass itself must be positive (
m > 0), a preprocessing invariant enforced here (the runtime CFL scan skips such cells). A non-positive cell mass is reported withratio = Infregardless of flux magnitude.The combined Strang-palindrome outgoing budget
2 * (out_x + out_y + out_z)must not exceedcfl_limit * m_ref, wherem_ref = min(m, m_next)whenm_nextis supplied by a caller that wants endpoint tightening, andm_ref = motherwise. The factor of 2 is required because the CS runtime applies the direction sequenceX-Y-Z-Z-Y-Xfor every met substep.
Returns a NamedTuple (direction, ratio, ratio_xy, ratio_z, location, ok):
direction :: Union{Symbol, Nothing}— the dominant contributor among:x,:y,:z, ornothingwhen no cell was inspected.ratio :: Float64— worst observed palindrome outgoing budget over the reference mass, orInfif any inspected cell had invalid mass or flux.ratio_xy :: Float64— worst observed horizontal-only palindrome outgoing budget over the reference mass.ratio_z :: Float64— worst observed vertical-only palindrome outgoing budget over the reference mass.location :: NTuple{4, Int}—(panel, i, j, k)of the worst cell.ok :: Bool—trueiffratio <= cfl_limit.
The replay gate (verify_write_replay_cs!) only checks endpoint continuity. A binary that drives a cell mass negative mid-sweep can still pass replay because the cell re-fills from inflow before the window ends — but the runtime cannot recover. This gate is the actual contract the runtime depends on.
halo_width defaults to 0 (panel arrays are stored unhaloed at preprocess time); pass > 0 to scan only the interior of a haloed buffer.
AtmosTransport.Preprocessing.verify_substep_positivity_ll! Method
verify_substep_positivity_ll!(m, am, bm, cm; cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT, m_next = nothing)Per-substep positivity contract of a structured lat-lon window, the same as the cubed-sphere one (verify_substep_positivity_cs!, here on one block). One stored flux amount is one application in the runtime palindrome X-Y-Z-Z-Y-X, so each direction acts twice per substep. For every cell (i, j, k):
m > 0(andm_next > 0when given); non-finite masses or fluxes and non-positive masses are reported withratio = Inf.The palindrome outflow
2 (out_x + out_y + out_z)must not exceedcfl_limit * m_ref, withm_ref = min(m, m_next)whenm_nextis given andmotherwise.out_dis the cell's outflow through its twodfaces for one application.
This bounds the outflow of a whole substep by the mass the cell has at either end of the window. The earlier lat-lon gate checked each direction's single application against m alone, so it admitted substeps the cubed-sphere gate rejects.
Returns (direction, ratio, ratio_xy, ratio_z, location, ok) as the cubed-sphere gate does, with location = (i, j, k).
AtmosTransport.Preprocessing.verify_substep_positivity_rg! Method
verify_substep_positivity_rg!(m, hflux, cm, face_left, face_right;
cfl_limit = DEFAULT_POSITIVITY_CFL_LIMIT,
outgoing_h = nothing, bad_h = nothing)Per-substep horizontal+vertical positivity scan for a face-indexed RG window. Mirrors verify_substep_positivity_cs! / ..._ll! but operates on the face-indexed RG mass-flux representation.
For every cell (c, k):
m > 0. A non-positive cell mass is reported withratio = Inf.Horizontal outgoing mass per substep ≤
cfl_limit * m. Only interior faces (face_left > 0 && face_right > 0) contribute, matching the runtime advection inAdvection/sweeps.jl. Boundary stubs are not counted as outflow here — seeverify_boundary_stub_flux_rgfor the separate "non-zero flux on a boundary stub" invariant.Vertical outgoing mass per substep ≤
cfl_limit * m. Same as CS / LL:max(0, -cm[c, k]) + max(0, cm[c, k+1]).
NaN/Inf cell mass and NaN/Inf fluxes are flagged as ratio = Inf (matches the CS round-2 fix).
outgoing_h and bad_h can be passed in as workspace-owned scratch to suppress per-window allocation once this is wired into the unified driver. Default nothing → allocate locally.
Returns (direction, ratio, location, ok) with:
direction :: Union{Symbol, Nothing}—:h/:z/nothing.ratio :: Float64— worstoutgoing / m.location :: NTuple{2, Int}—(cell, level).ok :: Bool—ratio ≤ cfl_limit.
AtmosTransport.Preprocessing.verify_window! Function
verify_window!(window, contract::AbstractWindowContract, win_idx::Int)
-> (; replay, positivity)Run the contract's replay and positivity gates for one window. The replay gate throws on violation; the positivity gate is non-fatal at this layer (the run-level accumulator + summarize_status! decides fatal-vs-warn based on require_substep_positivity).
window is the topology's per-window payload (a NamedTuple today, a typed ReadyWindow{G, FT} later).
AtmosTransport.Preprocessing.verify_window! Method
verify_window!(window, contract::CubedSphereContract, win_idx::Int;
write_replay_on = true) -> (; replay, positivity)Run the per-window CS contract on a NamedTuple window with fields m_cur, am, bm, cm, m_next (each a 6-tuple of panel arrays). Delegates to verify_cs_window_contract!; the replay gate throws on violation, the positivity gate is non-fatal here. With write_replay_on = false ([numerics] write_replay_check = false) only the positivity gate runs and the replay diagnostic is zero.
AtmosTransport.Preprocessing.verify_write_replay_cs! Method
verify_write_replay_cs!(m_cur, am, bm, cm, m_next, steps_per_window, tol_rel, win_idx;
replay_scratch = nothing)Run the CS write-time replay gate for one window and return its diagnostic.
The check integrates the stored panel-local fluxes from m_cur under the runtime palindrome-continuity contract and verifies, cell by cell, that the result matches the explicit endpoint m_next (the metric of verify_window_continuity). A failure here means the binary would produce a runtime day-boundary or window-boundary mass inconsistency.
replay_scratch (a ReplayScratch) may be pre-allocated by the caller (workspace-owned scratch or contract-owned lazy scratch) so the gate doesn't allocate the panel-shared work arrays per call. Default nothing → allocate locally. Shape must match size(m_cur[1]).
AtmosTransport.Preprocessing.window_metadata Function
window_metadata(reader::AbstractMetReader) → NamedTuplePer-source window timing metadata. Standard fields:
windows::Int—windows_per_day(reader).substeps::Int— sub-windows per write window (e.g. GEOS'sdt_met_seconds ÷ mass_flux_dt).dt_substep::Float64— substep wall-clock in seconds.
Concrete readers may add source-specific keys (e.g. GEOS's mass_flux_dt).
AtmosTransport.Preprocessing.windows_per_day Function
windows_per_day(settings::AbstractMetSettings, date::Date) -> IntNumber of preprocessing windows per UTC day for this source on date. For most sources this is constant (24 for hourly), but date-dependent sources (e.g. a leap-second day) may override.
AtmosTransport.Preprocessing.write_window! Function
write_window!(writer, ready)Write one validated ready window through a topology-specific binary writer. Concrete methods are topology-indexed by AbstractBinaryWriter and ReadyWindow so writer/window mismatches fail by dispatch.
AtmosTransport.Preprocessing.write_window! Method
write_window!(io, win_idx, storage, settings, merged, last_hour_next) -> Int64Write one window's payload blocks to the output stream in v4 on-disk order.
source