GriddedIO API
AtmosTransport.GriddedIO owns the conventions of native gridded input files (surface-flux inventories, land fractions, initial-condition fields). Each rule is a type the caller names, and the methods dispatch on it:
Coordinates.
find_coordinatefinds a coordinate by variable name only (never byunitsorlong_name, which real files mislabel).latlon_orientation,orientandorient_coordinatesput a field into ascending latitude and longitudes starting at the first value≥ 0.Fill values.
read_rawwithMaskedAsZero,MaskedAsNaNorFillValueOnly, straight into the caller's element type.Source meshes.
source_latlon_meshderives cell faces from a file's centres by a face rule.SnappedFaces(the default): the snapping rule for regular global axes, faces at the centres ± half a cell for longitudes centred on 0° or ±180°, midpoint faces with pole and truncation rules for pole-centred, truncated and Gaussian latitudes.UniformSpanFaces: the uniform mesh the runtime surface-flux and initial-condition loaders used before they moved toSnappedFaces; the runtime no longer uses it. Bounds variables are never used.Time axes.
decode_timeclassifies an axis by its rawunitsbefore any CFTime call and decodes it withCFDecoder,MonthsSince,DecimalYearsorAuxiliaryTimeVariable;MonthKey,TimeIntervalandseconds_sincecarry the result;decode_slice_intervalsgives the interval each slice of a file stands for (the time coverage of a time-varying surface flux), fromknown_slice_intervals(CF bounds,months since) orstamp_intervals, the regular step of a series of stamps, which a series of several files applies to the stamps of all its files together. Origins are validated, never normalised.decode_timereturns proleptic GregorianDateTimes, so a360_day30 February is an error there;decode_month_keysgives the months of any axis in its own calendar.Staging.
stage_inputdecompresses.nc.gzand single-member.nc.tar.gzinputs into a cache.Flux units.
parse_flux_unitsreads every units string in use intoFluxUnits;mass_rate_factorconverts to kg of a species per second with the molar masses ofParameters.cs_nativeflux files (the runtime contract ofsurface_flux.kind = "cs_native", defined once): layoutCS_NATIVE_FLUX_DIMS; the writeropen_cs_native_writer,write_slices!,publish!andabandon!(start-stamped intervals withtime_bnds, mesh geometry and cell areas, no fill values, atomic publication); the streaming readercs_native_flux_infoandread_cs_native_flux!, the runtime's loader, panel by panel into the caller's arrays; the whole-file readerread_cs_native_flux; the cell-area gatecheck_cell_area.Lat-lon flux files for
kind = "file":write_latlon_flux, static fields only, on meshes whose faces the runtime rebuilds exactly from the written centres withSnappedFaces.Observation ground fields (
[output.observations]vertical placement): the per-grid model surface height,write_model_surface_heightandread_model_surface_height(grid-checked, intoOutput.ModelSurfaceHeight), and a lat-lon DEM for point lookups,read_point_elevation→LatLonPointElevation.
The rules were measured on the TRENDY v14, GFED5, FLUXCOM-X, CAMS, GridFED and EDGAR files; src/GriddedIO/README.md lists them with their invariants.
AtmosTransport.GriddedIO Module
GriddedIOThe one owner of the conventions of native gridded input files (surface-flux inventories, land fractions, initial-condition fields): how their coordinates are found, which values are fill, how a latitude-longitude field is oriented, which cell faces its coordinates imply, how its time axis is decoded, how a compressed input is staged, and how its flux units are read. It also owns the schema of the surface-flux files the model writes and the runtime reads (cs_native and lat-lon), with their one writer and reader.
Every rule here was measured on the files the model reads (the TRENDY v14 S3 models, GFED5, FLUXCOM-X, CAMS/LMDZ, GridFED, EDGAR); the measurements are in docs/memos/2026-10-10_python_flux_port_p0_facts.md. The rules are typed: the caller names an axis (LongitudeAxis), a fill policy (MaskedAsZero), a time decoder (CFDecoder), and the methods dispatch on it.
Contents
Coordinates (
coordinates.jl):find_coordinateby variable name only,latlon_orientation/orient/orient_coordinates, andread_rawwith the fill policiesFillValueOnly,MaskedAsNaNandMaskedAsZero.Source meshes (
source_meshes.jl):source_latlon_mesh, the cell faces of a file's latitude-longitude grid, by the face ruleSnappedFaces(onGrids.LatLonMesh(λᶠ, φᶠ; radius)) orUniformSpanFaces(the legacy uniform-span reconstruction).Time axes (
time_axes.jl):decode_timewith the decodersCFDecoder,MonthsSince,DecimalYearsandAuxiliaryTimeVariable, chosen from the rawunitsstring before any CFTime call, anddecode_month_keys, the months of an axis in its own calendar;MonthKey,TimeInterval,month_keys,interval,seconds_since,decode_month_seconds,check_last_month; the interval each slice stands for,decode_slice_intervals(one file),known_slice_intervalsandstamp_intervals(a series of files).Staging (
staging.jl):stage_inputdecompresses.nc.gzand single-member.nc.tar.gzinputs into a cache directory.Flux units (
flux_units.jl):parse_flux_units→FluxUnits,mass_rate_factor,kg_per_unit,period_seconds,canonical_units_string.The
cs_nativeflux file (cs_native_flux.jl), the runtime contract:CS_NATIVE_FLUX_DIMS, the writeropen_cs_native_writer/write_slices!/publish!/abandon!, the streaming readercs_native_flux_info/read_cs_native_flux!(the runtime's loader, panel by panel into the caller's arrays), the whole-file readerread_cs_native_fluxand the cell-area gatecheck_cell_area.Lat-lon flux files for
kind = "file"(latlon_flux.jl):write_latlon_flux.Static ground fields for observation placement (
surface_height.jl): the per-grid model surface height dataset, written bywrite_model_surface_heightand read byread_model_surface_heightintoOutput.ModelSurfaceHeight, and a lat-lon DEM for point lookups,read_point_elevation→LatLonPointElevation(bilinearOutput.point_elevation).
Molar masses and the year length come from Parameters (SPECIES_MOLAR_MASS, CARBON_MOLAR_MASS, DAYS_PER_JULIAN_YEAR).
src/AtmosTransport.jl loads this module after Regridding and before Preprocessing; it imports Parameters, Grids, Output (the shared cubed-sphere geometry and provenance blocks) and AtomicFiles. See src/GriddedIO/README.md for the file map and invariants.
AtmosTransport.GriddedIO.CS_NATIVE_FLUX_DIMS Constant
CS_NATIVE_FLUX_DIMSDimension names of a cs_native flux series in Julia order, ("Xdim", "Ydim", "nf", "time"); in NetCDF (CF) order (time, nf, Ydim, Xdim). Xdim and Ydim index the cells of a panel and nf the six panels, in the panel convention of the mesh (Output.define_cs_horizontal_geometry!).
AtmosTransport.GriddedIO.DEFAULT_TIME_DECODERS Constant
The decoders decode_time uses unless given others: (CFDecoder(), MonthsSince()). Axes without usable units need a decoder named per product.
AtmosTransport.GriddedIO.STAMP_SPACING_TOLERANCE Constant
STAMP_SPACING_TOLERANCESpacings of interval-start stamps that differ by at most this much are the same step in stamp_intervals (float stamps such as days since eighths decode to the millisecond).
AtmosTransport.GriddedIO.AbstractAreaBasis Type
AbstractAreaBasisWhat a flux is per: PerSquareMetre (a density) or PerCell (a per-cell rate; the caller divides by the cell area).
AtmosTransport.GriddedIO.AbstractAxis Type
AbstractAxisA coordinate axis of a gridded input file: LongitudeAxis, LatitudeAxis, TimeAxis or VerticalAxis. find_coordinate dispatches on it.
AtmosTransport.GriddedIO.AbstractFaceRule Type
AbstractFaceRuleHow source_latlon_mesh derives cell faces from a file's cell centres: SnappedFaces (the default) or UniformSpanFaces.
AtmosTransport.GriddedIO.AbstractFillPolicy Type
AbstractFillPolicyWhich attributes of a variable mark fill values, and what read_raw puts in their place:
| Policy | Fill marked by | Fill becomes | Use |
|---|---|---|---|
MaskedAsZero | _FillValue and missing_value | 0.0 | runtime flux loaders; FLUXCOM-X land fraction |
MaskedAsNaN | _FillValue and missing_value | NaN | flux sources whose statistics count only valid cells |
FillValueOnly | _FillValue only | NaN | variables whose missing_value is wrong (IBIS land fraction) |
A _FillValue of NaN matches NaN values. scale_factor and add_offset are applied as NCDatasets applies them (no flux variable of the measured inputs has them). valid_min, valid_max and valid_range are not applied.
AtmosTransport.GriddedIO.AbstractMonthStamp Type
AbstractMonthStampWhere a monthly time stamp sits in its month, for DecimalYears and AuxiliaryTimeVariable: MonthStart, MidMonth or MonthEnd. Which one a product uses is a fact about the product, named in its configuration; the catalog's last_month check (check_last_month) catches a wrong choice.
AtmosTransport.GriddedIO.AbstractSubstance Type
AbstractSubstanceWhat a flux amount counts: TargetSpecies, Carbon or a named Species.
AtmosTransport.GriddedIO.AbstractTimeBasis Type
AbstractTimeBasisThe time unit of a flux: PerSecond, PerDay, PerMonth or PerYear. period_seconds gives its length.
AtmosTransport.GriddedIO.AbstractTimeDecoder Type
AbstractTimeDecoderA rule that turns a time axis into DateTime stamps. decode_time classifies the axis by its raw units string and uses the first decoder of its list that accepts that class:
| Raw units | Decoder |
|---|---|
days/hours/minutes/seconds since <origin> | CFDecoder |
months since <origin> | MonthsSince |
none, years since …, or unparseable | DecimalYears or AuxiliaryTimeVariable, named per product |
AtmosTransport.GriddedIO.AuxiliaryTimeVariable Type
AuxiliaryTimeVariable(variable, origin::Date, unit::Dates.Period, stamp::AbstractMonthStamp)Decoder of a monthly time axis that is only an index (no units), whose stamps are in another variable of the file. CARDAMOM: time = 1 … 264; the Time variable (units "d", long name "Monthly time step given in days since 01/01/2003") holds 31, 59, 90, …, 8036, the end of each month, so AuxiliaryTimeVariable("Time", Date(2003, 1, 1), Day(1), MonthEnd()) gives 2003-01 … 2024-12. Stamp k is DateTime(origin) + v_k · unit, rounded to the millisecond, where v_k is value k of variable, which must have as many values as the axis; it decodes to the first instant of the month it belongs to under stamp.
AtmosTransport.GriddedIO.CFDecoder Type
CFDecoder()Decoder of <unit> since <origin> axes whose unit has a fixed length (day, hour, minute, second, millisecond; also the abbreviations d, h, hr, min, s, sec), through CFTime, honouring the calendar attribute:
standard,gregorian,proleptic_gregorian,julianor no calendar: NCDatasets' own decoding, rounded to the millisecond (the runtime's stamps, bit for bit; ajulianaxis keeps its instants, so its dates shift by the Julian–Gregorian difference);noleap/365_day,all_leap/366_day,360_day: the calendar date and time of day of each stamp in its own calendar (values rounded to the millisecond first), carried over to aDateTime. Everynoleap/365_daydate exists in the proleptic Gregorian calendar; anall_leap/366_dayor360_daydate that does not (29 February of a common year, 30 February) is anArgumentErrornaming it.decode_month_keystakes the months of such axes from the calendar dates themselves.
A trailing " UTC" on the origin is dropped. Fill values in the axis are an error.
AtmosTransport.GriddedIO.CSNativeFluxWriter Type
CSNativeFluxWriterA cs_native flux file being written, returned by open_cs_native_writer. Slices go in with write_slices!; publish! completes the file and renames it onto its path, and abandon! discards it. Until then only a staging file next to the target exists (AtomicFiles), so readers of the path see the previous file (or none), never a partial one.
AtmosTransport.GriddedIO.Carbon Type
Carbon()Amounts of carbon (kgC, g C, mol C). Converted to a carbon-bearing species with one carbon atom per molecule by M(species) / CARBON_MOLAR_MASS, which conserves carbon atoms (CO₂: SPECIES_MOLAR_MASS.co2 / CARBON_MOLAR_MASS ≈ 3.66419).
AtmosTransport.GriddedIO.DecimalYears Type
DecimalYears(stamp::AbstractMonthStamp)Decoder of monthly axes stored as decimal years without usable units (ELM-FATES, month starts: DecimalYears(MonthStart()); VISIT-UT, mid-month, units years since AD 0-Jan-1st (double): DecimalYears(MidMonth())). With m = 12 v months since year 0, the month index is k̂ = round(m − o) (o = 0 for month starts, ½ for mid-month stamps, 1 for month ends) and the residual |m − o − k̂| must not exceed 10⁻⁶ month, else an ArgumentError. Rounding to the nearest month is required: ELM-FATES has 1296 values 9.1e-13 month below the integer, where a plain floor gives 2592 bad steps. Each stamp decodes to the first instant of its month.
AtmosTransport.GriddedIO.FillValueOnly Type
FillValueOnly()Fill policy: only _FillValue marks fill (it becomes NaN); missing_value is ignored. For variables whose missing_value attribute is wrong: the IBIS land-fraction file declares missing_value = 0, which is a valid fraction.
AtmosTransport.GriddedIO.FluxUnits Type
FluxUnits{S, A, T}Parsed flux units (parse_flux_units).
Fields
text— the units string as givenamount_per_unit— kg (mass units) or mol (molar units) per stated amount unit:1for kg and mol,1e-3for g and mmol,1e3for tonnesmolar—truefor molar units (mol, mmol, µmol)substance::S—TargetSpecies,CarbonorSpeciesarea::A—PerSquareMetreorPerCell
AtmosTransport.GriddedIO.LatLonOrientation Type
LatLonOrientation(flip_lat, lon_shift, wrap_lon)How orient reorders a latitude-longitude field (lon, lat, ...):
flip_lat: reverse the latitude axis (the file stores north to south);lon_shift: move the firstlon_shiftlongitude columns to the end (a circular shift by-lon_shift);wrap_lon: map the reordered longitudes into[0°, 360°)(orient_coordinates).
Built by latlon_orientation.
AtmosTransport.GriddedIO.LatLonPointElevation Type
LatLonPointElevationA latitude-longitude digital elevation model read by read_point_elevation: ascending longitudes lon (any convention, spanning less than 360°) and latitudes lat, the ground elevation values[i, j] (m above sea level, Float32, NaN where the file has fill), whether the longitudes wrap around the globe (periodic), and the file path. Output.point_elevation(dem, lon, lat) interpolates it bilinearly between the four surrounding centres.
AtmosTransport.GriddedIO.LatitudeAxis Type
Latitude axis; coordinate variable names lat, latitude, y (in that order).
AtmosTransport.GriddedIO.LongitudeAxis Type
Longitude axis; coordinate variable names lon, longitude, x (in that order).
AtmosTransport.GriddedIO.MaskedAsNaN Type
MaskedAsNaN()Fill policy: values equal to _FillValue or to any missing_value become NaN, so a caller can count valid values (isfinite) before replacing them. VISIT, VISIT-UT and iMAPLE mark fill only through missing_value = -99999.
AtmosTransport.GriddedIO.MaskedAsZero Type
MaskedAsZero()Fill policy: values equal to _FillValue or to any missing_value become 0.0. This is the runtime convention (nomissing(ds[var][...], 0)).
AtmosTransport.GriddedIO.MidMonth Type
Stamps in the middle of the month: decimal year y + (k + ½)/12; the stamp lies in its month.
AtmosTransport.GriddedIO.MonthEnd Type
Stamps at the end of the month: the first instant of the next month (CARDAMOM's Time = 31, 59, 90, … days since 2003-01-01 for January, February, March, …); decimal year y + (k + 1)/12.
AtmosTransport.GriddedIO.MonthKey Type
MonthKey(year, month)
MonthKey(t::Dates.TimeType)
MonthKey("YYYY-MM")A calendar month of the proleptic Gregorian calendar. Keys order chronologically; k + n is the month n months later.
AtmosTransport.GriddedIO.MonthStart Type
Stamps at the start of the month: decimal year y + k/12; the stamp lies in its month.
AtmosTransport.GriddedIO.MonthsSince Type
MonthsSince()Decoder of months since <origin> axes (CABLE-POP, ISAM, LPJml, VISIT, iMAPLE). The month of stamp v is the origin month plus ⌊v + 10⁻⁶⌋. The origin must be a valid date Y-M[-D] with an optional time of day (an ArgumentError otherwise, see decode_time); only its year and month are read (CABLE-POP's origin is 1700-1-16 12:00:00). Integer stamps (month starts) and stamps k + ½ (mid-month) both give month k. Each stamp decodes to the first instant of its month; the calendar attribute does not matter.
AtmosTransport.GriddedIO.PerCell Type
Flux per grid cell (no area in the units, or cell-1): GFED g C per month, EDGAR Tonnes.
AtmosTransport.GriddedIO.PerMonth Type
Per calendar month (month-1, /month, per month): the length of the month containing the interval.
AtmosTransport.GriddedIO.PerYear Type
Per Julian year of DAYS_PER_JULIAN_YEAR days (yr-1, /year; EDGAR Tonnes).
AtmosTransport.GriddedIO.SnappedFaces Type
SnappedFaces()Face rule of source_latlon_mesh, its default: regular global axes snap to the grid their resolution implies, with faces built from integers; pole-centred, truncated, Gaussian and regional latitudes get midpoint faces with the pole and truncation rules of the source_latlon_mesh docstring.
AtmosTransport.GriddedIO.Species Type
Species(name::Symbol)Amounts of the species name (kgCO2 → Species(:co2)). Names are lower case; the molar mass of a species comes from SPECIES_MOLAR_MASS.
AtmosTransport.GriddedIO.TargetSpecies Type
TargetSpecies()The units name no substance (kg m-2 s-1): the amount is of the output species, the runtime's reading of such files. Molar units cannot be of an unnamed substance.
AtmosTransport.GriddedIO.TimeAxis Type
Time axis; coordinate variable names time, t (in that order).
AtmosTransport.GriddedIO.TimeInterval Type
TimeInterval(start::DateTime, stop::DateTime)The half-open time interval [start, stop); start < stop is required.
AtmosTransport.GriddedIO.UniformSpanFaces Type
UniformSpanFaces()Face rule of source_latlon_mesh that gives the uniform mesh of the former runtime surface-flux and initial-condition loaders (Models.InitialConditionIO), bit for bit. The mesh is uniform: for n centres x, the spacing is the mean spacing (x[n] − x[1]) / (n − 1) and the faces lie half a spacing outside the first and last centres. A latitude extent within 1e-3 of a cell of ±90° becomes ±90°; a longitude span within 1e-3 of a cell of 360° becomes exactly 360°, measured from the west face.
Taking the spacing from the whole span, not the first difference, keeps Float32-stored centres from biasing every cell (GridFED stores its 0.1° centres in Float32; their first difference errs by about 1e-4 relative), and the extent snapping keeps the regridder from missing a sliver of the sphere (GridFED: 1.5e-5° of longitude, 2.1e-6 of the global flux). The faces still follow the centres' round-off: GridFED's west face is 7.45e-10°, where SnappedFaces gives 0. The rule assumes uniform, cell-centred centres; pole-centred rows (the 143 LMDZ latitudes of the CATRINE SF6 initial state, CLM, ELM, iMAPLE), truncated and Gaussian latitudes get uniform faces that do not bound their cells.
AtmosTransport.GriddedIO.VerticalAxis Type
Vertical axis of initial-condition files; coordinate variable names lev, level, plev, z, hybrid, nhym (in that order).
AtmosTransport.GriddedIO.abandon! Method
abandon!(w::CSNativeFluxWriter) -> nothingClose the writer and remove its staging file; the path keeps its previous file, if any. Abandoning a closed writer does nothing.
sourceAtmosTransport.GriddedIO.canonical_units_string Method
canonical_units_string(species::Symbol) -> StringThe units string the flux writers use for a density of species: "kg CO2 m-2 s-1" for :co2 ("kg SF6 m-2 s-1", "kg Rn222 m-2 s-1"). parse_flux_units reads it back as kg of Species(species) per m² per second. The species must be one of SPECIES_MOLAR_MASS.
AtmosTransport.GriddedIO.check_cell_area Method
check_cell_area(ds, mesh::CubedSphereMesh; file = path of ds) -> nothingThe cell-area gate of a cs_native file. The runtime multiplies the stored densities by mesh.cell_areas; a file prepared on another cube geometry (an equiangular instead of the GMAO cube, another radius) has other cell areas, and its emitted mass would be silently wrong. When the dataset ds has a cell_area variable, shape (Nc, Nc) or (Nc, Nc, 6), each panel a must satisfy isapprox(a, mesh.cell_areas; rtol = 1e-4): the norm-based relative difference ‖a − A‖ ≤ 10⁻⁴ max(‖a‖, ‖A‖) (Frobenius norm over the panel). Fill values in cell_area fail the gate; a file without cell_area passes.
This is the gate the runtime has applied since the cs_native kind exists; legacy files written with Float32-era geometry (per-cell area differences up to 1.75e-4 on C90 polar panels) pass it.
AtmosTransport.GriddedIO.check_last_month Method
check_last_month(keys, last_month::MonthKey; context = "time axis") -> keysThrow an ArgumentError unless the last of the month keys is last_month (a catalog's last_month, e.g. MonthKey(2024, 12) for TRENDY v14); return keys otherwise.
AtmosTransport.GriddedIO.check_latlon_cell_area Method
check_latlon_cell_area(area, mesh_area, units; file, rtol = 1e-2, flux = nothing) -> areaThe cell-area gate of a lat-lon flux file whose cell_area or area variable sets the inventory's total (area [m²], oriented like the flux). Its units must be square metres ("m2", "m^2", "m**2"; an empty string, for an absent attribute, is accepted), and it must agree with mesh_area, the cell areas of the mesh that the file's coordinates define on the runtime sphere, as isapprox(area, mesh_area; rtol): the norm-based relative difference ‖a − A‖ ≤ rtol max(‖a‖, ‖A‖) (Frobenius norm). The default 1 % passes any Earth-radius convention (6.371 versus 6.378137 × 10⁶ m is 0.22 % in area) and the files' own cell definitions, while areas in km² or a land fraction, which would scale the emitted total, fail. A size mismatch is a DimensionMismatch.
With flux (the flux slice, oriented like area), every cell whose flux is nonzero must have a finite, positive area: the norm test alone passes a few masked (zero) rows, such as polar rows, and their emission would be dropped without a word (per-cell units give a cell without area no flux; per-m² units give it no rate).
AtmosTransport.GriddedIO.cs_native_flux_info Method
cs_native_flux_info(path_or_ds, variable; mesh::CubedSphereMesh, slices = :)
-> (; slices::UnitRange{Int}, times::Vector{DateTime}, units::FluxUnits)The checks of read_cs_native_flux that need no flux values (layout, time axis, units, cell-area gate) on the cs_native series variable of a file path or an open NetCDF dataset, and the slices slices selects (:, a range of slice indices, or a function of the decoded times returning one): their indices, times and the parsed units. A caller allocates its arrays from length(slices) and streams the values in with read_cs_native_flux!.
AtmosTransport.GriddedIO.decode_month_keys Function
decode_month_keys(ds, name, decoders::Tuple = DEFAULT_TIME_DECODERS) -> Vector{MonthKey}The month of each stamp of the time axis name of ds, in the axis's own calendar: the flux builders' monthly keys. Classification, decoder choice and origin checks are those of decode_time. A CFDecoder axis in noleap/365_day, all_leap/366_day or 360_day gives the year and month of each calendar date without converting it to DateTime, so 30 February of a 360_day axis is February; other axes give month_keys(decode_time(ds, name, decoders)).
AtmosTransport.GriddedIO.decode_month_seconds Method
decode_month_seconds(ds, name) -> Vector{Float64}Length in seconds of the calendar month represented by each slice of time axis name. This is the conversion basis for flux variables whose units are per calendar month.
When the coordinate names a CF bounds variable, each slice is assigned from the midpoint of its bounds. For fixed-length CF units the bounds' actual duration is returned; for months since bounds, one calendar month is required. Without bounds, the coordinate stamp assigns the month. Month lengths use the coordinate's calendar (standard by default), including 360_day, 365_day/noleap, and 366_day/all_leap. Thus neither a slice's ordinal position nor the proleptic-Gregorian month length is guessed.
AtmosTransport.GriddedIO.decode_slice_intervals Method
decode_slice_intervals(ds, name) -> Vector{TimeInterval}The interval of time that each slice of the time axis name of ds stands for: known_slice_intervals when the axis defines them (CF bounds, a months since axis), else stamp_intervals of its stamps (decode_time). A series spread over several files must derive the intervals of its bare stamps from the stamps of all its files together (stamp_intervals(sort(all stamps))): one file of a daily series holds one stamp, whose period no single file tells.
AtmosTransport.GriddedIO.decode_time Function
decode_time(ds, name, decoders::Tuple = DEFAULT_TIME_DECODERS) -> Vector{DateTime}Time stamps of the time axis name of the NetCDF dataset ds. The axis is classified by the units attribute of the raw variable before any CFTime call (a months since axis never reaches CFTime), and decoded by the first decoder in decoders that accepts its class (see AbstractTimeDecoder); an ArgumentError names the decoder that is needed when none does. The length of the axis is never used to guess its meaning, and time bounds are not read.
The origin of <unit> since <origin> units must be a valid instant: an origin Y-M-D[ h:m[:s]] with a month outside 1:12, a day outside 1:31 (or beyond the month's length in the axis's calendar, for CFDecoder) or an invalid time of day is an ArgumentError, never normalised (months since 2000-13-01 is not January 2001). A months since origin must have that form.
The month-resolution decoders (MonthsSince, DecimalYears, AuxiliaryTimeVariable) return the first instant of each month; CFDecoder returns the stamps themselves. DateTime is proleptic Gregorian: a stamp of a non-Gregorian calendar whose date that calendar lacks (30 February of 360_day, 29 February of a common year in all_leap or 360_day) is an ArgumentError naming the date; decode_month_keys gives the months of such an axis.
AtmosTransport.GriddedIO.decode_time_bounds Method
decode_time_bounds(ds, name) -> Union{Nothing, Matrix{DateTime}}Decode the optional CF bounds named by time coordinate name. The returned matrix has shape (2, ntime), with interval starts in row 1 and ends in row 2. nothing means that the coordinate has no bounds attribute. Bounds inherit the coordinate's units and calendar and are validated as finite, ordered intervals before decoding. They are decoded as decode_time decodes the coordinate: a bound of a julian axis, or a standard bound before 1582-10-15, is the same instant in the proleptic Gregorian calendar (Julian 2000-01-01 is 2000-01-14), never its Julian calendar fields.
AtmosTransport.GriddedIO.find_coordinate Method
find_coordinate(ds, axis::AbstractAxis; length = nothing, required = true) -> StringName of the coordinate variable of axis in the NetCDF dataset ds: the first of the axis's candidate names (coordinate_names(axis)) that is a variable of ds. Coordinates are identified by variable name only; units, long_name, standard_name and axis attributes are never consulted, because real files mislabel them (FLUXCOM-X: lon with units = degrees_north) or omit them (GFED5 NRT).
The found variable must be one-dimensional and, when length is given, have length values (the size of the matching dimension of the data variable); otherwise an ArgumentError (not 1-D) or DimensionMismatch (wrong length) is thrown. Bounds variables (lat_bnds, lon_bnds, time_bnds) are never returned. With required = false a missing coordinate returns nothing instead of throwing.
AtmosTransport.GriddedIO.interval Method
interval(k::MonthKey) -> TimeIntervalThe month k as [first instant of k, first instant of the next month).
AtmosTransport.GriddedIO.kg_per_unit Method
kg_per_unit(u::FluxUnits, species::Symbol) -> Float64Kilograms of species per stated amount unit of u:
mass units of the
TargetSpeciesor ofSpecies(species):amount_per_unit;molar units of
Species(species):amount_per_unit × M(species);Carbon:amount_per_unit × M(species) / M(C)(mass) oramount_per_unit × M(species)(molar), for species with one carbon atom (CO₂). ForkgCand:co2this is exactlySPECIES_MOLAR_MASS.co2 / CARBON_MOLAR_MASS.
M is SPECIES_MOLAR_MASS and CARBON_MOLAR_MASS. Units of another species than species throw.
AtmosTransport.GriddedIO.known_slice_intervals Method
known_slice_intervals(ds, name) -> Union{Nothing, Vector{TimeInterval}}The intervals that the time axis name of ds defines by itself: its CF bounds (decode_time_bounds), or, for a months since axis (MonthsSince), the calendar month of each stamp. nothing for CF stamps without bounds, whose intervals depend on the whole series (stamp_intervals).
AtmosTransport.GriddedIO.latlon_orientation Method
latlon_orientation(lon, lat) -> LatLonOrientationOrientation that puts a file's latitude-longitude field into the runtime's convention: latitude ascending (south to north), and, when any longitude is negative, the longitude axis rolled to start at its first value ≥ 0 and the coordinates taken modulo 360°. A longitude axis without negative values, or with no value ≥ 0, is left as it is.
Examples: centres -179.5 … 179.5 become 0.5 … 359.5 (shift 180); the wrapped CLM-FATES axis 0 … 177.5, -180 … -2.5 keeps its order (shift 0) and becomes 0 … 357.5; latitudes 89.75 … -89.75 (VISIT) are flipped.
AtmosTransport.GriddedIO.mass_rate_factor Method
mass_rate_factor(u::FluxUnits, species::Symbol, iv::TimeInterval) -> Float64
mass_rate_factor(u::FluxUnits, species::Symbol) -> Float64Factor that turns a value in units u into kg of species per second (per m² or per cell, as u.area says): kg_per_unit(u, species) /period_seconds(u, iv). The form without iv is for units whose time unit does not depend on the calendar (not per month). For per-second units it is kg_per_unit exactly.
AtmosTransport.GriddedIO.month_keys Method
month_keys(times) -> Vector{MonthKey}The calendar month of each time stamp.
sourceAtmosTransport.GriddedIO.open_cs_native_writer Method
open_cs_native_writer(path, mesh::CubedSphereMesh, intervals::Vector{TimeInterval},
variables; species::Symbol, chunk_time::Integer, deflate = 1,
attributes = Dict(), config_text = nothing) -> CSNativeFluxWriter
open_cs_native_writer(f, path, mesh, intervals, variables; kwargs...) -> StringOpen a cs_native surface-flux file for mesh with one slice per interval. variables lists the flux series, each a name or name => attributes (for example "GPP_CO2_FLUX" => Dict("long_name" => "gross primary production")). The file is written to a staging file next to path and appears at path only when publish! succeeds; a relative path is taken relative to the working directory at opening. The do-block form calls f(writer), then publish!, and returns the absolute path; if f throws, the file is abandoned.
The file (NetCDF-4):
Geometry from
Output.define_cs_horizontal_geometry!(ds, mesh): dimensionsXdim,Ydim,Xcorner,Ycorner,nf; cell centreslons/lats, corners,cell_area = mesh.cell_areas(Float64, m²) and thecubed_spheregrid mapping withsemi_major_axis = mesh.radius.Time: the
intervalsmust be contiguous (stopof one isstartof the next).timeholds each interval's start (Float64,"hours since <first day> 00:00:00",calendar = "proleptic_gregorian"),time_bnds(nv, time)its start and stop. Slices are start-stamped, so the runtime's stepwise scheme holds each slice for exactly its interval.Series: one Float32 variable per entry of
variables, dimensionsCS_NATIVE_FLUX_DIMS,units = canonical_units_string(species)("kg CO2 m-2 s-1"for:co2),sign_convention = "positive into the atmosphere",grid_mapping = "cubed_sphere",coordinates = "lons lats", and no_FillValue. Chunks are(Nc, Nc, 1, chunk_time)(Julia order;chunk_timeis clamped to the number of slices), deflate leveldeflatewith shuffle:chunk_time = 1for daily files and24for hourly files (the P0 read benchmarks, design Section 2.4).Global attributes:
Conventions = "CF-1.8",species, the provenance block ofOutput.define_provenance_attributes!(itscreation_dateis the publication time),config_textandconfig_sha1(SHA-1 of the text) whenconfig_textis given, thenattributes. A caller attribute may not replace one the writer sets.
species must be a key of SPECIES_MOLAR_MASS.
AtmosTransport.GriddedIO.orient Method
orient(A::AbstractArray, o::LatLonOrientation) -> ArrayA new array with the two leading dimensions (lon, lat) of A reordered by o (longitude rolled, latitude flipped); trailing dimensions (time, level, ...) are untouched. Values are copied, never computed, so the result is bitwise the input in the new order.
AtmosTransport.GriddedIO.orient_coordinates Method
orient_coordinates(lon, lat, o::LatLonOrientation) -> (lon, lat)The coordinates that match orient(A, o), as Vector{Float64}: latitude reversed when o.flip_lat; longitude rolled by o.lon_shift and, when o.wrap_lon, taken modulo 360° (mod(λ, 360)).
AtmosTransport.GriddedIO.parse_flux_units Method
parse_flux_units(s; substance = nothing, molar_default = nothing) -> FluxUnitsParse the flux units string s. Case, whitespace, ./* separators and LaTeX exponent markup (m$^{-2}$) are ignored. The grammar is
<amount>[<substance>]<factors>[, <qualifier>]amount:
kg,g,tonnes,mol,mmol,µmol/umol;substance (optional, right after the amount):
C(carbon) or a species name such asCO2,(CO2);factors, in any order: an area (
m-2,/m2,cell-1) and a time (s-1,/s,d-1,/day,month-1,/month,per month,yr-1,/year); without an area the flux is per cell. A time is required, except forTonnes, the EDGAR convention for tonnes per cell and year;qualifier: only
per unit land area occupied by the PFT(SDGVMra).
substance names the substance when the string does not (TRENDY files write kg m-2 s-1 for kg of carbon: substance = :carbon; ECCO-Darwin writes mmol m-2 s-1 of CO₂: substance = :co2). :c/:carbon give Carbon, other names a Species. A substance named both in the string and by the keyword must agree. Without either, mass units are of the TargetSpecies; molar units without a substance take molar_default (the runtime loaders pass the tracer's species, so mmol m-2 s-1 is moles of that species) and throw when it is nothing.
Examples: "kg m-2 s-1", "kg m-2 s-1", "kg.m-2.s-1", "kg m$^{-2}$ s$^{-1}$", "kgC m-2 s-1", "gC m-2 d-1", "g C per month" (per cell), "kgCO2/month/m2", "Tonnes", "mmol m-2 s-1" (with substance), and the runtime's "kg/m2/s", "kg(CO2) m^-2 s^-1", "kg/m²/s". Anything else throws an ArgumentError.
AtmosTransport.GriddedIO.period_seconds Method
period_seconds(u::FluxUnits, iv::TimeInterval) -> Float64
period_seconds(u::FluxUnits) -> Float64Length in seconds of the time unit of u: 1 (per second), 86400 (per day), DAYS_PER_JULIAN_YEAR × 86400 (per year), or the length of the calendar month that contains the interval iv (per month; iv must lie in one month, and the form without iv throws).
AtmosTransport.GriddedIO.publish! Method
publish!(w::CSNativeFluxWriter) -> StringComplete the file and publish it at its path by one atomic rename (AtomicFiles.commit_atomic_staging); return the absolute path. Every slice of every series must have been written; otherwise, and on any other error, the staging file is removed (the writer is abandoned) and the error is thrown. The provenance creation_date is set to the publication time.
AtmosTransport.GriddedIO.read_cs_native_flux! Function
read_cs_native_flux!(dest_panels, path_or_ds, variable, slices = :; mesh::CubedSphereMesh)
-> (; slices::UnitRange{Int}, times::Vector{DateTime}, units::FluxUnits)Read slices slices of the cs_native series variable (of a file path or an open NetCDF dataset) panel by panel into the caller's arrays: dest_panels[p], size (Nc, Nc, length(slices)), receives panel p, each stored value converted once to its element type (FT(x), as FT.(ds[variable][:, :, p, slices]) does). Beyond the checks it allocates one panel-slab read buffer (two when the variable declares fill values or packing), so a caller that fills its final arrays holds the series once, never the whole cube twice. This is the runtime's loader (surface_flux.kind = "cs_native").
Every check of read_cs_native_flux applies (cs_native_flux_info runs first); a value that is not finite once converted (a Float64 value beyond the Float32 range read into Float32) is an error too. slices is any selection cs_native_flux_info takes. Returns the slices read, their times and the parsed units; the caller converts with mass_rate_factor(units, species) and the mesh cell areas.
AtmosTransport.GriddedIO.read_cs_native_flux Method
read_cs_native_flux(path, variable; mesh::CubedSphereMesh, slices = :)
-> (; density, times::Vector{DateTime}, units::FluxUnits)Read the cs_native flux series variable of the file path for the runtime mesh mesh into one array, the whole-file form of read_cs_native_flux! (the runtime streams panel by panel into its own arrays instead). The surface-flux builder uses it for its readback.
Checks, each an error when it fails:
the variable has dimensions
CS_NATIVE_FLUX_DIMSwithXdim = Ydim = mesh.Nc, six panels and at least one slice;its time axis (found by
find_coordinate) decodes withdecode_timeand the default decoders, and is strictly increasing;its
unitsparse (parse_flux_units) as kg of a species or of carbon per m² per second (kg CO2 m-2 s-1,kg m-2 s-1,kgC m-2 s-1); gram, molar, per-cell, other time units and missing units are refused;a stored
cell_areapassescheck_cell_area;no value of the slices read is fill (
_FillValueormissing_value) or non-finite.
slices selects the slices to read: : (all), a range of slice indices, or a function of the decoded times (Vector{DateTime}) returning such a range, so a caller can select by time without opening the file twice.
Returns density (Julia order (Xdim, Ydim, nf, slice), in the stored floating-point type: Float32 for files of open_cs_native_writer), the times of the slices read, and the parsed units. The caller turns densities into kg of its species per second per cell with mass_rate_factor(units, species) and the mesh cell areas.
AtmosTransport.GriddedIO.read_model_surface_height Method
read_model_surface_height(path, mesh) -> Output.ModelSurfaceHeightRead the per-grid model surface height dataset written by write_model_surface_height for mesh (LatLonMesh, ReducedGaussianMesh or CubedSphereMesh). The variable surface_height must be in metres; its stored cell centres must be the mesh's (unit-sphere chord within 1e-7, about 6e-6°, of the Float64 centres on the cubed sphere; within 8 eps of a Float32 lat-lon or reduced-Gaussian mesh), a stored cubed-sphere cell_area must pass check_cell_area, and every value must be finite. Returns the field in the mesh's native layout (a matrix, a vector of cells, or six panels) as Output.ModelSurfaceHeight, which the observation sampler queries per cell.
AtmosTransport.GriddedIO.read_point_elevation Method
read_point_elevation(path; variable = "z") -> LatLonPointElevationRead a lat-lon DEM for point lookups. The file holds 1-D longitude and latitude centre coordinates (found by name, find_coordinate) and a 2-D variable variable on those two coordinate dimensions, with the ground elevation in metres above sea level; units must say metres. The axes are identified by the variable's dimension names: (lon, lat) in Julia order (CF/NetCDF order (lat, lon), the usual layout) or the transpose, which is read transposed back; any other dimensions are an error. Fill values (_FillValue, missing_value) become NaN. Latitudes may be stored in either order (they are flipped to ascending with orient); longitudes must be strictly ascending in any convention (−180…180, 0…360, or a regional window). A gridline-registered global DEM, whose first and last longitudes are the same meridian 360° apart, has its last column dropped. The DEM must give the land or water surface: over the ocean the ground is the sea surface (0 m), so a bathymetric product (ETOPO) must be clipped at 0 over the sea first; a value below −450 m (deeper than the Dead Sea shore) is rejected as bathymetry or an undeclared fill value, and a value above 9000 m (higher than Mount Everest) as an undeclared fill value.
The whole variable is read into memory as Float32: about 0.9 GB for a global 1-arc-minute grid (21600 × 10800).
sourceAtmosTransport.GriddedIO.read_raw Method
read_raw([T = Float64,] ds, name, policy::AbstractFillPolicy, idx...) -> Array{T}Values of variable name of the NetCDF dataset ds at indices idx (all values when idx is empty), as element type T (Float64 by default), with fill values replaced by the rule of policy (MaskedAsZero, MaskedAsNaN, FillValueOnly). Indices follow the Julia (reversed CF) dimension order, as ds[name][idx...]. All-integer indices give a 0-dimensional array.
Each stored value is converted once, straight to T, so a Float32 run reads into Float32 without a Float64 copy, and MaskedAsZero reproduces T.(nomissing(ds[name][idx...], zero(T))) bit for bit.
AtmosTransport.GriddedIO.seconds_since Method
seconds_since(t, reference::DateTime) -> Float64
seconds_since(times, reference::DateTime) -> Vector{Float64}Seconds from reference to each time stamp, Dates.value(t - reference) / 1000 (millisecond resolution, the resolution of DateTime); negative before reference. This is the runtime's slice-time convention.
AtmosTransport.GriddedIO.source_latlon_mesh Function
source_latlon_mesh(lon, lat, rule = SnappedFaces(); radius) -> LatLonMesh{Float64}
source_latlon_mesh(ds, rule = SnappedFaces(); radius) -> LatLonMesh{Float64}The latitude-longitude mesh of a gridded input file whose cell centres are lon and lat [degrees], with faces derived by the face rule rule (SnappedFaces, described below, or UniformSpanFaces). The second form reads the centres of the dataset's coordinates (find_coordinate). radius [m] has no default: the source mesh must sit on the sphere of the mesh it is regridded to.
Under SnappedFaces, the mesh is built on the explicit-face constructor LatLonMesh(λᶠ, φᶠ; radius); latitude rows of a descending axis are stored ascending (Grids.latitude_row_order gives the data permutation), and longitude columns keep their input order (a wrapped axis is unwrapped, not rolled).
Faces under SnappedFaces
Regular global axes: the snapping rule. For n stored centres x_k, k = 0 … n-1, with res = 180/n (latitude) or 360/n (longitude) and the origin x₀ = -90 (ascending latitude), +90 (descending latitude), -180 (longitude with a negative centre) or 0 (longitude without one), direction s = ±1:
k̂ = round((x_k − x₀)/(s·res) − ½); requirek̂ = kand|(x_k − x₀)/(s·res) − ½ − k̂| ≤ 1e-6cells plus the storage rounding of the centres,2 eps(T) max|x| / rescells for storage precisionT(Float64 values that are all Float32 numbers, for longitudes also after subtracting 360°, count as Float32: GridFED's Float32 0.1° centres sit up to 3e-5 cells off). Otherwise anArgumentError: a half-cell offset or a wrong resolution.The faces are
x₀ + s·j·res,j = 0 … n, built from integers (the correctly rounded ratio(x₀ n + s j 180)/n), so the outer faces are exactly±90and-180/180(or0/360).
A longitude axis may start anywhere on that grid (after a roll or a wrap); a global axis whose centres sit on multiples of res (a centre on 0° or ±180°) gets faces at the centres ± res/2, so its first face is not ±180 or 0 (CLASSIC -180.5°, CLM -0.625°, CLM-FATES and ELM-FATES -1.25°, ELM -180.625°, JSBACH -180.9375°). Any other registration throws.
Other latitude axes (pole-centred, truncated, Gaussian, regional): faces at the midpoints of adjacent centres. An outer face is the pole when the outer centre is closer to the pole than the adjacent spacing (pole-centred rows of CLM, ELM and iMAPLE, whose polar cells are half-width caps; the Gaussian JSBACH rows, whose midpoint faces with polar outer faces reproduce the Python audit); otherwise it is the outer centre minus half the adjacent spacing (the FATES grids, truncated south of their row at -59.684°: southern face -60.6316°). Such meshes have non-uniform latitude faces (Δφ = NaN), so dx, dy and other Δφ-based metrics throw.
Regional longitude axes (spanning less than 360°): uniform faces from the first centre minus to the last centre plus half the mean spacing.
Faces are never taken from lat_bnds/lon_bnds.
AtmosTransport.GriddedIO.stage_input Method
stage_input(path; staging_dir) -> StringA path at which the input file path can be opened as NetCDF:
an uncompressed file (any name not ending in
.gz,.tar.gzor.tgz) is returned as it is;*.gzis decompressed withgzip -dc;*.tar.gz/*.tgzmust hold exactly one file, extracted withtar -xzOf.
The decompressed file is written to staging_dir (created if needed; there is no default, in particular not /tmp) as <key>-<name>, where key is the SHA-1 of the source's absolute path, size and modification time, so a staged file is reused until its source changes. It is published with AtomicFiles.atomic_publish only after it decompressed completely and starts with a NetCDF signature, so a concurrent or interrupted stager never leaves a partial file. gzip and tar must be on the PATH. Staged files are not removed; the staging directory is the caller's.
AtmosTransport.GriddedIO.stamp_intervals Method
stamp_intervals(stamps; source = "the time stamps") -> Vector{TimeInterval}The interval each of the strictly increasing interval-start stamps (the bare stamps of a whole series, merged over its files) stands for. A series has a regular step, and each slice holds for that step:
Month starts (every stamp the first instant of a month): a slice holds until the next stamp when that is one month later, or one year later from a 1 January to the next 1 January (an annual series, also after a monthly part); the last stamp holds for one calendar month, or one year when the last spacing is such a year. A slice before any other spacing (Jan 1 then Mar 1) holds for its own month only, so the months in between are a gap.
Other stamps: the step is the smallest spacing; a slice holds until the next stamp when the spacing is that step (within
STAMP_SPACING_TOLERANCE), else for one step (a gap follows), and the last stamp holds for one step.
A gap is not an error here: the coverage check of the surface-flux loaders refuses a run only when the run crosses it. These are ArgumentErrors, since the period of a slice cannot be inferred:
a single stamp: one stamp gives no spacing, and a 1 January 00Z stamp may stand for a day, a month or a year;
stamps about one month (28 to 31 days) or one year (365 to 366 days) apart that are not all month starts, e.g. mid-month stamps (Jan 15, Feb 15, …): whether each slice holds from its stamp or is centred on it is unknown;
stamps that do not increase strictly (a stamp in two files).
The remedy is CF bounds on the time coordinate, or interval-start stamps. source names the stamps in these messages.
AtmosTransport.GriddedIO.write_latlon_flux Method
write_latlon_flux(path, mesh::LatLonMesh, variables; species::Symbol,
attributes = Dict(), deflate = 1, config_text = nothing) -> StringWrite a static latitude-longitude surface-flux file for surface_flux.kind = "file" atomically (AtomicFiles.atomic_publish) and return path.
Static only: each variable is one field
(lon, lat), with no time dimension. The runtime'skind = "file"loader reads one static field per tracer and is not a time-varying kind, so a series would never be read as one; time-varying fluxes on the cubed sphere go tocs_nativefiles (open_cs_native_writer).variableslists the fields asname => dataorname => (data, attributes);datais Float32 kg ofspeciesper m² per second, positive into the atmosphere, size(Nx, Ny), Julia order(lon, lat). Every value must be finite; the file stores no_FillValue.The mesh must be one the runtime rebuilds exactly. The runtime takes the cell faces from the written centres (oriented, then
SnappedFaces), never fromlon_bnds/lat_bnds. A mesh whose faces that rule does not reproduce bit for bit is refused with anArgumentError: its file would be regridded on other cells.Coordinates:
lon,lat(Float64 cell centresmesh.λᶜ,mesh.φᶜ) withlon_bnds,lat_bnds(the mesh faces) andcell_area(lon, lat)fromcell_area(mesh, i, j)[m²].Each field:
units = canonical_units_string(species),sign_convention = "positive into the atmosphere",cell_measures = "area: cell_area"; chunks(Nx, Ny), deflate leveldeflatewith shuffle.Global attributes as for the
cs_nativewriter:Conventions,species, provenance,config_text/config_sha1when given, thenattributes.
AtmosTransport.GriddedIO.write_model_surface_height Method
write_model_surface_height(path, mesh, field; attributes = Dict()) -> StringWrite the per-grid model surface height dataset atomically (AtomicFiles.atomic_publish) and return path. field holds the surface height in m above sea level, finite everywhere, in the mesh's layout: (Nx, Ny) on a LatLonMesh, (ncells,) on a ReducedGaussianMesh, (Nc, Nc, 6) on a CubedSphereMesh (panels in the mesh's convention).
The file holds the Float64 variable surface_height (units = "m", standard_name = "surface_altitude") and the mesh geometry: lat-lon lon, lat, bounds and cell_area; cubed-sphere Xdim/Ydim/nf, lons/lats, corners, cell_area and the cubed_sphere mapping (Output.define_cs_horizontal_geometry!); reduced Gaussian cell with longitude, latitude and cell_area. Global attributes: Conventions, title, the provenance block, then attributes (for example the source file and the regridding). read_model_surface_height reads it back.
AtmosTransport.GriddedIO.write_slices! Method
write_slices!(w::CSNativeFluxWriter, name, k::AbstractUnitRange,
density::AbstractArray{Float32, 4}) -> wWrite the densities density, size (Nc, Nc, 6, length(k)), into slices k of the series name (kg of the species per m² per second, Julia order (Xdim, Ydim, nf, time)). Every value must be finite (an ArgumentError names the first that is not); the file stores no fill values. Slices may be written in any order and rewritten; publish! requires every slice of every series.