AtmosTransport.jl

Appearance

DataDownloads API ​

The download module turns source/protocol TOML configuration into verified ERA5, GEOS-FP, GEOS-IT, and MERRA-2 input files. See Data sources for the configuration-oriented guide.

AtmosTransport.DataDownloads Module
julia
Downloads

TOML-driven download pipeline for meteorological and emissions data.

Provides a unified entry point for downloading ERA5, GEOS-FP, GEOS-IT, and MERRA-2 data. All download parameters are specified in TOML config files; the pipeline dispatches on source type and access protocol.

Architecture

Two orthogonal dispatch axes:

julia
AbstractDownloadSource    — what data to download
├── ERA5Source            — ECMWF ERA5 reanalysis
├── GEOSFPSource          — NASA GEOS-FP forward processing
├── GEOSITSource          — NASA GEOS-IT retrospective
└── MERRA2Source           — NASA MERRA-2 reanalysis

AbstractDownloadProtocol  — how to download it
├── CDSProtocol           — CDS API (Python subprocess)
├── MARSProtocol          — MARS API (Python subprocess, CDS fallback)
├── OPeNDAPProtocol       — NCDatasets remote subset
├── HTTPProtocol          — Direct HTTP with Content-Length verification
└── S3Protocol            — AWS S3 CLI

Usage

bash
julia --project=. scripts/downloads/download_data.jl config/downloads/era5_native_monthly.toml \
    [--start 2021-12-01] [--end 2021-12-31] [--dry-run] [--verify]

Output paths follow the canonical Data Layout hierarchy: <data_root>/met/<source>/<grid>/<cadence>/<payload>/

source
AtmosTransport.DataDownloads.CDSProtocol Type
julia
CDSProtocol

Copernicus Climate Data Store API (Python cdsapi subprocess). Requires ~/.cdsapirc.

source
AtmosTransport.DataDownloads.DownloadConfig Type
julia
DownloadConfig{S, P}

Complete download configuration parsed from a recipe TOML. Parameterized on source and protocol for concrete dispatch.

source
AtmosTransport.DataDownloads.ERA5Source Type
julia
ERA5Source

ERA5 reanalysis from ECMWF. Supports spectral GRIB, gridded model-level, and surface fields. Monthly chunking is the production default.

source
AtmosTransport.DataDownloads.GEOSFPSource Type
julia
GEOSFPSource

GEOS-FP forward processing from NASA GMAO. Native cubed-sphere (C720) or regridded lat-lon (0.25° × 0.3125°).

source
AtmosTransport.DataDownloads.GEOSITSource Type
julia
GEOSITSource

GEOS-IT retrospective from NASA GMAO. Native cubed-sphere (C180).

source
AtmosTransport.DataDownloads.HTTPProtocol Type
julia
HTTPProtocol

Direct HTTP file download via Downloads.jl with Content-Length verification.

source
AtmosTransport.DataDownloads.MARSProtocol Type
julia
MARSProtocol

ECMWF MARS API (Python ecmwfapi subprocess). Falls back to CDS if MARS credentials are unavailable. Requires ~/.ecmwfapirc.

source
AtmosTransport.DataDownloads.MERRA2Source Type
julia
MERRA2Source

MERRA-2 retrospective reanalysis (1980–present). 0.5° × 0.625° lat-lon.

source
AtmosTransport.DataDownloads.OPeNDAPProtocol Type
julia
OPeNDAPProtocol

OPeNDAP remote subset via NCDatasets.jl. No Python dependency.

source
AtmosTransport.DataDownloads.PythonEnvironment Type
julia
PythonEnvironment

Cached Python interpreter configuration. Detected once, reused for all CDS/MARS API calls in the session.

source
AtmosTransport.DataDownloads.S3Protocol Type
julia
S3Protocol

AWS S3 download via aws s3 cp subprocess.

source
AtmosTransport.DataDownloads.detect_python_env Function
julia
detect_python_env(python_path="python3") -> PythonEnvironment

Probe the Python installation for available packages and API credentials. Called once during module initialization.

source
AtmosTransport.DataDownloads.download_data! Method
julia
download_data!(cfg::Dict{String,Any}; start_date=nothing, end_date=nothing,
               dry_run=false, verify_only=false)

Main entry point. Parses TOML config and executes downloads.

  • dry_run: print what would be downloaded without executing

  • verify_only: check existing files against expected sizes

source
AtmosTransport.DataDownloads.parse_download_config Method
julia
parse_download_config(cfg::Dict{String,Any}) -> DownloadConfig

Parse a download recipe TOML into a fully resolved DownloadConfig.

source
AtmosTransport.DataDownloads.verified_download Method
julia
verified_download(url, dest; max_retries=3) -> Bool

Download url to dest with Content-Length integrity checking.

  1. HTTP HEAD → Content-Length

  2. If file exists with matching size → skip (verified)

  3. If exists but wrong size → retain it until a verified replacement is ready

  4. After download: verify local size matches Content-Length

  5. If mismatch: delete corrupt file and retry

source
AtmosTransport.DataDownloads.verify_downloads Method
julia
verify_downloads(dir, dates, file_pattern; url_builder=nothing) -> NamedTuple

Scan a download directory for missing or corrupt files. Returns (; ok, corrupt, missing).

source