Skip to content

3. Observational data standardization and final preparation for downstream use

Objective: Take QC’d CHIRPS and the shared target grid + mask, produce a VECTRI-friendly daily dataset, split calibration / validation / operational periods, and archive a versioned “ready-to-use” observational product.

Companion notebook: 05_observational_standardization_vectri.ipynb

Prerequisites

pip install xarray netCDF4 numpy pandas matplotlib

Step 1 — Regrid (if needed) and apply land mask

  1. Interpolated regrid: xarray interp(lat=..., lon=...) on the target grid (quick; not conservative). For production precipitation, consider xesmf or conservative weights — document method.
  2. Apply mask: where(mask > 0) or multiply precip by mask.
  3. Preserve NaN over excluded cells.
ds_t = ds.interp(lat=target_lat, lon=target_lon, method="linear")
ds_t["precip"] = ds_t["precip"].where(mask > 0)

Step 2 — VECTRI-compatible structure

VECTRI drivers differ by version; common needs:

  • Daily timestep, monotonic time
  • Precipitation in mm/day
  • Dimensions named consistently (lat, lon, time)
  • CF-style coordinates and bounds if your VECTRI preprocessor expects them

Export a thin wrapper NetCDF or the exact namelist-driven format your compile expects — validate with a one-year smoke run.

Step 3 — Temporal splits

Period Years (example policy) Use
Calibration 1993–2016 Bias correction / parameter fitting
Validation 2017–2025 Out-of-sample verification
Operational 2026+ Near-real-time / outlook workflow 
cal = ds.sel(time=slice("1993-01-01", "2016-12-31"))
val = ds.sel(time=slice("2017-01-01", "2025-12-31"))
ops = ds.sel(time=slice("2026-01-01", "2030-12-31"))  # extend as data exist

Document any partial years at slice boundaries.

Step 4 — Baseline climatology and anomalies

  1. Choose a baseline window (e.g. 1993–2016 to match calibration, or WMO 1991–2020 — be consistent across variables).
  2. Compute climatology (e.g. day-of-year mean or monthly mean).
  3. Anomaly = value − climatology (align time.dayofyear or calendar month).

Store climatology in a separate file for reuse in forecast verification.

Step 5 — Versioned archive

Naming example:

processed/chirps/ethiopia/
  chirps_v2p0_p25_ethiopia_masked_cal_1993_2016_v1.0.nc
  chirps_v2p0_p25_ethiopia_masked_val_2017_2025_v1.0.nc
  chirps_v2p0_p25_ethiopia_clim_doy_1993_2016_v1.0.nc

Include processing_version, git_commit (if applicable), and input_manifest path in global attributes.

Next

4. Documentation, versioning, and reproducibility.

In Partnership With