Skip to content

4. Documentation, versioning, and reproducibility setup

Objective: Make the observational pipeline auditable: short processing report, version-controlled configs and masks, clear folder layout, logging for automated runs, and optional summary statistics by region.

Companion notebook: 06_documentation_reproducibility.ipynb

Step 1 — Processing report (short form)

Create reports/chirps_pipeline_report.md (or .pdf via Pandoc) summarizing:

  1. Data sources (CHIRPS version, download date, files)
  2. QC outcomes (link to Section 1 report)
  3. Grid and mask (link to Section 2 config files)
  4. Standardization (regridding method, splits, climatology baseline)
  5. Outputs (list of NetCDF paths + versions)
  6. Known limitations and next actions

Use a template so each re-run only updates a YAML front matter or JSON block.

Step 2 — Version control

Track in Git (or equivalent):

  • All Python / Jupyter sources under docs/day6/notebooks/ and docs/scripts/
  • JSON/YAML domain and grid configs
  • Small mask NetCDFs if policy allows; otherwise store checksum + download instructions for large binaries

Exclude huge raw CHIRPS archives via .gitignore; commit manifests instead.

Step 3 — Folder structure and logging

Suggested layout:

data/
  raw/chirps/              # immutable provider files (optional local mirror)
  processed/chirps/        # QC + masked + split products
  ancillary/grid_mask/     # target grid, land mask, configs
reports/
  qc_chirps/               # QC figures + qc_report.md
  logs/                    # pipeline run logs

Logging (standard library example):

import logging
from pathlib import Path

Path("reports/logs").mkdir(parents=True, exist_ok=True)
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    handlers=[
        logging.FileHandler("reports/logs/chirps_pipeline.log"),
        logging.StreamHandler(),
    ],
)
logging.info("QC step completed: %s", path_out)

Rotate logs in operations if runs are frequent.

Step 4 — (Optional) Regional summary statistics

For reporting or bulletins:

  1. Zonal averages over admin regions (GeoJSON + regionmask / geopandas overlay).
  2. Metrics: mean seasonal rainfall (MAM, JJAS), wet-day frequency, simple SPI-style indices if needed later.

Document coordinate reference alignment between raster and vector layers.

Step 5 — Reproducibility checklist

  • Fixed package versions in requirements.txt or environment.yml
  • Random seeds (if any stochastic steps) recorded
  • history attribute appended on each NetCDF write
  • Report references notebook and commit hash used for the run

Module wrap-up

You now have a traceable path from download → QC → grid/mask → standardized obs → documented archive, aligned with Module 1 sessions 1.2–1.5.

In Partnership With