1. CHIRPS daily ingestion, preprocessing, and quality control (QC)¶

This lesson follows Session 1.2 in Module 1. You should already have a merged or yearly CHIRPS NetCDF from Download CHIRPS rainfall.

Objective: Ingest raw CHIRPS data, run quality assurance, standardize metadata and structure, and produce diagnostics so the rainfall baseline is trustworthy for calibration and VECTRI.

Companion notebook: 03_chirps_ingestion_qc.ipynb (run order: after CHIRPS download).

Prerequisites¶

pip install xarray netCDF4 numpy pandas matplotlib scipy cftime

Step 1 — Standardize naming and embed metadata¶

Goal: Predictable filenames and CF-style global attributes for traceability.

1. File naming (example convention):
   chirps_v2p0_p25_ethiopia_raw_{start}_{end}.nc → after QC: chirps_v2p0_p25_ethiopia_qc_{start}_{end}.nc
2. Global attributes to set or overwrite on the Dataset:
3. title, source, institution
4. references (CHIRPS citation URL)
5. history (append line with UTC timestamp and tool version)
6. Conventions = CF-1.8 (or your target)
7. Variable attributes on precipitation:
8. long_name, standard_name (e.g. lwe_thickness_of_precipitation_amount if appropriate)
9. units = mm day-1 (after conversion if needed)

Usage (pattern):

import xarray as xr
from pathlib import Path
from datetime import datetime, timezone

path_in = Path("data/chirps_notebook_demo/chirps-v2.0.2020.days_p25_clip.nc")
ds = xr.open_dataset(path_in)

precip_var = next(v for v in ds.data_vars if "precip" in v.lower())
ds = ds.rename({precip_var: "precip"})

ds["precip"].attrs.update(
    long_name="daily precipitation",
    units="mm day-1",
)
ds.attrs.update(
    title="CHIRPS v2.0 daily precipitation (Ethiopia box)",
    source="CHIRPS v2.0",
    processing_date=datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
)

Step 2 — Temporal QA (missing days, duplicates, continuity)¶

1. Decode time with xr.decode_cf if needed.
2. Sort by time; drop duplicate timestamps (keep first or flag).
3. Build a complete daily range for the expected period; compare to actual times.
4. Record missing days as a list or count per year.

Checks:

• No duplicate time indices
• Monotonic increasing time
• Expected calendar (standard Gregorian for CHIRPS daily global NetCDF)

import pandas as pd

time = ds["time"].values
dups = ds.indexes["time"].duplicated()
assert not dups.any(), "Duplicate timesteps present"

full = pd.date_range(ds.time.values.min(), ds.time.values.max(), freq="D")
missing = full.difference(pd.to_datetime(ds.time.values))
# len(missing) -> report in QC

Step 3 — Value QA (invalid and extreme rainfall)¶

1. Negative values: CHIRPS should be ≥ 0; flag or set to NaN with valid_range in attrs.
2. Physical plausibility: optional upper cap (e.g. daily max threshold for Ethiopia) — document choice; use for flags, not silent deletion unless policy says so.
3. Wet-day definition: e.g. precip ≥ 0.1 mm for wet-day frequency.

import numpy as np

p = ds["precip"]
neg = (p < 0).sum().item()
extreme = (p > 300).sum().item()  # example threshold mm/day — adjust for domain

Step 4 — Units, calendar, and time encoding¶

1. Confirm units are mm/day (CHIRPS daily files are typically already mm/day — verify units attr).
2. Ensure time has units and calendar attributes compatible with CF.
3. If you change units in code, update attrs and history.

Use xarray.encode_cf() before writing if you need consistent encoding.

Step 5 — Missing data policy¶

Default for observational baseline: keep missing as NaN; document coverage in the QC report.

• Interpolation along time is generally avoided for daily rainfall used in epidemiological/hydrological models unless you have an approved gap-fill method.
• Add a quality flag variable (optional), e.g. qc_flag(time, lat, lon) with bitmask for negative, suspected outlier, ocean masked, etc.

Step 6 — Spatial clip (Ethiopia) and coordinate checks¶

If your file is still global, clip to your Ethiopia bounding box (with buffer if you plan regridding). Align names to lat, lon (south to north, west to east).

N, S, W, E = 15.0, 3.0, 33.0, 48.0  # add buffer if needed, e.g. 0.5°
ds_box = ds.sel(lat=slice(S, N), lon=slice(W, E))

Verify lat increasing, lon order matches your mask/forecast products.

Step 7 — Diagnostic plots and maps¶

Generate and save figures under e.g. reports/qc_chirps/figures/:

Optional: generalized ESD (e.g. per-pixel quantiles) or Peaks-over-threshold for extremes — document methodology.

Step 8 — QC report (HTML / PDF / Markdown)¶

Export a single Markdown or HTML file under reports/qc_chirps/ that includes:

• Input path, date range, grid shape, CHIRPS version
• Summary stats (mean, std, wet-day fraction, min/max)
• Counts: negative values, duplicates, missing days, flagged outliers
• Embedded or linked figure paths
• Recommendations (approve for pipeline / fix source / exclude period)

Pandoc can convert Markdown → PDF if needed: pandoc qc_report.md -o qc_report.pdf.

Step 9 — Archive: raw vs processed¶

Keep immutable copies of provider files (raw/ or read-only mirror). Write processed outputs to processed/chirps/ with versioned names, e.g.:

• chirps_v2p0_p25_ethiopia_qc_1993_2020.nc

Include a small manifest.json or YAML listing source checksums and processing command.

Next¶

Define the shared spatial framework: 2. Ethiopia target grid and land mask.