# Done-Test Checks: Data Pipelines and Integrations

**Check library, version 1.** Rows to lift into the [done-test template](https://mangrove.co.uk/templates/done-test-template.md). Take what applies, delete the rest, and put a number in every threshold.

## The problem this solves

Integration work passes its demo and fails in production more reliably than anything else, because a demo runs once, on a good day, against data somebody chose. Production runs every day against data nobody chose, while the far end changes without telling you.

So the checks worth writing are about **what happens on the bad day**: the malformed record, the duplicate delivery, the source that changed shape, the run that silently did nothing.

## Correctness

| ID | Check | Evidence | Pass |
|---|---|---|---|
| DI1 | For [named period], record counts reconcile between source and destination, and [named spot-checks] match field by field | Reconciliation report + live spot-check | |
| DI2 | Transformation rules are documented per field, and each is demonstrated on a real record | Live walkthrough | |
| DI3 | Records that fail validation are quarantined with a reason, visible and re-processable, never dropped | Live with a deliberately bad record | |
| DI4 | Re-running the same input produces the same result and no duplicates | Live replay | |

**On DI4.** Idempotency is the row that lets anyone safely re-run a failed job at 2am. Without it, recovery needs the person who built it.

## Robustness

| ID | Check | Evidence | Pass |
|---|---|---|---|
| DR1 | A schema change at the source ([added, removed or retyped field]) fails loudly and specifically, rather than corrupting data silently | Live with a changed payload | |
| DR2 | When the source is unavailable or rate-limited, the pipeline backs off, retries to [N], and alerts on exhaustion | Live simulated outage | |
| DR3 | Partial failure leaves the destination in a defined state: [transactional, or resumable from the last checkpoint] | Live interrupted run | |
| DR4 | Late-arriving and out-of-order records are handled as specified | Live | |

## Operations

| ID | Check | Evidence | Pass |
|---|---|---|---|
| DO1 | A run that fails, or fails to start, alerts [named recipient] within [N]. Silence is never treated as success | Live suppressed run | |
| DO2 | Freshness is monitored: data older than [N] raises an alert | Live | |
| DO3 | Every record's origin, run and transformation version is traceable | Live query on a chosen record | |
| DO4 | Backfilling [period] is a documented procedure a client engineer can run unaided | Performed live by that engineer | |
| DO5 | Cost per run at production volume is at or under £[X] | Billing or metrics shown | |

## Contracts and compliance

| ID | Check | Evidence | Pass |
|---|---|---|---|
| DC1 | Credentials for each system are held by the client, scoped to least privilege, and rotated at handover | Access review | |
| DC2 | Personal data crossing a boundary is documented in the data map, with legal basis and destination recorded | Data map | |
| DC3 | Data retention at the destination matches the agreed period, and deletion propagates | Live walkthrough | |