Moving Data Between Things That Do Not Talk

A script is not an integration

A script that reads from one system and writes to another works until something is unavailable, something is malformed, or something is retried. Then it either loses data or duplicates it.

The difference between a script and an integration is entirely in how it behaves when something goes wrong, which is most of the engineering.

The four requirements

1. Idempotency — a retry must not create a duplicate
2. Retries with backoff, because most failures are transient
3. A durable queue, so nothing is lost during an outage
4. Reconciliation — a daily comparison of sent against arrived

Idempotency comes from the source

Derive a unique reference from the source record and use it when writing to the destination. The destination then rejects or updates rather than creating a second record.

That single practice prevents the most common and most damaging integration failure, and it is cheap to implement at the start and awkward to retrofit.

Map the data explicitly

• Document how each field in one system maps to the other
• Decide which system is authoritative for each field
• Handle the values that exist in one and not the other
• Version the mapping, because it will change

The mapping is business work rather than technical work, and it is where the time goes on most integration projects.

Reconcile daily

That last one is the alert that catches the failures that otherwise run for weeks.

Frequently asked questions