Point-to-point does not scale
Two systems connected directly is manageable. Five systems connected to each other is twenty connections, each with its own error handling, and changing any one of them affects four others.
The moment you have a third system, a middle layer stops being over-engineering and starts being the thing that keeps the whole arrangement comprehensible.
What the middle layer does
- Translates between each system's data format and terminology
- Queues work when a destination is unavailable
- Retries transient failures with backoff
- Deduplicates, so a retry cannot create two records
- Records everything that moved, for reconciliation
Mapping is the hard part
Systems disagree about what a customer is, what a product is and what an order status means. Reconciling those is business work rather than technical work, and it is where the time goes.
- Agree the canonical definition of each entity
- Document how each system's version maps to it
- Decide which system is authoritative for each field
- Handle the cases where they disagree, explicitly
Reconciliation is not optional
A daily comparison of what was sent against what arrived catches silent failures the same day. Without it, a broken connection can run for weeks with nobody noticing.
It is a small piece of work and it is the difference between a reliable integration and one people stop trusting.
Make it observable
| Need | Provide |
|---|---|
| What moved today? | A summary report |
| Why did this record fail? | Per-record logs with the payload |
| Is anything stuck? | Queue depth monitoring |
| Can we reprocess it? | A retry mechanism for quarantined items |
| Who changed the mapping? | Configuration in version control |