The moment it gets out of hand
Three systems connected directly is three integrations. Six is potentially fifteen. Nobody plans this; it arrives one sensible decision at a time, and then a field is renamed in one system and four things break in ways nobody predicted.
The transition point in our experience is the fourth system. Before that, direct connections are pragmatic and cheap. After it, they compound.
Define the canonical shape of each entity
What is a customer? Each system has its own answer, its own required fields and its own identifier. Without an agreed canonical definition, every integration re-litigates it and no two do it the same way.
- One canonical definition per entity: customer, order, product, invoice
- A stable internal identifier that never changes, whatever the source systems do
- A mapping table per system, translating to and from canonical
- One written owner per entity who decides what the fields mean
This is a governance exercise disguised as a technical one. The hardest part is getting two departments to agree what “active customer” means, and no integration platform solves that for you.
Events beat polling
Polling every fifteen minutes is simple and wasteful, and it means your data is up to fifteen minutes stale. Events — this happened, here is what changed — are more timely and cheaper to run.
Where a system cannot emit events, poll it and emit events yourself, so everything downstream works the same way. That one adapter keeps the rest of your architecture consistent.
Design for failure, because it is routine
- Retry with backoff for transient failures, which are most failures.
- Idempotency, so a retried message does not duplicate an order.
- A dead letter queue for messages that keep failing, with someone responsible for reading it.
- Visible sync status, so support can answer “did it go through?” without a developer.
- Replay, so a fixed bug can be applied to the messages that failed because of it.
When point-to-point is still right
Two systems, a stable relationship, low volume, no plans to add more. Building a middle layer for that is over-engineering, and it will be pointed out as such.
The judgement is about direction of travel. If you know a fourth and fifth system are coming this year, building the middle layer at three is cheaper than retrofitting it at six.
Middleware or your own
Integration platforms buy speed and connectors and cost per task at volume, with less control over error handling. Custom middleware costs more up front and gives complete control.
A common sensible pattern: middleware for low-volume, standard connections; custom code for the high-volume path that is core to your business. You are allowed to use both, and most mature setups do.