Pending, for the third day
A customer writes in: their payment left their account two days ago, the recipient has it, and your app still says pending. Support checks the admin panel, which also says pending. They check the provider portal, which says completed. They post in the ops channel, and an engineer eventually runs a script to update it.
It happens a few times a week. Nobody knows how many other payments are in the same state, because only the customers who notice write in.
How events go missing
Most providers tell you about status changes through webhooks, and they are reliable most of the time. The problems are at the edges, and a fintech sees enough volume to meet every edge.
| What happens | Why | Result in your app |
|---|---|---|
| Webhook arrives while your service is deploying | Endpoint briefly unavailable, provider gives up after retries | Status never updates |
| Handler throws an error part way | Unexpected field or a database timeout | Event acknowledged but not applied |
| Events arrive out of order | Completed arrives before processing | Status goes backwards to processing |
| Same event arrives twice | Provider retries after a slow response | Duplicate ledger entry or notification |
| Signature check fails after a key rotation | Old secret still configured | Every event rejected quietly |
The common thread is that the handler both receives and processes the event in one step. If processing fails, the event is often gone.
The cost of trusting a single webhook
Customers see wrong statuses and lose confidence in the app. Support spends time on tickets that are really data problems. Engineers run one-off scripts, which fix the symptom but leave no record. And your ledger may disagree with the provider, which shows up later in reconciliation as a break someone has to explain.
Webhook intake that does not lose things
What we build separates receiving an event from acting on it, and adds a safety net behind both.
- A small intake endpoint verifies the signature, stores the raw event and returns success immediately. It does nothing else, so it rarely fails.
- A worker processes stored events from a queue, such as AWS SQS or a database queue, and retries on failure without losing the event.
- Each event is checked against an event ID and your own record, so duplicates are ignored and a status never moves backwards.
- Events that still fail after retries go to a dead-letter list that ops or engineers can see and replay with one click.
- A scheduled sweep finds payments that have been pending longer than a threshold you set, asks the provider's API for the current status, and applies it through the same processing path.
- An alert fires if no events have arrived from a provider for longer than normal, which catches expired secrets and endpoint changes.
Because every raw event is stored, you can see exactly what the provider sent and when, which is useful when a partner or your own engineers ask.
What support sees afterwards
Stuck payments are rare, and when one does stick, the sweep usually corrects it before the customer notices. When it cannot, the payment appears in the dead-letter list with the reason, rather than in a customer email.
Support stops relaying status questions to engineering. Engineers stop running scripts in production, and the few manual replays that happen are logged.
Could this be your problem?
- Customers report payments stuck in pending that the provider shows as done.
- Engineers fix statuses with scripts.
- Your webhook handler does its processing inline.
- Duplicate notifications or ledger entries appear occasionally.
- Nobody would notice if a provider stopped sending events for an hour.