Screen by screen, every day
The data is yours: your orders on a supplier's portal, your policies on an insurer's system, your shipments on a carrier's site, your claims on a scheme portal. But the portal only shows it one screen at a time. There is no export button, or the export is a PDF, or it misses the field you need.
So somebody logs in each morning, clicks through the list, and copies reference numbers, statuses and dates into a spreadsheet or your own system. When they are busy, it waits.
Why the portal has no export
Many portals were built to let customers look things up, not to hand data over. Adding an API costs the portal owner money and support, and they may simply not have been asked by enough customers.
That is worth remembering, because the best fix is often a request. Portal owners sometimes have an export, a report emailed on schedule, or an API for larger customers that is not advertised. We always ask before building.
- Portals built for looking up, not extracting
- APIs offered only to larger customers or on request
- Exports that exist but miss key fields
- Nobody at your end has asked formally
What the copying costs
Staff time every day on a pure transfer job. Typos in reference numbers that break matching later. Delays, because data only moves when someone has time. And your own systems are always slightly behind the portal, so customers get out-of-date answers.
There is a security angle people often overlook. When copying is a shared chore, the portal login tends to be shared too, written on a note or saved in several browsers. Nobody can say who looked at what, and when someone leaves, the password rarely changes.
The data also stays shallow. Because copying is tedious, only the essential fields make it across. The details that would help with analysis, such as the dates each status changed, never leave the portal.
How we get your data out properly
- We ask the portal owner for an API, bulk export or scheduled report on your behalf, with a clear explanation of what you need. If they provide one, we build on that.
- If not, we check the portal's terms and your agreement with the owner. Automated access to your own account is often acceptable, sometimes needs written permission, and occasionally is forbidden. We only proceed where it is allowed.
- We build a browser automation with Playwright that logs in using a dedicated account, with credentials held in a secrets store, and handles two-factor authentication in an agreed way.
- It reads the screens a person would, at a human pace, and extracts the fields into structured records.
- Records are validated: required fields, formats, and consistency with what you already hold, such as matching order numbers.
- Clean records go into your system through its API, or into a database or Google Sheet. Anything that fails validation goes to a review list.
- Every run is logged, and if the portal changes or a login fails, a named person is alerted.
| Route | When we use it |
|---|---|
| Portal API or bulk export | Whenever the owner offers one |
| Scheduled email report parsed automatically | When the portal can email reports |
| Logged-in browser automation | Only with permission or where terms allow |
Data that arrives without anyone copying it
Your system updates on schedule with what the portal shows, and staff only look at exceptions. Customer queries are answered from current data. If the portal owner later launches an API, we switch over and nothing downstream changes.
Access is also tidier. The automation uses its own dedicated account, every run is logged, and staff no longer need to share a login just to copy data out.
Is this happening in your office?
- Someone copies data from a third-party portal every day or week
- The portal has no export, or the export is missing fields
- Typos from copying have caused mismatches
- Nobody has formally asked the portal owner for an API