Computer-Use Agents: AI That Clicks, Types and Reads the Screen

The software nobody will ever integrate

Most businesses have at least one system that sits outside everything else. The desktop accounts package from 2011. The insurer's claims portal. The supplier ordering screen that only works in one browser. Somebody spends two hours a day reading from one window and typing into another, and every quote to connect them properly has come back too expensive or simply impossible because the vendor offers no API.

That is the gap computer-use agents are aimed at. Instead of talking to software through code, the agent operates it the way a person does: it sees the screen, moves the pointer, types into fields and reads what comes back. It is one of the most talked-about agentic AI trends of 2026, and for once there is a real use underneath the noise.

How a computer-use agent actually works

Strip away the demo video and there is a simple loop. The agent takes a screenshot, a vision-capable model works out what is on it, the model picks an action, the action is executed, and a new screenshot is taken. Repeat until the task is finished or a limit is hit.

1. Observe. Capture the screen, sometimes with the accessibility tree or page structure alongside it, which makes element detection far more reliable.
2. Decide. The model compares what it sees with the goal and chooses the next action: click here, type this, scroll, wait.
3. Act. A controller moves the mouse or sends keystrokes inside a sandboxed virtual machine or remote desktop.
4. Check. A new screenshot confirms whether the action did what was intended, or whether a pop-up, error or slow page got in the way.

The important word there is sandboxed. You do not run one of these on a member of staff's laptop with their email open in the next tab. It gets its own virtual desktop, its own restricted credentials and nothing else.

Where computer-use agents earn their cost

They do best on tasks that are repetitive but not perfectly identical, in software that cannot be reached any other way. Some typical examples:

• Copying approved orders from a web shop into a desktop accounting package that has no import
• Checking claim or policy status on several insurer portals and writing the result back to a spreadsheet
• Downloading monthly statements from a dozen supplier portals, each with a different layout
• Filling government or regulator forms that change slightly every few months
• Testing your own internal software by clicking through it the way a user would

A 25-person freight forwarder that keys 60 shipments a day into a carrier's booking portal is a reasonable illustration. At four minutes each that is four hours of someone's day. An agent doing it in six or seven minutes each, unattended and with a person reviewing exceptions, is slower than the human but frees the human completely.

Computer-use agent vs API integration vs RPA

The honest ranking is simple. If a usable API exists, integrate directly and do not let anyone sell you an agent. If the process is identical every time and the screens never change, a scripted bot is cheaper. The computer-use agent wins in the messy middle: no API, moderate variation, and a tolerance for something taking a few minutes. We compare these options in more depth in our post on agent workflows versus RPA.

Where they break

The failure modes are predictable, which is good news, because predictable failures can be designed for.

• Pop-ups and interruptions. Cookie banners, session timeouts and "your password expires soon" dialogs derail more runs than anything clever.
• Look-alike elements. Two buttons called Submit, or a table where rows shift as data loads.
• Silent success. The agent believes it saved the record, but the page showed a validation error in small red text it did not register.
• Captchas and multi-factor prompts. These exist precisely to stop automated operators, and working around them is usually a breach of the site's terms.
• Volume. At thousands of tasks a day the per-task model cost and the minutes per task stop making sense.

The fix for silent success is verification by a separate route wherever possible: after the agent submits, read the record back, or check the confirmation email, rather than trusting the agent's own report.

Security and permissions before anything else

An agent that can use a screen can use any screen it can reach. That makes permissions the first design decision, not the last. We give each agent a dedicated account with the narrowest role the software allows, a virtual desktop with no other applications, no access to email, and a list of domains it may visit. Anything that spends money or sends something to a customer goes to a person for approval.

Prompt injection is a real concern here too. A web page the agent reads can contain text written to redirect it. Treat everything on screen as untrusted input, and never give the agent a capability that would be dangerous if the page lied. Our guide to agent permissions and guardrails covers the controls in more detail.

How we would start

At SpiderHunts we begin by recording a person doing the task twenty or thirty times, including the awkward cases. That recording becomes the test set. We then build the agent against a copy or test account of the target system, run it in shadow mode alongside the human for a couple of weeks, and only then let it work unattended with approval on anything consequential.

If, halfway through, we discover the vendor has an export file or a half-documented API after all, we switch to that without apology. A cheaper, deterministic route beats a clever one every time. If you want the agent built properly, that sits within our AI agent development work.

Frequently asked questions