The short answer
A computer-use agent is AI software that operates a computer through its screen. It takes a screenshot, asks a vision model for the next action, moves the mouse or types through the operating system, then looks again to confirm it worked. No API or connector needed.
The loop: look, decide, act, check
Every computer-use agent runs the same four-beat cycle.
-
Look
Capture the screen as an image. Some agents also read the page structure when there is one.
-
Decide
The model gets the goal, the screenshot and the history, and returns the next action: click here, type this, scroll, wait.
-
Act
Software on the machine performs it. The model never touches the computer directly.
-
Check
A fresh screenshot shows whether the step worked, and the next one is chosen.
The loop ends when the task is done, a step limit is hit, or an action fails.
Grounding: turning "click Post" into a pixel
Grounding is where an instruction becomes a spot on the screen. Most errors start here. Agents have three ways to find a target.
| Source | What it reads | Where it breaks |
|---|---|---|
| Accessibility tree | Control names the app publishes for screen readers | Custom-drawn screens, remote sessions |
| Page structure (DOM) | HTML inside a browser | Anything outside the browser |
| Pixels | The screenshot itself | Tiny icons, look-alike buttons, faint text |
Every app draws pixels, which is why vision makes this category possible. The trade-off is precision on crowded professional screens.
A sneaky failure: if the screenshot is shrunk before the model sees it, clicks must be scaled back up. Skip that and every click misses, more so toward the bottom-right corner.
Two years, fast progress
-
2024
First model APIs for screen control and the first real-desktop benchmarks. Best scores were a fraction of human performance.
-
2025
Consumer agents launch. Scores climb fast. New tests on professional software show grounding is the hard part.
-
2026
Screen control leaves beta. Top benchmark scores now pass the human baseline on unfamiliar tasks.
Reading the benchmarks
Whole-task tests run an agent in a virtual machine and pass it only if the machine ends up in the right state. Grounding tests ask a narrower question: did the click land on the right element?
Read the numbers with care:
One pass rate hides inconsistency. Repeat runs matter. Testing Before Production
Leaderboard rows are often self-reported under different rules.
No benchmark includes your ERP or your vendor portal.
Benchmarks score success, not damage. A wrong posting in a live ledger costs real cleanup.
Where they fit
Use one when the work lives on screens with no usable API: legacy desktop ERPs, terminal emulators, other companies' portals, PDFs. Where a supported API already covers a step, that step can use it, and the agent carries on to the screens the API doesn't reach.
Build in security from day one: minimal privileges, no credentials in front of the model, and treat on-screen text as data, since a page can try to give the agent instructions.
- Green-screen terminal
- Desktop ERP
- Vendor portal
- Scanned PDF
- Spreadsheet
Computer-use vs RPA vs browser agents
| Classic RPA | Browser agent | Computer-use agent | |
|---|---|---|---|
| How it acts | Replays a recorded script | Drives a web page | Chooses each action from the screen |
| Reach | Apps with stable selectors | Browser only | Whole desktop |
| Screen changes | Breaks | Sometimes copes | Usually copes |
| Speed per step | Fastest | Fast | About human pace |
How LaunchAI applies it
LaunchAI's Vision Layer runs this loop on your own machine: look, find, act, verify. It tries the cheapest evidence first and uses the vision model last. A target it can't resolve is never guessed; the step re-plans, goes to a person, or stops. Product →
Questions
If agents beat the human baseline, why supervise them?
The baseline measures people trying an unfamiliar task once. A business needs the same right answer on the thousandth try, and one bad ledger write can outweigh hundreds of good ones.
Do screenshots go to the model provider?
A cloud model has to receive the images it reads. What it keeps depends on your settings and contract. Local vs Cloud Agents →
Keep learning
- What Is an ADK An agent development kit (ADK) is a packaged set of building blocks for creating, testing and running AI agents: a model interface, tools that act on systems, orchestration, guardrails, human checkpoints, evaluation, tracing and a runtime. Read the guide
- Automate Without an API First figure out which "no API" you have. Read the guide
- Which Workflow First Start with a workflow that is frequent, slow by hand, low in variation, spread across a few systems, cheap to get wrong and rarely off the happy path. Read the guide
No prompts. No babysitting. No Integrations.