What is a computer use agent?

2026-10-29 · Avi Agola, The Seattle Compute Company

A computer use agent is an AI that operates a computer the way a person does. It looks at the screen, decides what to do, and then moves the mouse, clicks and types. It doesn't need a special connection to the software it's using. If you could do the task in a browser or a desktop app, so can it.

That matters because most business software doesn't offer a clean way for an agent to plug in. This article explains how computer use works, where it beats the alternatives, and why the hardest part is rarely the clicking. It's the login.

How a computer use agent works

The basic loop is simple.

1. The agent takes a screenshot, or reads the page structure.
2. A model decides the next action: click this, type that, scroll, wait.
3. The agent performs the action on a real browser or desktop.
4. It looks at what changed and repeats until the job is done or it gets stuck.

Everything else is engineering around that loop: keeping it fast, keeping it from wandering, noticing when a page hasn't finished loading, and recovering when a click lands on the wrong thing.

The agent also needs somewhere to run. Running it on your own laptop would mean it shares your files, your cookies and your sessions. So well-built computer use agents run on their own isolated machine. In Smith, each agent has its own computer with a real browser, separate from everyone else's.

Computer use versus API tools

There are two main ways to give an agent abilities.

API tools are pre-built connections: "send an email," "create a ticket," "read this spreadsheet." They're fast and precise when they exist. The agent gets a clear list of functions to call and clear results back. Smith agents use them too, through connected apps and MCP servers.

Computer use is the fallback for everything without a clean connection. County property websites, a candidate's profile page, a vendor's customer portal, an old internal tool. Many of these have no API at all, or an API that doesn't expose what you need, or one you have no access to.

The practical answer is to use both. Take the API when there is one. Use the screen when there isn't. An agent that can only do the first is limited to the tools with good integrations. An agent that can do both can finish the job.

Why logins are the hard part

Clicking through a website is the easy bit. The work that's valuable is almost always behind a login, and logins bring four problems.

Sessions. A site that you signed into yesterday may have logged you out. The agent has to notice, sign in again and carry on, without losing its place.

2FA. Many accounts want a code from an app or a phone. An agent that can't clear that stops at the door. Some codes the agent can handle on its own. Others live on a device only the owner has, and a person needs to supply them.

CAPTCHAs and bot checks. Sites that want to keep automation out put challenges in the way. These are a signal that the site doesn't want what the agent is doing. Sometimes there's a legitimate way around it (an official integration, a partner arrangement, the account owner stepping in). Sometimes the right answer is that the agent shouldn't be there.

Whose account? This is the one most tools skip. When an agent signs in, it is signing in as someone, and that changes what you can safely hand it.

Smith splits this into two modes:

• Delegated identity. When the account itself is the asset, like your own professional profile with its network and reputation, the agent works from your account. Your password stays in an encrypted vault. The vault fills in the sign-in on the right site, so the agent never sees the password.
• Agent-owned identity. When the account is interchangeable and you want to know who did what, like a GitHub login or a research-portal seat, the agent gets its own. You can revoke it without touching yours.

Delegated vs agent-owned identity for AI agents walks through how to choose.

Agents that learn a flow once

The first time an agent does a job on a site, it has to read the screen and work things out. The second time shouldn't need that.

Smith agents can save a flow they've worked out as a skill and reuse it, instead of rediscovering the same steps every run. That's what makes repeated work like checking the same portal each week practical rather than just possible. A saved flow is a shortcut, not a script: the agent is still looking at the screen as it goes.

Safety: what you should expect

Giving an agent a browser and a login should come with controls. These are the ones to look for.

Sandboxing. The agent runs on its own isolated machine, so a mistake or a bad page doesn't reach your files or your other sessions.

Scoped credentials. The agent gets only the access the job needs, and you can revoke it. Credentials sit in a vault, not in the agent's context.

An audit trail. Every action should be traced to the agent that took it and the account it used. Without that, you can't answer "who looked at this?" or "what changed?"

Human takeover. You should be able to stop the agent, and take over its screen when a step needs a person: a 2FA code on your phone, or a decision you don't want to delegate.

Keeping AI agents safe covers these in more detail.

What it looks like in real work

Three examples of workflows Smith agents run for teams we work with.

• Recruiting. One of our customers is a technical recruiting agency. Its agent signs in to the agency's search platform, researches candidates on LinkedIn and personal sites, runs their public GitHub repos on its own machine, and posts a sourced shortlist in Slack. The full walkthrough is on the AI recruiting agent page.
• Venture capital. A pre-seed VC firm has an agent that reads new university research as potential companies and discusses it with the team in Slack. See AI deal sourcing.
• Property research. A land-acquisition startup has an agent that works through 50+ county websites, creates its own accounts on them with its own email, and handles the verification emails. See county property research.

Each one involves a login, a site with no useful API, and a need to know afterwards what the agent did. That's where computer use earns its place.

Questions to ask before you pick one

Not every computer use tool is built for work that matters. Before you trust one with a login, ask:

• Where does it run? On your machine, or on its own isolated one?
• Where do passwords live? In the agent's context, or in a vault that fills in the sign-in?
• Whose account is it using, and can you tell afterwards? A log that can't separate the agent from you isn't an audit trail.
• What happens at a 2FA prompt or a CAPTCHA? Does it stop and ask you, or does it try to push through?
• Can you stop it mid-run? Including a job that is already running, not just the next message.
• Can you take over its screen? Some steps need a person.

An agent that has good answers to all six is one you can hand real work. One that doesn't is a demo.

How to get started

If you want to try one, you don't need to write code. AI agents for small teams explains how hiring an agent works: you write its role once, then give it jobs in Slack or by email.