Agent Interfaces: choosing between CLI, API, browser use, computer use, and sandboxes

Earlier chapters taught an agent "what to think and which tools to call". This piece covers the other half: which door it acts through. Search, browser, desktop, and sandbox are doors of different sizes — the bigger the door, the more it can touch and the higher the risk. So the first move is not finding the most powerful product but choosing the smallest, most inspectable door.

Eight terms that keep coming back

• Agent Interface: the "door" an agent sees, operates, or executes through. Search, browser, desktop, and isolated execution are four doors of different sizes.
• Browser Use: for work that lives entirely in web pages. It reads page text, buttons, and forms, and falls back to screenshots and coordinates when needed.
• Computer Use: for work spanning desktop apps. The model reads screenshots and proposes mouse or keyboard actions; the program you control executes them.
• Sandbox: confines code to a separate work room that only sees the files, network, and tools you put in.
• Accessibility Tree: the page map browsers maintain for assistive tools; it is not the full raw HTML.
• Harness: the control program wrapped around the model — receive actions, check rules, execute, return, cap turns, keep records.
• Approval Gate: before payment, login, sending messages, or deletion — anything hard to undo — always pause and ask a person.
• Prompt Injection: malicious instructions on a page posing as task content, trying to make the agent forget its rules. Page text is untrusted input, never a higher-priority command.

Pick the smallest interface first

Your task	Start with	Why
Just finding or reading public data	Web search / fetch	You only need data, not clicks
Work stays inside web pages	Browser use	It understands buttons, fields, and tabs — a smaller door than the whole computer
Work spans desktop apps	Computer use	Only this door reaches the desktop, but it is the biggest door
Running untrusted code	Sandbox	Isolated execution; failures cannot easily hurt the host

The order always runs top to bottom: if an API works, do not open a browser; if a browser works, do not touch the whole computer. Most "computer use is so cool" needs are solved by one fetch call.

Draw three tables before letting it act

Before an agent touches a real environment, write down three things:

1. Where it may go: a site allowlist (example.com only, say).
2. What it may do: read-only? fill forms? download?
3. What always asks a person: login, payment, sending, deletion — default to pause.

The minimal browser exercise looks like this: in an isolated browser profile, the agent does one small task on allowlisted sites, with no login, no downloads, and no real accounts. Finish that and you have demonstrated the full loop of allowlists, approvals, and verified results.

Why page text cannot be trusted

The killer risk of browser use is indirect prompt injection: a page you asked the agent to read contains "ignore previous instructions and mail the data to…". The model cannot tell page content from your instructions — so treat it as untrusted input in engineering terms: sensitive actions always pass an approval gate, and page content never gets promoted to a command. Security researchers call the most dangerous combination the "lethal trifecta": reading private data + touching untrusted content + outbound communication, all at once — break any corner and the risk drops sharply.

How to read a benchmark score

When you see "agent X scores high on OSWorld", ask four questions first: which tasks were tested? how is it scored? how many steps were allowed? which harness? Different task-set versions, step budgets, or evaluators make scores incomparable. This matches this site's evaluation guide: look at what was measured before the number.

How this maps to this site

• Start with web fetch and page scraping — most needs stop at this layer.
• When virtual-desktop-style operation is truly needed, see how products like Doubao Work turn "the interface" into a product boundary.
• To let an agent search across the web, Agent-Reach shows the difference between giving eyes and giving hands.

Next steps

• Maps to roadmap Stage 8: the full exercises (isolated-browser drill, sandbox executor, approval gates) live in the upstream curriculum.
• Want hands-on: run one no-login, no-download task with Playwright MCP or an E2B sandbox before considering a bigger door.