OpenGCUA Guide: The Open-Source Approach to Letting AI Actually Use a Computer
By the end of this guide you will have three things: a capability list with evidence grades (what the paper and code actually support, and what is marketing), a full set of steps from clone to a working Quick Start on macOS, and a clear comparison of three adoption paths — including why you should run a read-only proof of concept first instead of handing your machine to the whole harness.
Time cost: about 15 minutes to read, 30–60 minutes to install and run Quick Start, 0.5–1 day for a full PoC. Difficulty is intermediate: you need macOS 14+, basic command-line skills, Python 3.12, and a working understanding of macOS privacy permissions (TCC, Transparency, Consent, and Control) — the most underestimated and most security-critical part of this stack.
What OpenGCUA Is: From Screenshot Recognition to Actually Operating a Computer
OpenGCUA is a generalist computer-use agent, open-sourced under Apache-2.0 by the Tongyi-MAI team. The fundamental difference from the current approach: mainstream computer-use pipelines render screenshots in a headless browser and let the model point at pixels; OpenGCUA works through four channels at once and turns "operating a computer" into "writing code."
| Channel | What it reads | When it is used |
|---|---|---|
| accessibility tree (AX) | element role / name / value | acting on elements directly, no coordinates, no foreground focus |
| App API | the application's own HTTP API | apps with a documented API |
| shell | stdout / stderr / return code | batch, file, and system work |
| GUI (screenshot + coordinates) | pixels, coordinates normalized to 0-1000 | interfaces with neither AX nor API |
The four channels:
- accessibility tree (AX): reads element role, name, and value, and acts on elements directly — no coordinates, no bringing windows to the front.
- App API: calls the application's own HTTP API from the same script. The paper's CRM example PATCHes a lead status and asserts on it without touching the GUI.
- shell: bash commands returning stdout, stderr, and return code.
- GUI (screenshot + coordinates): screenshots on demand, coordinates normalized to 0–1000, real foreground input only when necessary.
The real weapon is code execution: the model writes one Python script mixing AX reads, shell commands, API calls, and GUI actions — with loops and conditionals — and runs it in one shot, eliminating the screenshot-wait-respond cycle. Ablation numbers: +6.1 partial score and −14.8% steps (T0, paper §3.2).
The second design highlight is background operation: background_mode is on by default; Chromium launches without activation and tabs stay in the background, native apps are driven via AX actions — the agent works alongside the user without stealing mouse or keyboard (paper §4.2).
Architecture and Capability Boundaries: Evidence vs Marketing
The short version: code execution, AX reads, background operation, and the benchmark numbers are backed by paper and code. "Hands-off autonomous completion of real work" is not. Item by item:
| Capability | Evidence | Verdict |
|---|---|---|
| AX reads and element-level actions | SDK examples, QUICKSTART, paper §4.2 | Code + docs + paper |
| Background window operation | background_mode design and docs | Supported; demo videos not independently verified |
| Shell / App API | System prompt verbatim; paper §2 (OSWorld) | Verbatim prompt and trajectory |
| Code execution | Ablation: +6.1 points, −14.8% steps | Numbered |
| GUI screenshot-coordinate actions | Code and paper §4.3.4 cases | Present; misclick risk depends on the model |
| Foreground input approval | QUICKSTART: one Allow/Reject per task | Covers SDK actions only, not arbitrary Python/Bash |
| Confirmation gating, multi-display, AX-tree diff | harness README flags 🚧 (v0.2 / v1.0 / v0.2) | None shipped |
| Cross-device (Android/Linux) | README shows macOS examples only | Partially unverified |
| Native menu-bar app (DMG) | Local dev build, not notarized | Not an official release |
Two takeaways: its safety model assumes a user present, approving each foreground action — the opposite of unattended operation. And the project was created on 2026-10-15; barely two weeks old at writing, a research preview by its own framing.
The Numbers: Big Efficiency Gains, One-in-Four Full Success
The short version: averaged over 8 frontier models, observation and execution efficiency improve dramatically, but full-task success sits around one quarter — good enough as an efficiency tool, not as an unattended employee.
Paper Table 2 (82-task offline subset of OSWorld-v2, 8 models, T0) averages:
- partial score: +19.6 points
- steps: −53.0%
- output tokens: −27.2%
Take the biggest gainer, DeepSeek-V4.1-Flash: partial score 28.0 → 57.7 (+29.8), steps 167.3 → 84.9 (−49.3%), output tokens 316.0K → 180.4K — yet full-task success only moved from 8.5% to 26.8%. Three out of four tasks still need a human.
Real-device evidence is thinner: 4 macOS tasks, one selected run per model, self-labeled preliminary by the authors. Best is GPT-6 Astra at 4/4; open flash-tier models manage 2/4. Cost differs by an order of magnitude — roughly US$18 per task for GPT-6 Astra versus US$0.1–1.5 for flash-tier models (T0, Table 5).
Repriced for a budget model at similar token volumes, API cost lands around US$0.25–0.35 per task (internal estimate, not a measured bill) — 10–50× a pure shell workflow. It only pays off for GUI work that would take a human 20–30 minutes (form filling, cross-app data moves); if shell can do it, don't hand it over.
Paper §4.3.4 catalogs three failure modes: over-reasoning trivial actions (2,105 vs 124 output tokens); observing the wrong target (foreground window instead of the background page); and incomplete scripts (85 of 125 tasks done, unchecked).
System Requirements: macOS 14+ and TCC Permissions
The short version: hardware and software bars are low. The real gate is that TCC grants attach to the responsible process — switch interpreters and the grant evaporates, which directly affects background and unattended scenarios.
| Requirement | Spec |
|---|---|
| OS | macOS 14+ (Apple Silicon for the native app) |
| Python | 3.11–3.14 (3.12 recommended) |
| uv, Xcode CLT | Required |
| Rust/Cargo | Only for source installs (1.89+); the SDK ships prebuilt wheels |
| CuaDriver.app | Installed by mu setup macos, resident daemon |
Two TCC grants go to CuaDriver.app: Screen Recording and Accessibility.
A pitfall the docs call out explicitly: TCC authorization follows the responsible process, not the global state. Switch venvs, swap the Python binary, or change terminals and the grant breaks — every GUI call returns PERMISSION_DENIED. macOS usually refuses to add a bare python3.12 to the Screen Recording list. Only three reliable fixes exist: run under an already-authorized interpreter, restart CuaDriver.app, or authorize the terminal/IDE that launched it. If you drive it from cron or background jobs, the responsible process is likely not the one you authorized — test first.
On background execution: supported by design, background_mode on by default; foreground actions need one Allow/Reject per task. GUI operations require a logged-in, unlocked desktop session; SSH/headless has no documented support.
Install and Quick Start
The short version: five official steps — clone, create a venv, install the SDK, run setup, run doctor. The menu-bar app still requires a local build and is not notarized.
git clone https://github.com/Tongyi-MAI/OpenGCUA
cd OpenGCUA
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ./packages/sdk
Then install and check the macOS driver layer (CuaDriver):
mu setup macos # installs CuaDriver.app and walks through permissions
mu doctor macos # environment and permission self-check
Then grant Screen Recording and Accessibility to CuaDriver.app under System Settings → Privacy & Security. Pass criterion: every capability shows ready in mu doctor macos.
The menu-bar assistant is a SwiftUI native app bundling Python and the SDK. It exists only as a local dev build and is not notarized — first launch requires right-click → Open in Finder to bypass Gatekeeper. Before granting it TCC permissions, read the next section.
Execution parameters worth knowing upfront (names and defaults from the official QUICKSTART, T0):
background_mode: true # background operation on by default
compaction_trigger_tokens: 200000 # compaction trigger threshold
compaction_max_tokens: 100000 # post-compaction cap
img_keep_n: 5 # keep the last 5 screenshots
img_interval: 10 # screenshot every 10 steps
Compaction is lossy and does not resume interrupted actions. Models connect via the openai protocol (compatible with Qwen, vLLM, Ollama, LiteLLM), plus openai-thinking and anthropic protocols. Vision input is not required — AX + shell + API already drive most tasks.
Three Adoption Paths: Harness, Plugin, SDK
The short version: path A is two days and high-risk; path B means rewriting a Windows engine, so leave it alone; path C is half a day to one day at the lowest risk, and should go first.
| Path | What it involves | Effort | Risk |
|---|---|---|---|
| A | Run the macOS harness directly (SDK + CuaDriver) | about 2 days | High: model-generated code runs with your user privileges; confirmation gating is a v0.2 item |
| B | Wire up the dsh-gcua community plugin | 4-6 days | Medium: its engine is Windows-only and needs a rewrite; single maintainer |
| C | Use only the SDK to read AX, no LLM loop | 0.5-1 day | Low: read-first, reversible |
Path A: run the macOS harness directly. Install the SDK and CuaDriver, grant TCC, and drive real tasks through the example scripts or the menu-bar app. Roughly 0.5 day to install plus 1–1.5 days of permission and safety acceptance — about 2 days total. High risk: model-written code executes with your user privileges, confirmation gating is unfinished, and GUI actions fight resident services for focus. Only attempt inside a dedicated macOS user.
Path B: the dsh-gcua community plugin. Looks like wiring; it is actually a rewrite. Its engine (~19.6 KB of Python) self-identifies as a Windows GCUA harness — the PowerShell lane must become bash, pywinauto/UIA must become AX (pyobjc), taskkill /T /F must become process groups, Ctrl+V must become Cmd+V. Estimate 4–6 days. Add a single maintainer, 4 stars, and a 25 KB repo: not a production dependency.
Path C: SDK only. Install the prebuilt wheel (no Rust toolchain), call the macOS adapter to read the AX tree and perform background AX actions — no LLM loop. 0.5–1 day, read before write, lowest risk. The value is filling the gap shell and headless cannot reach: chat apps, broker terminals, spreadsheets, PDF signing.
Security Risks: Model-Written Code Runs with Your Privileges
The short version: the biggest risk is not a misclicked button — it is model-generated Python/Bash executing directly with your login user's privileges, while the official confirmation gate only arrives in v0.2.
The harness README says it plainly: confirmation gating is a v0.2 item, and until then "run only on machines whose state you can afford to lose." Treat that as an install prerequisite, not a disclaimer.
Four layers of risk:
- Arbitrary code execution: model-written scripts can read your SSH keys, browser profiles, and API keys; Allow/Reject covers SDK actions only — the docs explicitly say it is not a host-level gate for arbitrary Python/Bash.
- Prompt injection → GUI actions: web or document content can steer the model toward "approve payment" or "delete" buttons; issued actions are not rolled back (already-issued actions are not rolled back).
- Oversized TCC grants: whatever process can launch CuaDriver.app inherits its Screen Recording and Accessibility powers.
- Focus stealing:
background_modelowers the odds, but system dialogs and foreground-requiring actions can still steal focus; alongside cron/heartbeat jobs, timing collisions cause clicks in the wrong window.
Minimum isolation plan for testing:
- Create a dedicated macOS standard user: no iCloud, no mount of the primary home, no credentials.
- Use a separate API key (in a gitignored local config), never shared, rotated after testing.
- Read-only AX first, no writes; once writing, approve every action manually.
- Keep trajectory logging and screen recording on throughout; audit afterwards.
- Do not register the harness as a tool in automation systems, so other agents cannot trigger it autonomously.
Our Recommendation: Read-Only PoC First, Hard Acceptance Lines
The short version: run the path-C read-only PoC — 0.5–1 day, five acceptance lines, all of which must pass before anything else. Write the criteria down first; don't improvise.
The PoC scope is deliberately narrow: read-only, no writes, no LLM loop. Steps:
- Isolation (0.5h): create the dedicated standard user; verify the primary account's
.ssh, browser profile, and agent config directories are unreadable from it. - Environment (0.5d): venv + SDK install,
mu setup macos,mu doctor macos, grant TCC. - Read example: pull the AX tree and window list from a native app like TextEdit (exact APIs per the repo's examples directory).
Five acceptance lines (all must pass before the next stage):
- C1: every capability shows ready in
mu doctor macos. - C2: SDK imports cleanly, no native extension errors.
- C3: AX tree has ≥ 50 nodes with role/name/value; 3 consecutive reads under 5 seconds each; element ids map across two snapshots.
- C4: a single serialized observe is < 30 KB (roughly < 2K tokens), with zero screenshots.
- C5: zero writes; the target app's content diff is empty afterward.
Decision gate: only if C1–C5 all pass is it worth another 2 days on an isolated path-A trial (plus three write acceptance lines: the target element changes as expected, foreground app and mouse position unchanged before/after, PERMISSION_DENIED without crashing after revoking permissions). If C3 or C4 fails (AX tree too large or slow), stop — keep only low-frequency "read AX for queries" usage.
Explicit non-goals: no unattended runs on the primary machine, no production integration, no auto-approving foreground actions, and no payment/send/delete-class tasks.
One closing judgment: what is worth absorbing from OpenGCUA is the design — code execution plus selective observation plus a task ledger — not downloading it to take over your computer. Porting that design into an existing shell workflow costs far less and points the same way: fewer screenshots, more code, background completion.
Sources
Primary sources (T0, fetched 2026-10-27) are listed in the references above: the GitHub repo and QUICKSTART, the arXiv paper, the project site, and the dsh-gcua community plugin. The "internal estimate" cost figures are not verified bills; real-device sample-size limits and the authors' own caveats are in paper §4.3.3.