microsoft/amplifier-bundle-computer-use

microsoft/amplifier-bundle-computer-use

Description: Computer Use bundle for the Amplifier project

Language: Python

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-03T14:25:00Z

Pushed: 2026-08-04T04:34:29Z

Default branch: main

Fork: no

Archived: no

README:

amplifier-bundle-computer-use

Give an Amplifier session real control of a desktop — Windows, macOS, or Linux, on this machine or another one across your private network — using the LLM provider's native computer-use tool, not a homegrown imitation of it.

If a human could do it by looking at the screen and clicking, this can do it. No API, no CLI, no browser extension required.

"What's on my screen right now?"
"Click the Export button in that accounting app and save it to my desktop."
"Fill this dialog in for me — I'm done pressing buttons."

> ## ⚠️ Read docs/SETUP.md before you install > > This is not the usual one-line, `--app`-level pointer at a behavior bundle. It > drives a real desktop, so the install has four moving parts that live outside this > repository: a version floor on three upstream modules, a model that supports > native computer use, a target machine with per-platform prerequisites (WSL2 > interop / macOS TCC grants / X11), and — for remote targets — SSH key auth and > `uv` on the far end. > > [→ docs/SETUP.md](docs/SETUP.md) has the exact checks, the proven-vs-rough > matrix, the full config reference, and the operational facts (a locked screen > cannot be driven, on any platform) that otherwise cost you an afternoon.

---

Install

Registering this bundle is ordinary Amplifier bundle management:

# 1. Register it (the name "computer-use" comes from this repo's own bundle.md)
amplifier bundle add git+https://github.com/microsoft/amplifier-bundle-computer-use@main#subdirectory=behaviors/computer-use.yaml --app

# 2. Use it for a session
amplifier run --bundle computer-use "What's on my screen right now?"

Confirm it registered correctly with amplifier bundle show computer-use — it should list tool-computer-use, hook-computer-use, and computer-use:computer-operator. Working from a local clone instead of GitHub? amplifier bundle add file:///path/to/amplifier-bundle-computer-use instead.

Registering the bundle is not the same as the tools working. See [docs/SETUP.md](docs/SETUP.md) for the four things that decide whether computer and desktop actually function once a session starts: upstream module versions, a model with native computer-use support, a reachable target machine, and (for remote targets) SSH.

---

What makes this the native thing

Both Anthropic and OpenAI post-train their models on a specific server-side tool definition. Sending a lookalike function tool instead gets you noticeably worse targeting — or, on OpenAI, a hard 400. This bundle sends the real one, per vendor:

// Anthropic — versioned type, dimensions REQUIRED
{"type": "computer_20251124", "name": "computer",
"display_width_px": 1280, "display_height_px": 720, "enable_zoom": true}

// OpenAI — bare type, every extra field is a 400
{"type": "computer"}

Anthropic additionally needs the computer-use-2025-11-24 beta header (derived by the provider itself). Screenshots come back as genuine base64 image content blocks inside the tool result, exactly as each vendor's own loop does.

Both providers have driven a real desktop through this bundle. Anthropic: claude-sonnet-4-5, claude-sonnet-5, claude-opus-5. OpenAI: gpt-5.5, verified end-to-end against a real remote desktop. A Gemini dialect record exists (gemini-2.5-computer-use), transcribed from captured traffic — no live end-to-end run through this bundle is claimed for it.

Model support is capability-gated upstream: provider-openai and provider-anthropic both expose supports_native_computer_use on ModelCapabilities, and a model without it cannot use this bundle. See docs/SETUP.md §2 for the exact rules (OpenAI: minor >= 4 and not -nano; Anthropic: per-family version thresholds mapping to a specific computer_YYYYMMDD wire type).

The one thing that had to be solved

Amplifier's orchestrator used to stand between a mounted tool and native computer use: tool results are collapsed to str before reaching the provider, so a screenshot could never travel back as an image.

That is fixed at one seam — the provider's complete() call — by hook-computer-use, which:

• unwraps the orchestrator's ToolResult envelope and expands screenshot markers into

real image blocks,

• keeps only the N most recent screenshots inline so long sessions stay affordable.

(ToolSpec used to be built from name/description/parameters only, so a tool couldn't declare itself a server-side tool type either — hook-computer-use used to promote it and inject the beta header itself. That is now handled upstream: loop-streaming preserves a tool's native_tool_spec through its own ToolSpec construction, and provider-anthropic derives the required anthropic-beta header itself. hook-computer-use now only verifies that support is present and refuses to mount if it isn't — see _fail_if_orchestrator_native_tool_spec_unsupported, and docs/SETUP.md §1 for the exact commits you need.)

Remove the hook and the tool degrades cleanly to an ordinary function tool. Nothing is monkey-patched on disk; nothing rots when the orchestrator changes.

Components

| Piece | Role | |---|---| | modules/tool-computer-use | computer (native action set) + desktop (windows, clipboard) | | modules/hook-computer-use | The wire-format seam described above | | agents/computer-operator.md | Operating discipline for driving a live machine | | bridge.ps1 | WSL2 → Windows execution via powershell.exe + Win32 |

Tools

`computer` — the native action set: screenshot, zoom, cursor_position, mouse_move, left_click, right_click, middle_click, double_click, triple_click, left_mouse_down, left_mouse_up, left_click_drag, scroll, key, hold_key, type, wait, plus screen_info, list_windows, focus_window.

`desktop` — what the native schema cannot express: list_windows, focus_window, screen_info, get_clipboard, set_clipboard, list_monitors, select_monitor. Focus a window before typing into it; use the clipboard to pull exact text out of an app; use select_monitor to switch which...