Anthropic CUA Template update (Playwright -> Computer Controls)

[dprevoznik]

Anthropic Computer Use Template Overhaul

This PR overhauls both the TypeScript and Python Anthropic Computer Use templates to use Kernel's Computer Controls API instead of Playwright for all browser interactions.

Why This Change

The previous implementation used Playwright directly, which required maintaining browser connections and handling lower-level browser automation. By migrating to Kernel's Computer Controls API, users get:

• Native integration with Kernel's browser infrastructure
• Built-in replay recording for debugging and auditing
• Consistent API across all Kernel computer use templates
• Simplified session management with automatic cleanup

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     Entry Point (index.ts / main.py)        │
│  - Defines the Kernel app and action                        │
│  - Creates browser session with KernelBrowserSession        │
│  - Invokes the sampling loop                                │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Session Manager (session.ts / .py)      │
│  - Manages browser lifecycle (create/delete)                │
│  - Handles replay recording (start/stop/poll for URL)       │
│  - Configures viewport (1024x768 @ 60Hz)                    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Sampling Loop (loop.ts / .py)           │
│  - Implements the Anthropic prompt loop                     │
│  - Manages conversation history                             │
│  - Routes tool calls to ToolCollection                      │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Tool Collection (tools/)                │
│  - ComputerTool: Mouse, keyboard, screenshots via Kernel    │
│  - Maps Anthropic actions to Kernel Computer Controls API   │
│  - Tracks last known mouse position for drag operations     │
└─────────────────────────────────────────────────────────────┘

File Structure (TypeScript)

anthropic-computer-use/
├── index.ts              # Entry point - defines Kernel app and cua-task action
├── session.ts            # KernelBrowserSession - manages browser lifecycle + replays
├── loop.ts               # Anthropic sampling loop with tool routing
├── tools/
│   ├── collection.ts     # ToolCollection - routes tool calls, manages versions
│   ├── computer.ts       # ComputerTool - implements all mouse/keyboard actions
│   ├── types/
│   │   └── computer.ts   # TypeScript types for actions and results
│   └── utils/
│       ├── keyboard.ts   # Key mapping utilities
│       └── validator.ts  # Coordinate validation
├── types/
│   └── beta.ts           # Anthropic beta API types
├── utils/
│   ├── message-processing.ts  # Prompt caching, image filtering
│   └── tool-results.ts        # Format tool results for API
└── README.md

The Python template follows the same structure with equivalent modules.

Key Components

1. KernelBrowserSession (session.ts / session.py)

Manages the browser lifecycle as a context manager:

const session = new KernelBrowserSession(kernel, {
  stealth: true,
  recordReplay: true,  // Optional: capture video replay
});

await session.start();
// ... use session.sessionId for computer controls
const info = await session.stop();
// info.replayViewUrl contains the video URL if recording was enabled

Features:

• Automatic browser creation with configurable viewport (1024x768 @ 60Hz)
• Optional replay recording with grace period before stopping
• Polls for replay URL after stopping
• Automatic cleanup on exit

2. ComputerTool (tools/computer.ts / tools/computer.py)

Maps Anthropic's computer use actions to Kernel's Computer Controls API:

Anthropic Action	Kernel API
left_click, right_click, double_click	computer.clickMouse()
mouse_move	computer.moveMouse()
left_click_drag	computer.dragMouse()
type	computer.typeText()
key	computer.pressKey()
scroll	computer.scroll()
screenshot	computer.captureScreenshot()

Key implementation details:

• Maintains lastMousePosition to support drag operations from current position
• Maps Anthropic key names to Kernel/xdotool format
• Returns base64-encoded screenshots after each action
• Supports both computer_use_20241022 and computer_use_20250124 API versions

3. Sampling Loop (loop.ts / loop.py)

Implements the Anthropic computer use prompt loop:

• Sends messages to Claude with computer use tools
• Processes tool calls and executes them via ToolCollection
• Supports thinking mode with configurable budget
• Handles prompt caching for efficiency

New Features

Replay Recording

Users can enable video replay recording by passing record_replay: true in the payload:

kernel invoke ts-anthropic-cua cua-task --payload '{"query": "...", "record_replay": true}'

The response includes a replay_url field with a link to view the recorded session.

Known Limitations

Cursor Position: The cursor_position action is not supported with Kernel's Computer Controls API. If the model attempts to use this action, an error is returned. This is a known limitation that does not significantly impact most workflows, as the model tracks cursor position through screenshots.

Testing

Both templates have been tested with the magnitasks.com Kanban board task, which exercises:

• Navigation and clicking
• Drag-and-drop (left_click_drag)
• Multiple sequential actions

Updated Documentation

• Template READMEs updated with setup, usage, replay recording, and limitations
• QA command updated with new test task
• CLI post-install instructions updated with new example

---

Note

Overhauls the Anthropic Computer Use templates to use Kernel’s Computer Controls API instead of Playwright, with built-in browser session management and optional replay recording.

• Replace Playwright automation with Kernel controls in both TS (tools/computer.ts, loop.ts, session.ts, index.ts) and Python (tools/computer.py, loop.py, session.py, main.py) templates
• Add KernelBrowserSession to manage browser lifecycle, live view, and replays (configurable viewport 1024x768@60Hz; stop/poll for replay_url)
• Update sampling loops to construct ToolCollection with Kernel client and sessionId; handle thinking blocks and tool_use routing; enable prompt caching
• Implement comprehensive key/mouse/scroll mappings to Kernel APIs; drop unsupported cursor_position; track last mouse position for drags; standardize typing delay and screenshot flow
• Bump SDKs (@onkernel/sdk / kernel to 0.24.0); remove Playwright deps and code paths
• Refresh READMEs with setup/usage and replay instructions; adjust QA/template invoke commands to magnitasks task and optional record_replay flag
• Update template defaults in pkg/create/templates.go to new invoke payloads

^{Written by Cursor Bugbot for commit ac3baaa. This will update automatically on new commits. Configure here.}

[dprevoznik]

Verified the new templates work with recorded replays. Similar resulting video for python and typescript templates.

replays.3.mp4

[tembo]

Nice migration overall — the templates read a lot simpler without Playwright/CDP plumbing and the replay recording flow is a good addition.

Main things I called out inline:

• Make session teardown a bit more robust (avoid leaving stale state around in TS; ensure Python cleanup always deletes the browser even if replay polling fails).
• Small TS tool polish: remove now-unused import, and default scroll coordinates to the tracked mouse position instead of hardcoding (0,0).
• Preserve unexpected Anthropic content blocks in _response_to_params to avoid silently dropping future/new block types.

None of these block the PR, just small hardening/maintainability tweaks.

[dprevoznik]

working on tembo / cursor comments now

[rgarcia]

Good refactor from Playwright to Kernel Computer Controls API. The Python implementation looks solid. Main issues are in the TypeScript side:

• Key mappings need to use X11 keysym names (Python has correct mappings to reference)
• Minor naming/cleanup items

The existing comments from previous review cover the session cleanup and scroll behavior edge cases well.

[dprevoznik]

@rgarcia all fixed and tested the templates. tagging in case you want to do another check

[tnsardesai]

LGTM. should we consider adding an api to get cursor position?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}