OpenAI computer use

OpenAI's computer tool, in the Responses API, looks at screenshots and answers with batches of mouse and keyboard actions. Your code runs those actions on a Mandala computer and sends back a fresh screenshot, until the model stops asking.

Let Mandala run the loop

For a hosted run, send your task and OpenAI key to Mandala. The platform calls OpenAI's Responses API, executes computer-action batches, and returns the result. It also provides a bash function that runs commands on the same computer. The computer must already be running.

curl https://app.mandala.computer/api/v1/computers/$COMPUTER_ID/agent \
  -H "Authorization: Bearer $MANDALA_API_KEY" \
  -H "X-Model-Key: $OPENAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"gpt-6.1-sol","prompt":"Open Firefox","max_steps":5,"stream":false}'

A gpt-* or o-series model selects OpenAI automatically. Alternatively, send "provider":"openai" to use the default gpt-6.1-sol. Unknown model namespaces require an explicit provider; a known model/provider mismatch is refused. The same selection works on /chat/completions with computer_id and messages. Use a model with computer-tool support; the provider reports unsupported models.

OpenAI bills your key directly. Mandala does not store it. Prompts, screenshots and command outputs go to OpenAI, with stored Responses used for continuation. Mandala AI credits currently fund Anthropic runs only.

Each run that acts or answers is kept in a thread on the computer, with its prompt and closing text, and returns a thread_id. Send that thread_id with the next prompt to continue the conversation, or "store": false to keep nothing. /chat/completions takes thread_id and store the same way. See continuing a conversation for the details, which are the same for both providers.

max_steps counts individual actions, including actions inside a batch and bash calls. A batch that exceeds the remaining allowance is not started. Each batch reserves rate budget for every action in it plus the screenshot after it, each at the price of one input or screenshot call; the initial screenshot and each bash call cost one call too. A batch, bash command or screenshot that fails ends the run with an error rather than a stop: the error has the status and message the platform answered that call with, and the steps already taken, including the actions the desktop acknowledged in the failing batch. Nothing is replayed automatically. A command that exits non-zero is not a failure; its output goes back to the model. A run that finished reports stop; after a failure, read the steps and look at the desktop before deciding whether to continue a task. An incomplete OpenAI response never has its actions executed. The response includes provider-reported input, output, cache-read and cache-write tokens for cost reconciliation.

Run your own loop

Your own loop keeps the OpenAI key on your machine and gives you control over model requests and tool execution. The following example uses this approach.

Before you start

pip install mandala-computer openai httpx

How the exchange works

  1. Send the task with tools=[{"type": "computer"}]. No display size is needed.
  2. The reply contains a computer_call whose actions array lists what to do in order: click, double_click, drag, move, scroll, keypress, type, wait and screenshot.
  3. Run the actions, take a screenshot, and send it back as a computer_call_output with the same call_id. Pass previous_response_id so the model keeps its context.
  4. Stop when a reply has no computer_call. Its text is the model's answer.

Coordinates are pixels in the screen space the computer's resolution reports (1280×800 unless you chose another at create time). A screenshot is usually that size, so the model's coordinates go straight to the SDK. If a screenshot's X-GC-Capture-Size header differs from resolution, or you shrink screenshots before sending them, scale the coordinates to resolution and round each to a whole pixel before acting. POST /computers/{id}/input truncates a fraction instead of rounding it, and the SDKs' pointer methods refuse one.

Forwarding actions

Post the actions array unchanged to POST /computers/{id}/input. The endpoint validates the whole array first, then runs the actions in order and stops on the first runtime failure. Key names are case-insensitive. OpenAI's arrow names (ARROWLEFT and the rest), META, CMD, COMMAND and OPTION are mapped to the platform's own; any other name must already be one the platform knows, such as ENTER, ESC, CTRL or an X11 name like Caps_Lock. An unknown name, such as CAPSLOCK, refuses the whole batch before anything runs. The endpoint holds modifier keys and performs a multi-point drag with one held mouse button. Drag paths contain {x,y} objects.

Scroll distances are approximate: 100 pixels per wheel notch, rounded to the nearest notch with at least one for a nonzero distance. Zero stays zero. Both axes are supported (vertical first), at most 50 notches combined and ±5000 pixels per axis. Actual scrolling depends on the application. Drags accept 2–128 points. Across an entire batch, typing accepts at most 400 Unicode characters, with the platform and application restrictions in the input reference.

The screenshot action is an acknowledgement marker. It does not return image bytes; fetch the screenshot separately after the actions. wait lasts one second. Do not retry a failed input automatically: earlier events may already have reached the application.

A batch has at most 32 actions, 128 total drag points, 50 total scroll notches, and 10 one-second waits. It uses one input session and a shared execution budget; the proxy allows up to 93 seconds. Runtime errors include completed, results, and a zero-based failed_index. Earlier actions and part of the failing action may have applied. Validation errors apply no input. Do not split and retry a rejected batch automatically; let the model reconsider using the error and screen.

Python

import base64
import sys
import os
import httpx

import mandala_computer
from mandala_computer import Client
from openai import OpenAI

MODEL = "gpt-6.1-sol"
TOOLS = [{"type": "computer"}]
MANDALA_BASE_URL = os.environ.get("MANDALA_BASE_URL", "https://app.mandala.computer/api/v1").rstrip("/")


def act(computer, actions):
    """Forward one batch. httpx does not retry this request."""
    reply = httpx.post(
        f"{MANDALA_BASE_URL}/computers/{computer.id}/input",
        headers={"Authorization": f"Bearer {os.environ['MANDALA_API_KEY']}"},
        json=[action.model_dump(mode="json", exclude_none=True) for action in actions],
        timeout=100,
    )
    reply.raise_for_status()


def screenshot(computer):
    png = computer.screenshot(fresh=True)
    return {"type": "computer_screenshot", "detail": "original",
            "image_url": "data:image/png;base64," + base64.b64encode(png).decode()}


def run(computer, task, max_turns=40):
    client = OpenAI()  # reads OPENAI_API_KEY
    response = client.responses.create(model=MODEL, tools=TOOLS, input=task)
    for _ in range(max_turns):
        if response.status != "completed":
            return f"The response stopped with status {response.status}."
        call = next((item for item in response.output if item.type == "computer_call"), None)
        if call is None:
            return response.output_text

        # Some actions need a person's approval first.
        checks = [{"id": s.id, "code": s.code, "message": s.message} for s in (getattr(call, "pending_safety_checks", None) or [])]
        for check in checks:
            print(f"Safety check {check['code']}: {check['message']}")
        if checks and input("Continue? [y/N] ").strip().lower() != "y":
            return "Stopped at a safety check."

        # Run the batch in order. Stop at the first failure and say what failed.
        problem = None
        actions = call.actions or ([call.action] if call.action else [])
        try:
            act(computer, actions)
        except httpx.HTTPStatusError as e:
            # Runtime errors report completed and failed_index. Do not replay.
            problem = f"Input failed; inspect the screenshot before deciding what to do: {e.response.text}"
        except (httpx.RequestError, ValueError) as e:
            problem = f"Input outcome is uncertain; do not replay the batch: {type(e).__name__}: {e}"

        output = {"type": "computer_call_output", "call_id": call.call_id, "output": screenshot(computer)}
        if checks:
            output["acknowledged_safety_checks"] = checks
        items = [output]
        if problem:
            items.append({"role": "user", "content": problem})
        response = client.responses.create(
            model=MODEL, tools=TOOLS, previous_response_id=response.id, input=items
        )
    return "Stopped after max_turns."


if __name__ == "__main__":
    task = " ".join(sys.argv[1:]) or "Open a terminal and show today's date."
    client = Client()  # reads MANDALA_API_KEY
    with client.computers.ephemeral(template="base", name="openai-computer-use") as computer:
        computer.wait_for_guest()
        print(run(computer, task))

ephemeral() deletes the computer when the block exits, even after an error. Use client.computers.launch(...) to keep it.

The OpenAI protocol has no error flag on a computer_call_output. When an action fails, the loop still sends the screenshot, and adds a short user message saying what failed. The model can see the state of the screen and try something else.

Making it better

For code-execution style computer use, where the model writes a script instead of returning actions, give the model a function tool that runs its script with computer.exec(...) and returns the output and a screenshot.

See also: Claude computer use for the TypeScript method names, and Troubleshooting.