Harnesses: Why the Model Is Not the Product

A language model only turns text into text. Reading files, running tests, remembering the plan and asking before it deletes something are all jobs done by ordinary software around the model. That software is the harness. With today's models, the harness often decides whether an agent works at all.

The model is not the product

A harness, also called a scaffold, is all the code between the model and the world. It decides what the model sees, which actions it can take, what happens to those actions and when the work stops. An agent is the model plus its harness. Neither one is an agent on its own.

You can see the split in public benchmark results. On software-engineering task suites, leaderboards record the scaffold next to the model because the same model scores differently in different scaffolds. The SWE-agent authors showed that redesigning the tools alone, without changing the model, clearly raised the share of real GitHub issues the agent resolved. Anthropic's write-up of its SWE-bench agent reports the same thing: the team spent more time refining the tools than the prompt.

In practice, this matters in three ways:

• Upgrades go in both directions. A stronger model in a weak harness can lose to a weaker model in a well-designed one.
• Most failures are harness failures. Missing context, confusing tool output and an unbounded loop look like "the model is dumb", but you fix them in code.
• The harness is where you have control. You cannot retrain a frontier model, but you can change every layer around it today.

The agent loop

At the centre of every harness is a loop. The harness sends the model the conversation so far plus a list of tool definitions. The model replies with text, one or more tool calls, or both. The harness runs each allowed call, adds the results to the conversation and calls the model again. When the model replies without asking for a tool, the harness treats that reply as the final answer. If the task hits a step, token or time limit first, the harness stops it.

A minimal harness in 30 lines

The core fits on one screen. The sketch below has two tools, a permission check, error handling and a step limit. Everything a production harness adds is a refinement of one of these lines.

# Pseudo-code: call_model() stands in for any chat API with tool calling,
# TOOL_SCHEMAS for the JSON Schema definitions of the tools below.
import subprocess

TOOLS = {
    "read_file": lambda path: open(path).read()[:20_000],          # cap output size
    "run": lambda cmd: subprocess.run(cmd, shell=True, capture_output=True,
                                      text=True, timeout=120).stdout[-10_000:],
}
SAFE_PREFIXES = ("ls", "git status", "git diff", "pytest")       # naive allowlist

def allowed(name, args):
    if name == "run" and not args["cmd"].startswith(SAFE_PREFIXES):
        return input(f"Allow `{args['cmd']}`? [y/N] ").strip().lower() == "y"
    return True

def run_agent(task, system_prompt, max_steps=30):
    messages = [{"role": "system", "content": system_prompt},
                {"role": "user", "content": task}]
    for _ in range(max_steps):
        reply = call_model(messages, tools=TOOL_SCHEMAS)
        messages.append(reply)
        if not reply.tool_calls:                    # no tool request: final answer
            return reply.content
        for call in reply.tool_calls:
            if not allowed(call.name, call.args):
                result = "DENIED by the user. Try a different approach."
            else:
                try:
                    result = TOOLS[call.name](**call.args)
                except Exception as exc:            # errors go back to the model
                    result = f"ERROR: {exc}"
            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    return "Stopped: step limit reached."

The allowlist here is deliberately naive: pytest; rm -rf ~ starts with an allowed prefix. Real harnesses parse the command before deciding. That one detail shows where the engineering effort in a harness actually goes.

Instructions: system prompts and project files

The system prompt sets the agent's role, working style and safety rules. Harness vendors tune it to their tools, so it is long and specific. It says when to search before editing, how to report progress and which actions need confirmation.

Project knowledge comes from instruction files that the harness loads automatically: CLAUDE.md for Claude Code, GEMINI.md for Gemini CLI, and AGENTS.md, an open convention that several tools read, including OpenAI's Codex. These files are a README for the agent. They hold the commands to build and test, the conventions that are not obvious from the code, and the boundaries.

# AGENTS.md

## Commands
- Install: `uv sync`
- Test: `uv run pytest -q` (run before declaring a task done)
- Lint: `uv run ruff check .`

## Conventions
- Database migrations are generated, never hand-edited.
- Public functions need docstrings and type hints.

## Boundaries
- Do not modify anything under `infra/`.
- Ask before adding a new dependency.

Keep these files short and factual. Every line costs tokens on every task, and long lists of vague advice ("write clean code") dilute the rules that matter. A useful test for each line: would a new human colleague need to be told this?

Tools as the agent–computer interface

The SWE-agent paper introduced the term agent–computer interface (ACI): tools should be designed for a model's strengths and weaknesses, just as a UI is designed for people. Its authors gave the agent a file viewer that shows a bounded window of numbered lines, search commands that return concise summaries, and an edit command that runs a linter and rejects edits that break the syntax. Those changes mattered more than prompt tweaks.

Principles that carry over to any harness:

• Few, clear tools. Most coding harnesses settle on a small set: read a file, search by name and content, edit by exact string replacement or patch, run a shell command. Overlapping tools make the model hesitate between them.
• Bounded output. A test run that prints 50,000 lines will flood the context. Truncate, paginate, and tell the model how to see more.
• Errors that teach. "old_string not found; it appears 0 times in the file" lets the model recover. A raw stack trace often does not.
• Mistake-proof arguments. Anthropic describes switching a tool to require absolute paths after the model kept getting relative paths wrong. Change the interface rather than adding another warning to the prompt.

Tools can also come from outside the harness. Through the Model Context Protocol, a harness can connect to servers for issue trackers, databases or browsers without custom code. Every tool definition still takes up context, so connect the servers a task needs, not every server available. The mechanics are covered in agents, tool calling and MCP.

Managing the context window

A long agent session produces far more text than any context window holds: file contents, command output, failed attempts. Even before the hard limit, models attend less reliably to details buried in a huge context. Deciding what the model sees on each turn is context engineering, and harnesses use several techniques together.

Compaction and pruning

When the conversation approaches the limit, the harness asks the model to summarise the work so far (goal, decisions, files changed, open problems) and continues from that summary. A cheaper step is pruning, which replaces old tool outputs with short placeholders. Both lose detail, so good harnesses keep the task statement and the latest state intact.

Sub-agents with their own context

A search across a large codebase may read dozens of files to produce one useful paragraph. Delegating that job to a sub-agent, which has its own fresh context and returns only its conclusion, keeps the main conversation clean. The cost is extra tokens and the risk that the summary leaves out something important.

Memory files and selective reading

Models have no memory between sessions, so harnesses write to files: instruction files, notes, to-do lists the agent updates as it works. Within a session, coding harnesses generally do not load the whole repository up front. The agent searches and reads only the files it needs, just in time, as an engineer would.

Permissions, sandboxes and hooks

An agent that can run shell commands can do anything your user account can do. It also reads untrusted text, such as web pages, issue comments and dependency READMEs, where prompt injection can hide. Harnesses defend in layers.

Approval modes set how much the agent may do unprompted. They range from asking before every edit and command, through auto-approving edits inside the project, to fully autonomous runs. Allowlists and denylists refine this per command. These are human-in-the-loop checkpoints, and they only work if prompts are rare enough that people still read them.

Sandboxing limits the damage when approval fails. Options include OS-level sandboxes that restrict file writes to the workspace, containers or virtual machines, and network access that is off by default. Autonomous modes are only reasonable inside such boundaries.

Hooks run your code at fixed points in the loop: before a tool call, after an edit, when the agent finishes. Unlike instructions, they always run. In Claude Code, hooks are declared in a settings file:

{
  "hooks": {
    "PreToolUse": [
      { "matcher": "Bash",
        "hooks": [{ "type": "command", "command": ".claude/hooks/check-command.sh" }] }
    ],
    "PostToolUse": [
      { "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": ".claude/hooks/format-and-lint.sh" }] }
    ]
  }
}

The hook script receives the tool call as JSON on standard input. A pre-tool hook that exits with code 2 blocks the call, and its error message goes back to the model as feedback.

Observability and cost control

When an agent fails after forty steps, the final answer tells you almost nothing. You need the full trace: every model call with its input size, every tool call with its arguments, output and duration, every permission decision. Store traces in a structured form so you can search them, compare runs and replay a failure.

Cost needs the same attention. Agent loops resend the growing conversation on every call, so input tokens dominate. Harnesses keep this under control with prompt caching of the stable prefix (system prompt, tool definitions, instruction files), compaction, cheaper models for sub-tasks such as search or summaries, and hard budgets on steps, tokens and wall-clock time. A budget that ends a run with a clear message is better than an invoice that reveals the problem a month later.

Harnesses in practice, as of 2026

Products differ less in their models than in their design choices:

• Terminal coding agents such as Claude Code, OpenAI Codex CLI and Gemini CLI run in your repository with file and shell tools, load a project instruction file, and support MCP servers. They differ in their default approval modes, how they sandbox commands, and extension points such as hooks, sub-agents and custom commands. Codex CLI and Gemini CLI are open source, so their loops and prompts can be read directly.
• IDE agents such as Cursor or GitHub Copilot's agent mode use editor state (open files, selections, diagnostics) as context and present changes as reviewable diffs.
• Background or cloud agents run the same kind of loop in a remote container and deliver a branch or pull request. They trade interactivity for parallelism and isolation.
• Open-source frameworks and research harnesses such as SWE-agent, OpenHands and Aider make the scaffold itself the object of study. Aider, for instance, is known for its repository map and its compact edit formats.

The same layers appear outside coding. A business automation agent needs a loop, tools, permissions and traces just as much; LLM automation covers when a fixed workflow is the better choice.

Evaluating a harness

Because the harness changes outcomes, evaluate the model and harness together, and treat harness changes like code changes, with evals that run before release.

Public suites are the starting point. SWE-bench collects 2,294 real issues from 12 popular Python repositories. An issue counts as resolved only if, after the agent's patch, the tests from the project's real fix pass and the existing tests still pass. A human-validated subset, SWE-bench Verified, has 500 tasks and is widely quoted. Running one model through two scaffolds on such a suite measures the scaffold directly.

Your own suite matters more. Take closed issues or past tasks from your repositories, record the starting commit and a check that defines success, and replay them. For each run, record:

• Success rate, over several attempts per task, because agent runs are not deterministic;
• Cost in tokens and money per solved task, not per attempt;
• Time to completion and number of steps;
• Interventions: approvals requested, corrections a human had to make, and actions that were blocked.

For background on why context and tokens dominate these trade-offs, see what is a large language model.