Harness Engineering Case Studies: What 4 Companies Changed Beyond the Model

Harness engineering is the practice of designing the instructions, tools, verification, permissions, and execution environment outside the model so that AI agents don't repeat the same mistakes.

From February to August 2026, OpenAI, LangChain, Anthropic, and Uber successively published case studies in which scores, quality, and cost shifted significantly depending on how the harness was built. This article compares the four companies' cases in terms of "what was changed and what changed as a result," and also introduces an October 2026 study in which, when only the harness was swapped for the same model, the difference in accuracy fell within the margin of error while costs differed by up to 3x. It is structured so that development leads and tech leads who have just started using Claude Code or Codex with their teams can decide what order to set things up in first. The basics of the terminology are explained in What Is Harness Engineering?.

What the four companies' cases show is that a harness can be a means of improving results, but it can also be a cause of degraded quality, and a factor that determines cost.

To cut to the conclusion: there were cases where a harness raised scores and cases where it degraded quality, and in the studies that measured cost—by Uber and Shandong University—both showed large differences.

The figures in the table are as published by each company and the researchers.

When reading these cases, the organization provided in Yuichiro Kinoshita's (Key Planning Inc.) seminar materials, "Introduction to Harness Design" (September 2026), is helpful. These materials divide everything outside the model into three layers: "harness" (AGENTS.md, Agent Skills) that conveys the route before running, "fences" (linters, type checks, tests, hooks) that mechanically judge deviations after running, and "boundaries" (sandboxes, blocking communication, least privilege) that decide the reachable scope from the start. The harness only works probabilistically. Fences judge mechanically according to set criteria, but some, like tests, can produce fluctuating results depending on the execution environment. Boundaries use isolation, communication restrictions, and least privilege to stop access or execution outside the permitted scope.

OpenAI's internal team decided to have Codex write all the code, with humans writing zero lines. What they spent their time on instead was building an environment where the agent wouldn't get lost.

The article OpenAI published on February 11, 2026—six days after Mitchell Hashimoto called this effort "harness engineering"—is a practical example of this.

If humans decide to write zero lines, what do engineers do? According to OpenAI's report, the internal team spent five months building a beta version of a product with internal users and external trial users. The repository reached about 1 million lines, and Codex wrote everything—app logic, tests, CI configuration, documentation, and even monitoring tools. Zero lines of code were hand-written (InfoQ article).

The team, which started with 3 people, merged about 1,500 pull requests, averaging 3.5 per person per day. When the team grew to 7 people, throughput increased without hitting a ceiling. A single Codex run would often continue a task for about 6 hours, including while the engineer was asleep.

The faster things moved, the more human review became the bottleneck. Ryan Lopopolo, who led this effort, said in a conversation that most human review now happens after merging, and that review and fixes are also left to the agent. The work of humans shifted from writing code to designing the environment, conveying intent, and building feedback mechanisms.

Speed emerged only after they organized the knowledge the agent references and the rules enforced mechanically.

First, they treated AGENTS.md not as an encyclopedia but as a table of contents. They kept it to about 100 lines, separating design documents, execution plans, and specifications into documents under docs/, with AGENTS.md containing only their locations. Links between documents and consistency of structure were also checked mechanically with a linter and CI.

Next was a custom linter whose error messages included how to fix the issue. Since the agent reads the error message when something fails, the message itself becomes the instruction for the next step. Dependency direction was fixed in the order of six layers, from types to screens, and references crossing layers were stopped by structural tests.

Finally, there was regular cleanup. The agent imitates the writing style found in the repository, good or bad. They wrote the principles to be followed into the repository and ran a regular process to clean up deviations, preventing bad patterns from spreading. This process is called garbage collection, by analogy with the mechanism that reclaims unused memory.

LangChain kept its model fixed and only fixed the harness of its own coding agent, raising its Terminal-Bench 2.0 score by 13.7 points.

According to the company's article published on February 17, 2026, what made the difference was not the phrasing of the prompt but a mechanism that has the agent verify its own work.

Terminal-Bench 2.0 is a benchmark that measures the capability of terminal-operating agents across 89 tasks spanning machine learning, debugging, biology, and more. LangChain's coding agent (deepagents-cli), with the default prompt and standard tool configuration, scored 52.8%. On the leaderboard, this fell just short of the top 30.

With the model fixed to OpenAI's coding-oriented model, repeatedly fixing only the harness brought the score up to 66.5%. This 13.7-point gap was enough to place it in the top 5.

The process of improvement itself was also turned into a system. The procedure for extracting execution records (traces) from LangSmith and summarizing the causes of failure was formalized as Agent Skills, making it possible to analyze things the same way every time. The traces revealed failures such as reasoning errors, failure to follow task instructions, missing tests or verification, and running out of time.

The first thing LangChain addressed was the moment just before the agent finishes its work. Models tend to stop at the first plausible answer they find. So they built in a mechanism that interrupts the process right when the agent tries to finish, prompting it to verify against the task specification.

The second was advance notice of time. Terminal-Bench has strict time limits, and agents are poor at estimating time. They inserted warnings about remaining time, prompting the agent to wrap up work and move to verification.

The third was detection of circular loops. There were records of the agent sticking to a policy it had once decided on and repeating minor fixes to the same file more than 10 times. They count the number of edits per file, and when it exceeds a certain threshold, they add context prompting the agent to reconsider its approach.

They also changed the allocation of reasoning effort. Setting maximum reasoning effort across the entire process increased timeouts, resulting in 53.9%; lowering it by one level across the entire process gave 63.6%. LangChain adopted an allocation that puts heavier reasoning into planning and final verification, and combined with the other improvements, this reached 66.5%. When they tried a Claude model with an older version of the harness that hadn't gone through the same improvement cycle, it stayed at 59.6%, showing that the harness needs to be tailored to each model.

A harness doesn't only raise scores. In April 2026, Anthropic announced that a quality degradation that had persisted in Claude Code from March into April was caused not by the model but by three changes on the harness side.

The API and inference infrastructure were unaffected, and all three issues were resolved in the April 20 release (v2.1.116).

The first was the default reasoning effort. On March 4, to reduce the problem of the screen appearing to freeze during long thinking periods, Claude Code's default reasoning effort was lowered from "high" to "medium." Complaints about reduced intelligence continued, and it was reverted on April 7. Anthropic itself acknowledged this was the wrong trade-off.

The second was a bug introduced during cache optimization. On March 26, a change was made to clear old thought records once when resuming a session that had been left idle for more than an hour. However, due to a bug, it ended up being cleared every turn thereafter as well. Claude continued working without retaining the memory of why it had chosen a given action, appearing forgetful and repeating the same things. The cache also kept missing, which also led to reports of usage limits being consumed faster than expected. The fix was made on April 10.

The third was a single line in the system prompt. On April 16, an instruction was added to shorten the text between tool calls, which, combined with other prompt changes, degraded coding quality. Removing instructions one by one and measuring the impact revealed that this one caused a 3% quality drop on one evaluation. This instruction was revoked on April 20.

The three changes each affected different users on different schedules, so as a whole they looked like scattered, inconsistent degradation, hard to distinguish from the usual fluctuation in feedback. The bug that erased thinking records had passed human and automated code review, unit tests, E2E tests, automated verification, and internal use. This was because two unrelated internal experiments were masking the symptoms.

No matter how many layers of fences you put up, some changes will slip through. Anthropic's recurrence-prevention measures are built on that premise. Every time they change Claude Code's system prompt, they now run broad evaluations across models, keep running verifications that remove one line at a time to measure its impact, and have built tools that make it easier to confirm prompt changes. For changes that could trade off against the model's capability, they are adding wait-and-watch periods, broad evaluations, and staged rollouts. They also say they will increase the proportion of employees using the same build as the public release.

What's interesting is that the team that builds Claude Code itself added guidance to CLAUDE.md so that model-specific changes only take effect for the targeted model. The makers of the harness are governing their own work with the harness.

As the number of users and requests grew substantially, Uber revisited its harness configuration and the way it hands tools to agents, keeping its total AI costs roughly flat since April.

The scale of usage underlying this is also large: at Uber, agents are involved in over 70% of pull requests, and employees have created more than 3,600 Agent Skills. These figures were published in the company's engineering blog on August 27, 2026.

Uber breaks down its total AI costs into the product of the following six terms.

Number of users × sessions per user × turns per session × requests per turn × tokens per request × price per token

The first two terms represent the breadth of usage, and it's fine for them to grow. The middle three terms represent overhead that the agent generates on its own, which can be reduced through harness improvements. The final term, the unit price, is determined by which model is assigned to which task. The advantage of this breakdown is that it lets you split a single number—"AI costs went up"—into growth that's acceptable and growth that should be reduced.

From February to August, weekly users of agents increased 7-fold, and weekly requests increased 9.4-fold. Even so, when comparing with the model held fixed, from February to July, the cost per 1,000 requests dropped about 34% from its peak, and the cost per session dropped 52% from its June peak.

The single most effective change was the default model for sub-agents. Because sub-agents handle tasks with fixed inputs and scope, many of these tasks don't require top-tier reasoning. Uber set the default to a cheaper model, let humans switch it only when needed, and left task decomposition and evaluation to the main model.

For sessions where a person interacts directly, Uber extended the prompt cache retention time from the default 5 minutes to 1 hour. This was because every time an engineer left a session idle for more than 5 minutes, the cache would expire, and the conversation history would have to be resent at full price. Sub-agents handling just a single short task are kept at the 5-minute setting.

They also changed how tools are handed over. In the approach Uber had been using until then, all tool definitions were loaded into the session regardless of whether they would be used. Once the number of tools exceeded 100, the definitions alone amounted to roughly 50,000–70,000 tokens, and these were resent with every turn. Uber made its over 1,000 internal MCP tools callable as commands (CLI), switching to a system that searches for and loads only the tools actually needed.

They also reduced the effort spent searching. They built an internal knowledge graph (roughly 24 million nodes and 80 million connections) linking internal services, teams, incident records, data usage, and more, so that agents could query it using natural language. For the same question, an agent using the graph answered in 38 seconds. An agent not using the graph took 20 minutes, spawned 2 sub-agents, produced 3 errors, and ultimately reached the mistaken conclusion that the data in question could not be queried.

It doesn't always go up. Under the comparison conditions in a Shandong University study published in October 2026, the accuracy of Claude Code and mini-SWE-agent was judged equivalent within 5 points, while costs differed by as much as 3x.

The research team ran five models across three harnesses—Claude Code, OpenCode, and mini-SWE-agent—and compared them on SWE-bench Verified, a collection of software development tasks. The equivalence judgment came from comparing the heaviest harness, Claude Code, against the lightest, mini-SWE-agent, across 447 tasks and 2 models. Among 45 difficult tasks, the proportion of tasks whose correctness flipped when the harness was switched, and the proportion that flipped when the same harness was simply rerun, were both 13%. The only difference exceeding margin of error was OpenCode underperforming by up to 9 points.

The scope of applicability is limited. The results come primarily from a single benchmark, and the authors explicitly note that the situation could differ for general-purpose agents, and that it hasn't been confirmed whether the same gap would appear with non-public frontier models.

On the other hand, the cost per task differed by as much as 3x even with the same model, the same task, and the same pricing table. What drove the difference was the preamble (system prompt and tool definitions) sent with every call. Claude Code used 16,581 tokens, OpenCode used 7,025 tokens, and mini-SWE-agent used 829 tokens—resent at every step. This is the same logic as Uber removing MCP definitions from the context.

The evaluation environment also affects results. In February 2026, Anthropic reported that running Terminal-Bench 2.0 with the same model, same harness, and same tasks, but varying only the CPU and memory allocation per task, caused performance to shift by as much as 6 points. The company's view is that on leaderboards where evaluation settings aren't standardized, differences of less than 3 points should be treated with suspicion.

There are also large improvements, like LangChain's 13.7-point gain. Still, when your own team changes harnesses, run the same configuration multiple times first to measure the range of variance, and make sure to record the costs alongside it.

The quickest way to bring these case studies into your own team is to set them up in the order of boundaries, rules, and verification gates and loops.

Kinoshita's material also points out that proceeding in the reverse order piles up rules that don't work and tools that don't get used. Below, we overlay the case studies from the four companies onto the procedure from that same material.

The first thing to decide is the scope the agent can touch. Run it in a sandbox or a cloud-based work environment, and don't hand over production credentials. The fence isn't perfect, but that's fine, because accidents that can't be undone—like deleting production data or sending confidential information—are too late to handle once detected. Even in Anthropic's case, a change that passed multiple rounds of review and testing still degraded quality.

The danger becomes greatest when three things are allowed at once: access to confidential data, reading untrusted content, and the ability to send data externally. Simon Willison called this the "lethal trifecta." For tasks that involve reading external pages or emails, you should assume that instructions can be rewritten via prompt injection, and structurally remove at least one of the three elements. How to divide permissions by tool is covered in How to Design AI Agent Tool Execution Permissions with Least Privilege.

Mitchell Hashimoto, the author of Ghostty, called the effort to fix the environment every time an agent fails—so that the same failure never happens again—"harness engineering" in an article dated February 5, 2026. There are two approaches. For simple failures, such as repeating a wrong command, you add one line to AGENTS.md. If that's not enough, you build a tool—such as a script that captures a screenshot or a script that runs a narrowly targeted test—and announce its existence in AGENTS.md. Hashimoto writes that each line in Ghostty's AGENTS.md was born from a specific instance of bad agent behavior, and that nearly all such behavior could be eliminated this way.

Addy Osmani similarly states that every line of a good AGENTS.md should be traceable back to a specific past failure, and recommends against adding lines on a whim. What should be written there is the project overview, build and test commands, project-specific conventions, and files that must not be touched. Common knowledge or things that can be learned by reading the code shouldn't be included, and lengthy procedures should be split out into Agent Skills, to be read only when needed. In Claude Code, CLAUDE.md serves the same role; examples of operating it are introduced in the Claude Code Team Adoption Guide.

Because rules only work probabilistically, whether they were followed should be judged by a machine. Combine the formatter, linter, type checker, build, and tests into a single command, so that both humans and agents pass through the same gate. If you write in AGENTS.md something like "done once this command fully passes," the agent can read the failure output, fix it itself, and run the loop of re-execution. The reason LangChain prompts verification right before finishing, and the reason OpenAI writes fix instructions into linter messages, is to speed up this very loop.

There are three places to run this gate. Agent-side hooks run the same verification at the moment of editing or execution; Git hooks run it before a commit or push; and CI runs it every time a pull request is created or updated. The more inward the check, the sooner issues are caught; the more outward, the fewer the loopholes. In CI, you should designate verification as a required check via branch protection, and also confirm that the settings don't allow bypassing by administrators or others—because GitHub's branch protection does not apply to administrators by default. The operational checklist is compiled in AI Agent Guardrail Operations Checklist.

We answer three questions: the lifespan of a harness, how to get started with a small team, and the state of support in Claude Code.

Some of it will become unnecessary, while some will remain. In a May 2026 article, Han Lee pointed out that much of the harness code—system prompts, tool wrappers, the separation of planning and execution, retry handling—gets absorbed into the next generation of models. He observed that teams shipping long-lived products treat the harness as a component built on the premise that it will be rebuilt every 90 days, and recommends building it that way. Meanwhile, Kinoshita's materials organize things differently, noting that tests, permissions and boundaries, and project-specific conventions persist even with the next model. An inventory review every few months—stripping rules down once and bringing back only what's needed—is essential.

This comes from Hashimoto's method. Create a short AGENTS.md (CLAUDE.md for Claude Code), decide on a single verification command and write it into the definition of done, and add one line each time a failure occurs. Cost breakdowns like Uber's can wait until after your user base grows and billing amounts start to become a concern.

AGENTS.md is usable. Starting with v2.1.277 in September 2026, Claude Code began reading AGENTS.md in projects that don't have a CLAUDE.md. If CLAUDE.md is present, it reads that instead, and it never reads both together (Claude Code changelog).

Agent Plugins is a common format for bundling Agent Skills and MCP configuration into a single folder for distribution, with 1.0 released on August 6, 2026. At launch it was supported by ChatGPT, Codex, Cursor, GitHub Copilot, Kiro, and VS Code; Claude Code is not included. Commands and hooks also fall outside the common format, so they still need to be prepared separately for each tool.

OpenAI and LangChain delivered results by speeding up the verification loop, Anthropic degraded quality through changes that slipped past the guardrails, and Uber cut costs by trimming the context sent each time and the effort spent searching. Research from Shandong University shows that if you want to claim an improvement in the harness, the difference needs to exceed the variance from simply rerunning the same configuration.

For your own team, start by drawing boundaries first, adding rules one line at a time from failures, consolidating verification into a single command, and letting the agent run the loop. Recording the variance in performance and the cost with every change will let you distinguish which measures worked from which didn't.

We support the in-house adoption of AI-driven development and the organizational structure needed for it. For more information, see AI Adoption Support.

References (accessed: October 28, 2026)

• Yuichiro Kinoshita, "Introduction to Harness Design: From Organizing Fundamentals to Stepping Up to Practice" (seminar materials, September 24, 2026)
• Mitchell Hashimoto, "My AI Adoption Journey" (February 5, 2026)
• OpenAI, "Harness engineering" (February 11, 2026)
• LangChain, "Improving Deep Agents with harness engineering" (February 17, 2026)
• Anthropic, "Infrastructure noise in agentic coding evals" (February 5, 2026)
• Anthropic, "An update on Claude Code quality reports" (April 23, 2026)
• Han Lee, "Hidden Technical Debt of AI Systems: Agent Harness" (May 8, 2026)
• Uber, "Running a Software Factory Efficiently at Uber Scale" (August 27, 2026)
• Vercel, "Introducing Agent Plugins" (August 6, 2026)
• Liu, Han, "What Does a Harness Buy? Tokens, Mostly" (October 3, 2026)
• Addy Osmani, "Agent Harness Engineering" (April 19, 2026)
• Simon Willison, "The lethal trifecta for AI agents" (June 16, 2025)