September 13, 2026 / 18 min read

The last level of coding-agent autonomy is a harness problem

Humans are reviewing less code, AI reviewers are filling the gap, and the teams that let agents merge unattended got there through harness work. Here is what your repo needs and how to start today.

Coding AgentsSoftware DevelopmentAI DeliveryHarness Engineering

Since the MoA post I've been letting agents run more and more in the background, and lately I've been wondering if I was being reckless or just early. So I did what I always do when I want to check my gut, I pointed my multi-research skill at the question. Four different model families, each one doing its own web search without seeing the others, then a merge step where I look at where they agree, where they disagree and what only one of them found. Same harness idea from the MoA post, just pointed at reading instead of coding.

The question was roughly: what are the levels of autonomy for coding agents, who is actually operating at the top one, and what does a small company need to get there. This post is what came out of it, mixed with what I actually do in my own projects, so you, the reader, can decide how much of it applies to yours.

The levels, quickly

There is no official standard, no SAE thing for coding agents, I checked. The paper that gets cited the most (Feng, McDonald and Zhang, Knight Columbia)¹ defines five levels by what the human does:

Operator. You drive the planning and the decisions, the agent suggests and helps on demand.
Collaborator. The agent drafts the plan, you edit it and you split the tasks between the two of you.
Consultant. The agent plans and executes over long stretches and comes back to you for feedback on the key decisions.
Approver. You only hear from the agent when it hits a blocker, needs an approval or needs a credential.
Observer. The agent runs fully on its own, you can read the logs and that's it.

They also make a point I kept coming back to during the whole research, autonomy is a design decision you make separately from how capable the model is.

The practitioner ladders look more like what you'd expect, bottom to top:

Autocomplete. The model suggests the next lines, you accept them or you don't.
Agent PRs, human review. The agent takes a ticket and opens a PR, a person reads every diff.
Automated review. Review agents read the PR before a person does, and they can block the merge.
Selective auto-merge. Routine classes of change merge on green CI with nobody reading the diff.
Mostly autonomous inside a scope. The agent owns a defined area end to end, humans set the boundaries and check the evidence.
Dark factory. Spec in, deployed software out, nobody reads anything.

Both ladders top out somewhere nobody actually lives. I could not find a company running the dark factory on a real product, and "Observer" as a steady state on a production codebase didn't show up either. So the last level in this post is the last occupied one: an agent takes a ticket, and a class of changes merges without a person reading the diff. Nobody does that for everything. Several companies do it for a lot, and the line they draw between the two is the subject of the rest of this post.

We are reviewing less code, whether we admit it or not

Let me start with the uncomfortable one. Faros AI looked at telemetry from 22,000 developers this year and found that under high AI adoption the median time a PR sits in review went up about 440%, incidents per PR about 240% and code churn about 860%². Three different signals, all pointing the same way.

More time in review would be fine if it bought more eyes on the code. IEEE Spectrum ran a piece this month suggesting it doesn't: one of the people interviewed calls the result "theater approval", engineers approving code they skimmed and don't fully understand³. The queue got longer, the reading didn't get deeper. And Anthropic says more than 80% of the code merged into their codebase in May this year was authored by Claude and, in the same report, that human code review has become their new bottleneck⁴.

I'd be lying if I said I read every line of every agent PR. When it's the tenth one of the day I read the description, I read the test names, and the diff gets a scroll. I noticed I was doing it and I didn't like it, which is more or less why this post exists.

AI reviewers are filling the gap

The companies that got past that queue did it by automating the review itself, and this was the most interesting part of the whole research run. What they automate varies. What they keep for a human varies more, and that second half is the one worth reading.

Cursor runs four always-on security agents over their own PRs, north of 3,000 a week, and the important detail is that the security reviewer can block CI⁵. People still review the code.
Anthropic runs an automated Claude reviewer over every PR for architectural defects, security flaws and regressions before merge, and it caught about a third of the bugs behind their historical outages on claude.ai⁶. Human review is still the constraint they are designing around, not one they removed.
Razorpay's internal agent reviews with one sub-agent per dimension, bugs, security, quality, design system, i18n, plus a pre-mortem, then a filter that throws away the false positives⁷. More than a third of their PRs now merge with no human in the loop, and the classes that qualify are the low-severity ones.
Checkout.com's agent HAL opens 18% of all their PRs, and the review is done by a separate persona with a fresh context that never saw the implementation reasoning⁸. A human still has the final approval.
Augment has a Risk Analyzer agent that auto-approves low-risk PRs, and they report 72% lower time to merge and a 14-day revert rate down from 1.9% to 0.4%⁹. Anything it doesn't classify as low-risk goes to a person.

One caveat before you go and install a reviewer bot. A reviewer agent that reads PR text and has merge authority is an attack surface. Never let an agent treat issue or PR text as instructions, and keep the credentials away from the thing that reasons.

In my own harness this is the multi-review and multi-security skills, the implementer is one model family and the reviewers are others, and yes, the second family keeps catching stuff the first one is blind to, which was more or less the point of the MoA post.

Your harness sets your level

What separates the companies above from everybody else is the environment around the agent, and the sources are unusually consistent about it. Raw model capability barely shows up as the differentiator, which is not the same as saying the model doesn't matter: every setup above puts a different reviewer, with a different context and often a different model, on the other side of the work. Capability isn't the scarce thing. Separation is.

Stripe is the cleanest example. Their Minions ship 1,000+ merged PRs a week with no human-written code, on devboxes that boot in about 10 seconds with no internet and no prod access, and each task is capped at two CI runs because of cost¹⁰. Every one of those PRs is still human reviewed, by the way. One of my researchers reported Minions as "unattended", another one read the same Stripe post and said no, every PR is reviewed. I went back to the source and the careful one was right. Even the research needs a verifier.

Factory scores the repo instead of the agent: eight pillars, five levels, and you unlock a level by passing 80% of its criteria plus everything below it. Their published examples are humbling. CockroachDB reaches L4 on the back of extensive CI, comprehensive testing, clear documentation and security scanning. FastAPI sits at L3, Express at L2 for missing foundational signals¹¹. Those are mature, heavily engineered open-source projects, none of them at the top of the scale, which is a decent hint about where your repo lands. There is an open-source version from Kodus, 39 checks across 7 pillars¹². I ended up writing my own as a Claude Code skill, harness-audit, 20 controls, because I wanted the remediation backlog to come out as tickets.

DORA's 2026 ROI research is the piece that made this click for me. AI is an amplifier of what your org already does, and they model the return as a J-curve: an initial dip from verification burden and workflow rework, which is exactly the phase leaders read as failure¹³. The uncomfortable implication is that strong developers on a mature repo, without an agent-shaped harness, can go backwards before they go forwards.

And this is where "what to review" comes in. The harness decides what a human has to read. If the type checker, the tests, the lint rules and the review agents are trustworthy, the human reviews the spec, the evidence, and the diff of anything that touched something scary. If they aren't trustworthy, the human has to read everything, and we already saw that we don't.

What I hand over completely, and what I don't

There is a study of 33,000 agent-authored PRs that gets close to answering the question¹⁴. Documentation, CI and build updates had the best merge outcomes, performance work and bug fixes had the worst, and the PRs that didn't merge were bigger and more often failing CI. At the extreme end, Spotify's fleet has merged more than 2.5 million automated maintenance PRs, the vast majority with no human in the loop¹⁵.

Be careful with what that study measures, though. Merge rate tells you where agents currently do well, not what is safe to merge unattended. Bug fixes merging worse might mean they need more iterations, not that they need a person reading the diff. It answers the capability question and leaves the risk question to you.

Which is why I ask two separate ones per ticket. Can a machine check that this is done, without me re-reading the diff? And if it lands wrong, how bad is it and how fast can I undo it? Handing over completely needs a yes to the first and a cheap answer to the second, and the two almost never fail for the same reason. The result is boring on purpose.

The work I hand over completely, machine-checkable and cheap to undo:

dependency bumps with green CI
flaky test repair
lint and codemod migrations
documentation that has to match the code
backfilling tests for code that already works
bugs where I already wrote the failing test

The work where a human reads the diff every time, and the reason is different in almost every row:

anything touching auth, payments or permissions (a machine can check this fine, the blast radius is what stops it)
database migrations and anything that deletes data (same, except here "undo" isn't a revert)
public API contracts (cheap to merge, expensive to unmerge once somebody depends on them)
performance work (this one the agents just do badly, they optimize what they can measure and break what they can't)¹⁴
anything where I can't write down the check that proves it's done (fails the first question outright)

That last bullet is the actual rule. If I can't express "done" as something a machine can verify, the ticket isn't ready to hand over, and most of the time the fix is in the spec. The rows above it are the other axis: perfectly verifiable, still not mine to automate, because being right 99% of the time is a different proposition when the 1% is a payment.

Is there anything we won't hand over any time soon?

Predictions in this space age badly, so take this one with salt.

What I think stays with humans for a while: deciding what to build, and owning the outcome when it's wrong. Berkeley RDI's argument is that when the same agent writes the implementation and the tests, passing tests demonstrate consistency rather than correctness¹⁶, and even with a separate verifier somebody has to decide the spec itself was right. Code that compiles and passes naive checks can still drift, slowly, away from what the product was supposed to do.

There is a governance paper that puts names on the two questions from the last section, and I found it really useful: the Autonomous Capability Level, what the agent can do, and the Allowed Autonomy Level, what you let it do given the risk and the oversight you actually have¹⁷. Their worked example is a deployed enterprise data engineering agent, assessed as highly capable, deliberately pinned to a lower autonomy level because of risk, reversibility and how ready the org was. That's the framing I'd use. Irreversible stuff, money, production data, access policy, anything that binds the company to a customer, stays behind a human approval regardless of how good the model gets. I wrote about the control-plane side of that in the previous post.

And one that people avoid saying out loud, which follows straight from the thesis. If the harness decides what a human has to read, it also decides what a junior ever gets to read, and reviewing other people's diffs is a large part of how engineers learn a codebase. Seniors reviewing less, plus juniors being hired less, breaks the pipeline that produces seniors. Bonterra's CTO in the IEEE piece: "If the industry stops hiring juniors, the industry stops producing seniors"³. I don't have a fix for that one, beyond deciding on purpose which reviews stay human for reasons that have nothing to do with risk.

What your project needs before an agent works unattended

This is the list I actually use, it's what harness-audit grades, and it's what decides whether I let something run in the background. It's long, I know, that's kind of the point of the post. The examples are Node and TypeScript because that's what this site is; the controls aren't.

Full isolation, per agent. Each agent gets its own worktree, its own environment and its own database. Git worktrees are the cheap version, I have a worktree skill that sets them up from a bare clone, and please don't put them under /tmp, on a lot of Linux setups that's a RAM disk and a few worktrees with node_modules and .next inside will quietly eat your memory until you remove the worktrees or reboot. My worktree setup checks the filesystem type and refuses before it puts anything there. A devcontainer gives the agent the same toolchain every time. Docker Compose with the project name and the ports parameterized per worktree, so the whole thing can run four or five times on the same machine without fighting over 3000 or 5432.

CI runnable locally, with one command. If the truth only lives in GitHub Actions the agent's feedback loop is ten minutes long and it costs money on every iteration, remember Stripe capping at two CI runs. Make CI call the same script the agent calls, npm run verify or a Makefile target, lint, typecheck, tests, build. For the GitHub-specific parts, act runs the workflow locally.

Self-verification tools. Unit and integration tests, obviously. Playwright for anything with a UI, I give the agent the Playwright MCP so it can click through the thing it just built and hand me a screenshot as evidence. A local database it can seed, migrate and query, with fixtures. Per-PR preview deploys if your host does them (Vercel does). The standard I hold agents to is proof over vibes: CI logs, test output, screenshots, and the human reviews the evidence instead of re-deriving it.

Mutation testing on a schedule. I don't see people talk about this one. An agent will happily write tests that pass, and tests that pass tell you nothing about whether they constrain behaviour. Stryker (JS/TS, C#, Scala) mutates your code and checks whether the tests notice; PIT does the same for Java and Kotlin, mutmut and Cosmic Ray for Python. Run it weekly rather than on every PR, it's slow, and turn every surviving mutant into a ticket. Improving existing tests is exactly the kind of bounded, machine-checkable task from the list above, so the agent that eroded your test suite's credibility can also be the one that fixes it.

Implementer and verifier are different agents. Different session, fresh context, ideally a different model family. The verifier gets the ticket, the spec and the diff, never the implementer's chat. That's what Checkout.com does with HAL's review persona, what Razorpay does with its sub-agents, and what Berkeley RDI says is the minimum to get from consistency to correctness. In my harness it's the multi-review step, in yours it can be a second Claude Code session with a reviewer prompt and no memory of the first one.

Spec first, sized to the ticket. For a one-liner, the ticket is enough. For anything bigger I use OpenSpec, propose, then the spec with requirements and scenarios, then tasks, and I only hand it to the agents once I've read the requirements and they look right. I've been using it on this very site. Reading 200 lines of spec beats reading 2,000 lines of diff, and it's where the "what does done mean" question gets answered before an agent burns tokens on the wrong thing.

Guardrails that are code. An AGENTS.md with the build, test and lint commands and the invariants of the repo. Branch protection with required checks. GitHub Actions pinned to commit SHAs. Lint rules that ban the patterns you don't want, no-restricted-imports and friends, because a rule in CI is a rule the agent can't argue with, while a rule in a markdown file is a suggestion. Credentials outside the reasoning layer, see the CVE above.

A revert path you have actually used. Everything above is about catching a bad change before it lands. This one is about the ones that land anyway, and for unattended merging it's the control that decides how much you can hand over at all. Small PRs, so a revert is a revert and not an archaeology project. Feature flags for anything with a runtime effect, so the undo doesn't need a deploy. A rollback you can trigger with one command and have rehearsed at least once. And alerting that tells you a bad merge reached production before a customer does. The governance framing above puts reversibility right next to risk for exactly this reason: auto-merge is a bet that the cost of being wrong is bounded, and this is the part that bounds it.

How to start today

You don't need a platform team for any of this. The first three steps are a couple of weeks and $20 a seat, the rest is a quarter of chipping at it.

Score the repo before buying anything. Factory's readiness report or the Kodus open-source one. You'll get a level and a gap list.
Close the cheap gaps first. AGENTS.md, a devcontainer, worktrees so two agents can't fight over the same checkout, one command that runs what CI runs in under ten minutes, branch protection. This is the highest-ROI step and it's the one people skip because it's boring.
Hand over one boring class of work. Dependency bumps or test backfill. Agents in worktrees, and a human reads every PR.
Add a reviewer that can block CI, with a different model family than the implementer if you can manage it.
Expect the dip. DORA's J-curve is real, the first weeks feel slower because you're building verification, that's the investment, don't abort during it.
Auto-merge exactly one category. Dependency patches with green CI is the classic. Add a hold label as a human veto before you widen it.

Three numbers tell you whether any of this is working, and you want them from day one rather than from the day you widen the auto-merge rule. First-pass acceptance rate, meaning how many agent PRs pass review with no changes requested. Revert rate. And code churn at two weeks, meaning how much of what merged got rewritten. That last one is in the Faros data at the top of this post, which is convenient: it means you can tell whether you're the exception or the median.

On tools: Claude Code, Codex or Cursor at $20 a month is the entry ticket, and for background work you already have options in the same price range, GitHub Copilot's coding agent, Google's Jules, Codex cloud, or Claude Code running headless from a cron. All four of my researchers converged on the same advice, buy more harness before you buy more model, a $20 tool in a repo with fast CI and a real AGENTS.md beats a $500 autonomy product in a repo that isn't ready.

Conclusion

Humans are reviewing less code and the data says it's already costing us in incidents and churn. The companies at the practical top of the autonomy ladder got there by automating review with adversarial agents and by building an environment where "done" is something a machine can check, and then they only let the boring, bounded classes of work merge alone. Raw model capability mattered less than any of us assumed; keeping the reviewer separate from the implementer mattered more.

For a small team the good news is that the whole pattern scales down, one person who owns the context files and one reviewer is enough to start. The bad news is that there is no shortcut through the harness work, and the first weeks feel slower.

My own research run was a nice reminder of the whole thesis, four researchers, one of them misread a primary source, and the merge step caught it because a second reader looked at the same evidence. Implementer and verifier, again.

Next time I'll go through the harness-audit controls one by one, and probably the day I had to add the /tmp warning to the worktree skill.