Mapping the Minefield: Why Tests Matter More When Agents Write the Code

I was in the shower, thinking about better ways to verify code that AI generates. I started going through the different kinds of tests we have, and a minefield came to mind. From there I started mapping each kind of test to something you'd find in a minefield.

In that picture, a codebase is the minefield.

Each feature you ship buries a few mines. A mine is whatever the feature quietly depends on: a side effect another module relies on, an assumption about the shape of the data, a rule enforced in one place and assumed everywhere else. Step on one and something you weren't touching breaks.

The field gets denser as the product grows. Many of the mines were placed by people who have since moved on to other work, and you've forgotten most of the ones you placed yourself.

Making a change means crossing the field. You start at one edge with a task and need to reach the other edge with the task done and nothing broken. Most crossings go fine. The ones that don't usually show up in production.

The walkers changed

For most of software's history, few people crossed any given field, and they crossed slowly. The people who had been around longest carried a map in their heads. They knew which module nobody touched without asking first, which table had a trigger, and which old endpoint the mobile app still called. Newcomers learned the map by asking, by pairing, and occasionally by stepping on something.

That worked because crossings were rare and the people who knew the map were usually around to ask.

Coding agents changed who walks the field. An agent can cross it many times a day, much faster than a person, and each session starts close to zero. It can read what's written down. It can't ask the engineer who remembers why billing is fragile, and it won't remember what it learned yesterday unless someone wrote it down.

So the map has to live in the field itself. Documentation helps, but it describes mines without checking for them. A test is a marker that reacts: step where you shouldn't and it tells you before production does. For an agent, tests are the part of the map that checks itself.

The other half of the change is cost. Marking a field used to be slow, careful work, and most teams did less of it than they knew they should. A characterization suite for a legacy module, or a contract for every service boundary, rarely beat shipping the next feature. Agents can now do much of that work. Tests became more valuable and cheaper to write at the same time, and that changes the math for several kinds of tests teams used to skip.

Here's how each kind of test marks the field, from the smallest to the largest.

Unit tests: defusing on the bench

A unit test lifts one mine out of the field and puts it on the bench. You can open it, see exactly how it works, and change it with nothing nearby to set off. Unit tests are cheap and fast, so you can defuse a lot of mines this way.

The bench is also their limit. A mine on the bench has no tripwires. The test tells you how one function behaves with the inputs you gave it and nothing about what's wired to it in the field. When a unit test mocks its neighbors, it cuts the wires on purpose. Once the change goes back into the field, the mine is live again.

Agents work fastest against unit tests. A test that runs in milliseconds lets an agent check each step of a change instead of only the end result.

Characterization tests: sweeping a field nobody mapped

Sometimes you inherit a field with no records. Nobody remembers what's buried, and the people who buried it are gone. Before you change anything, you sweep the field and mark each mine exactly where you find it.

A characterization test records what existing code does today, including behavior that looks wrong. If an invoice total rounds in a strange way, the test pins that rounding. The point is to notice when your change moves something, and then decide on purpose whether it should move.

Sweeping was always the tedious part: reading old code, finding representative inputs, recording the outputs. An agent can sweep a legacy module in an afternoon, which makes it reasonable to do before every risky change to code nobody fully understands.

Integration tests: the safe path

An integration test walks one real route through several parts of the system and confirms it still holds. A request goes through the API, the service and the database, and the right answer comes back. That's a marked path across the field. Someone walked it once, and every test run walks it again.

Mines next to the path light up along the way. If a change to the service breaks how it talks to the database, the path catches it. Everything off the path stays dark. An integration test only lights what it walks past.

A set of safe paths lets an agent make a change and walk the routes that matter, without a person having to remember which routes those are.

Contract tests: the border treaty

Larger products are several fields side by side, each owned by a different team or service. The border between two fields is an API. A contract is the treaty at that border: at this crossing, travelers arrive carrying this request and leave with this response.

Each side checks its half of the border against the treaty, on its own schedule, without walking the neighbor's field. The failure it catches is easy to miss otherwise. You change what you guarantee at a crossing, your own paths still pass, your neighbor's paths still pass, and the first traveler across the border steps on a mine.

An agent usually works in one repository at a time and never sees the field next door. The contract is the only record of the border it can read.

End-to-end tests: the full crossing

An end-to-end test sends someone across the whole field, edge to edge, the way a real user travels: open the app, sign in, add something to the cart, pay, get the confirmation email. It's the only test that tells you the whole trip works, as opposed to each piece of it.

Full crossings are expensive. They're slow, they fail for reasons unrelated to your change, like a slow page or a flaky third-party service, and when one fails it can take a while to find which step went wrong. Most teams keep a few for the journeys that matter most and rely on cheaper tests for the rest.

Agents can drive a real browser and read what happened, so writing and repairing these tests costs less than it used to. Running them costs the same, so they still belong on the journeys that matter most.

Behavior-driven tests: the minefield record

Armies that lay mines are supposed to record them: where each one is, what type it is and why it's there, so the field can be cleared later. Behavior-driven tests keep that record for a product.

A scenario is written in plain language, usually before or while the feature is built:

Given a customer with an empty cart
When they open checkout
Then the pay button is disabled

The product owner can read it, the developer can run it, and both agreed on it before the mine went in.

Behavior-driven tests pair with characterization tests. A characterization test rebuilds a record nobody kept. A behavior-driven test keeps the record from the start, so the field never goes dark.

When an agent builds the feature, the scenario is the most direct way to tell it what done means, in words both the agent and the business can check.

Stress tests: anti-tank mines

Anti-tank mines don't go off when a person steps on them. They need the weight of a vehicle.

Every test so far crosses the field one walker at a time, and each one steps right over these mines. A stress test drives the convoy across: the whole user base at once, a launch-day spike, ten thousand jobs landing in the queue together. Connection pools run out. A lock that never mattered starts to matter. A query that took 20 milliseconds takes 20 seconds.

These mines only exist under load, so nothing lighter will find them.

An agent can make a change that passes every functional test and adds a database query for each item in a loop. Only the convoy notices. Load scripts also used to be specialist work, and an agent can now write a first version of them.

Chaos engineering: controlled detonation

Every test so far tries to keep mines from going off. Chaos engineering assumes one will go off eventually and sets one off on purpose: in the live field, at a time you choose, with the medics standing by. You stop a server, cut a network link or make a dependency slow, then watch.

The question is what happens after the blast. Does traffic reroute? Does the alarm sound? How far does the damage spread?

Stress testing adds weight to find mines that only go off under load. Chaos engineering picks a mine, sets it off and studies the aftermath.

When code ships faster than anyone can review all of it, some mines will go off. Knowing the village survives one matters more.

Evals: mines that go off sometimes

A normal mine is deterministic. Step on it and it goes off, every time. A feature built on an AI model buries a different kind of mine. Same step, same spot, and sometimes nothing happens.

One walk proves nothing, so evals walk the same path hundreds of times and keep a tally: how often the summary missed the key fact, how often the classifier picked the wrong category. Passing means staying under a failure rate you've agreed to accept.

The ground also shifts. When the model provider ships an update, the mines move even though nobody touched your code. Running the tally again is how you find out where they went.

AI is the new walker in the field, and it also buries a new kind of mine.

Mutation tests: the drill

All of these tests are markers, and markers can be wrong. Mutation testing checks them.

A mutation tool makes small deliberate changes to the code, like flipping > to >=, deleting a line or returning an empty value, and runs the tests against each change. Each one is a fake mine buried on purpose. If a test fails, the markers caught it. If every test still passes, the fake mine went unnoticed, and you've found a blind spot.

When agents write the tests too, someone has to check the markers they placed. A suite an agent wrote to make its own change pass can look thorough and catch very little. The drill measures that directly, without reading every test. Mutation runs used to be expensive to triage, because each surviving mutant needs someone to decide whether it matters. An agent can do most of that sorting now.

A marked field

Each of these tests lights a different part of the field. Unit tests show you one mine. Paths, treaties and full crossings show you routes. Records and sweeps show you what's buried. The convoy, the controlled detonation and the tally show you what only happens under weight, after a blast or some of the time. The drill tells you whether the markers work.

A field marked this way lets anyone cross it quickly, including a walker that arrived this morning and will forget everything tonight. That's why I think tests went up in value as agents started writing most of the code.

I wrote recently about attention being the scarce part when agents produce more.¹ Tests are one way to check an agent's crossing without spending your own attention walking behind it.

TL;DR

A codebase is a minefield: every feature buries mines, and every change means crossing the field. People who knew where the mines were used to carry the map in their heads. Coding agents cross the field many times a day and start each session close to zero, so the map has to live in the field as tests. Agents have also made many kinds of tests much cheaper to write.

Each kind of test marks a different part of the field. Unit tests defuse one mine on the bench. Characterization tests sweep a field nobody mapped. Integration tests mark safe paths. Contract tests keep the treaty at the border between services. End-to-end tests make the full crossing. Behavior-driven tests keep the minefield record. Stress tests find mines that only go off under load. Chaos engineering checks what happens after a blast. Evals measure mines that go off only some of the time. Mutation tests bury fake mines to check that the markers work.