How AI Writes Code and Where It Breaks
An AI coding system turns a natural-language request into source code, then checks that code against the compiler, the test suite and the runtime output. The underlying model predicts one token at a time; the loop wrapped around that prediction is what makes the tool useful — write to a file, run it, read the error, patch, run again. The 2021 Codex paper measured both halves of this. On HumanEval the model solved 28.8% of problems on a single attempt, and 70.2% when 100 samples were drawn per problem and any passing sample counted, a spread of 41.4 points between one try and many. Agent tooling built since then spends its budget on the retry side of that spread, which is why one task can consume dozens of model calls.
Why generated code is judged differently from generated prose
An approximately correct sentence still does its job. An approximately correct program does not compile. Python's lexical analysis reference puts indentation inside the grammar itself: the parser emits INDENT and DEDENT tokens off a stack, and a tab advances the column to the next multiple of 8. Write three spaces into a file that uses four and the parser raises an IndentationError at load time. TypeScript enforces its own version of the same discipline. The strict flag arrived in version 2.3 as a single switch over a family of checks, and the tsconfig reference now lists nine of them under it, from noImplicitAny to strictNullChecks.
The compiler returns 0 or 1 and the test runner hands back something countable — 408 of 412 tests passing — so nobody has to ask a person whether the output is correct. That is what makes machine-written code tractable at all: the model can see and repair its own mistake because the feedback costs almost nothing. The channel is narrow, though. The compiler only checks form. A function that applies the wrong VAT rate compiles cleanly and passes every type check; the error surfaces in accounting, weeks later, on a customer's invoice.
Context windows are smaller than production repositories
Take a mid-sized enterprise repository at 300,000 lines with an average of 35 characters per line: 10.5 million characters, which at 3.5 characters per token works out to about 3 million tokens. A 200,000-token context window holds 6.7% of that. A 1-million-token window holds a third. At that scale you cannot tell the model to read the whole project first, so coding tools work selectively instead — filename and symbol lookup, grep-style text scans, vector search and import-chain following narrow 3 million tokens down to a slice of 20 to 40 files, and the model sees only the slice.
When the search layer picks the wrong slice, you get correct code nobody needed. Your repository has carried an HttpClient wrapper for two years, with retry and timeout logic already worked out. The model never sees that file, so it writes a second client from scratch in 80 lines. It compiles, the tests go green, the reviewer approves it in five minutes. Six months later the team is running two network layers with different timeout values and nobody remembers which one talks to which service. In the systems we build the pattern splits by size: under 5,000 lines in a single service, duplicate helpers almost never show up; past 50,000 lines, the odds of the model finding and reusing the existing helper drop off sharply.
Inside the agent loop: write, run, read the error, patch
An agent runs the test, reads the AssertionError line, opens the file it points at, changes one condition and runs the test again. Most coding tools implement that flow on top of tool use. Anthropic's tool use documentation describes the cycle: the model emits a tool call, the runtime executes it, the result returns to the conversation as a new message, and the model picks its next step from there. In a coding context the tools are read file, write file, run a shell command, run tests. The loop repeats until the tests pass or the step budget runs out.
On the Multilingual leaderboard at swebench.com, built on a 300-instance set, one agent reaches a 72.7% resolve rate while spending an average of 52.5 tool calls per instance; a different configuration on the same board hits 72.0% with 28.9 calls. That is near-identical accuracy at 1.8 times fewer steps, and the gap shows up as wall-clock time, then as cost. Call counts are what the board publishes; elapsed time depends on how fast your own tools run, and a test suite that takes 4 seconds and one that takes 4 minutes produce the same call count.
The loop does not pay off on small tasks. In our own workflow, handing a one-line configuration change to an agent costs 8 to 12 tool calls once you count finding the file, reading it, writing it and running the test; a person opens the file once and saves it. Our dividing line: work that touches more than one file and needs at least one test run goes to the agent, and anything below that stays manual.
Closing the loop requires something that answers back. In a module with no tests, the model announces a fix and has nothing to show for it. Logic errors that never print anything — a rounding drift noticed three weeks into production, a queue message processed twice — sit outside what the loop can observe at all. The compiler stays quiet, the test suite stays quiet, and the agent moves on to the next task with the bug still sitting in the file.
What SWE-bench scores actually measure
SWE-bench is where most published comparisons of coding models land. The original paper assembled 2,294 real GitHub issues from 12 popular Python repositories, and the best model at the time resolved 1.96% of them. On the same project's leaderboard today, top submissions on the 500-instance Verified set sit at 79.2%, roughly a 40-fold increase in under three years. Verified is a human-screened subset of those 2,294, filtered so that every issue has a well-specified fix and a fair test, which makes it the friendlier of the two measurements.
The best Multimodal result on the same board stands at 35.98% and the Lite set tops out at 60.33%, so issues that need visual evidence resolve at roughly half the Verified rate. Behind those figures sits a structural point: every problem in the benchmark is closed-ended. There is an error message, the full repository sits on disk, and a test suite defines correctness in advance. A 79.2% score describes that setting and does not transfer to enterprise work, which arrives as a sentence like "the payment provider changed its contract, update the refund flow accordingly" — no stack trace, no test that defines success, and usually two people who disagree about what the new behavior should be.
Where AI code generation breaks: knowledge that exists only in your company
Four clusters account for most of what AI code generation gets wrong, and they share one cause: the weak spot is knowledge that never appeared in training data and exists only inside your organization. Thinly documented internal libraries are the clearest case. Hand a model an internal library that seven people developed over four years behind a 12-line README, and the same model that uses a popular open-source package flawlessly, having seen it tens of thousands of times in training, will invent a method name that does not exist here and then build a plausible call chain around the name it invented.
Business rules fail for a different reason: they are not in the code at all. Whether a discount applies before or after VAT, whether the return window runs 14 days or 30, lives in a contract, in regulation, or in five years of institutional memory. The model writes the pattern that is common across the industry, and that pattern disagrees with your contract in precisely the places that cost money.
Trial and error stays cheap as long as operations can be undone. Schema migrations, bulk updates, payment calls and outbound email have no undo, so "run it and see" turns into direct damage across those four classes of operation. An agent that drops a column to make a test pass has done exactly what it was asked, and the rollback is a restore from backup.
Security failures come from copying, because a model reproduces the pattern it saw in example code. The hardcoded-credential flaw that MITRE catalogs as CWE-798 enters a codebase through that reflex: an API key becomes a string literal in a source file, one commit plants it in repository history, and cleaning up means rotating the key, since editing the file leaves the old value in every clone.
The human job moves from typing code to specification and verification
When a model writes most of the lines, the bottleneck moves to deciding whether the produced code does the right job, and that decision rests on two inputs. The first is the specification: which output is expected for which input, which boundary values are valid, which condition raises an error. In our own workflow, replacing a three-line prompt with a 20-line acceptance criterion takes a typical task from three or four correction rounds down to one or two. The second is verification, meaning the test itself. When a person writes the test, the model cannot produce both the code and the criterion that validates it out of one wrong assumption.
A 400-line changeset stays readable when it arrives as 10 separate steps of 40 lines; as one block, approval degrades into a mechanical click. Our own sequence runs a task through five stages — specification, first implementation, test, correction, review — with a person signing off at the last one, on a diff small enough to read line by line.
Frequently Asked Questions
Will AI coding tools replace software developers?
No, and the measurements do not point that way: the top result on the SWE-bench Verified set is 79.2%, and all 500 problems in that set arrive with an error message and a ready test suite, which is the easiest shape a coding task can take. In real work the specification, the business rules and the control of irreversible operations stay with people. The mix changes: hours move from typing lines toward writing specifications and checking output.
Why does an AI coding agent write irrelevant code in a large codebase?
A 300,000-line repository runs to about 3 million tokens, and 6.7% of that fits inside a 200,000-token context window. The model sees a small slice of the repository on each request. When an existing helper class falls outside the slice, the model writes a functionally correct second copy of code the project already holds. Two fixes work together: strengthen the search layer that selects files for the model, and name the relevant file paths directly in the prompt.
Which checks are mandatory before shipping generated code?
Four gates, at minimum: the compiler and type checker with the nine checks under TypeScript's strict switched on; unit tests written by a person; a secret scan covering CWE-798 style embedded keys; and human approval on every change that touches a database schema or a payment flow. Moving in 40-to-60-line pieces keeps those four gates workable, since a reviewer handed 400 lines at once stops reading.
