Software development has traditionally been a targeted activity. We decide what we want to build, humans translate that intention into source code, and other humans inspect the result. AI coding agents are changing the economics of that process.
An agent can create variants, run tests, repair failures and repeat; a hundred agents can explore a hundred approaches in parallel. The cost of producing a candidate change is falling faster than the cost of understanding one.
That moves the bottleneck. Researchers at JetBrains and Lund University now describe code review itself as the emerging constraint: agents can produce large multi-file changes quickly, while human reviewers still have to decide what deserves trust. Their framing is “trust calibration” rather than simple diff inspection. [1]
We think this is a transitional state. Humans will not review every patch produced by ever more numerous and capable agents. Review will itself be automated. At that point the central mechanism is no longer generation but selection.
From software construction to variation and selection
In the narrow technical sense, none of this is metaphorical. The field of genetic improvement of software has existed for decades: generate software variants, evaluate them, and retain changes that improve a target property. A major survey found evolutionary algorithms in 96% of the empirical genetic-improvement studies it identified. [2]
More recently, Google DeepMind described AlphaEvolve explicitly as an “evolutionary coding agent.” Language models propose programs, automated evaluators score them, and an evolutionary algorithm determines which programs become the basis for future generations. Some of its results have already been used in Google infrastructure. [3]
But our argument is broader than genetic algorithms. Evolution requires variation, differential selection and some mechanism through which successful characteristics persist. Once software agents make variation cheap and automated evaluators, markets, users and organizations decide which variants survive, software development acquires an evolutionary layer even when nobody explicitly calls genetic_algorithm().
Fitness is not correctness
The word fittest is easy to misunderstand. It does not mean “best engineered,” “most elegant,” or even “correct.” Fitness means surviving the selection pressures of the environment.
For commercial software those pressures can include revenue, adoption, demo quality, procurement, feature velocity, engagement, organizational politics, support cost and time to market. Correctness is only one pressure among many, and in some environments it is weak or slow to register.
A personal experience
Years ago, I was asked by a large company to convert an Excel/VB-based engineering program to C++. The existing program had passed through several developers and many iterations. Watching it run was strange: tables appeared and disappeared, calculations iterated for a while, and eventually a polished-looking result emerged.
The code was hard to follow. To reconstruct what it did, I went back to a book describing the mathematics behind the calculation. One detail stood out: a large iterative procedure could be reduced to a single algebraic equation. Worse, the complexity of the iterative implementation had introduced errors into the calculation itself.
The lesson I took is not that somebody had written bad VB. It is that the software had survived. People used it. Several developers maintained it. The output looked credible. Organizationally, it was fit enough to stay alive even though part of its implementation was unnecessarily complex and mathematically wrong.
That was expensive, human-written software.
Now consider the same selection environment when candidates can be produced by the thousand.
Markets are part of the fitness function
A useful mental model is not a single objective called “software quality,” but a changing fitness landscape:
fitness = f(
revenue,
adoption,
appearance,
speed,
correctness,
security,
maintainability,
regulation,
organizational incentives,
...
)Nobody chooses all of these weights centrally. They emerge from customers, markets, institutions, attackers, developers and measurement systems.
A product that looks convincing can therefore outperform a product that is correct but difficult to sell. A system that maximizes engagement can outcompete one that protects attention. An internal tool can survive because it makes a dashboard green even if its model of reality is poor.
The feedback rates also differ. Revenue can be measured this quarter. A subtle numerical error may take years to produce an obvious failure. Maintainability may not matter until the original team is gone. Privacy damage can remain invisible until data is abused. Evolution responds most strongly to pressures that are immediate and observable.
Powerful optimizers exploit bad proxies
This connects directly to specification gaming. DeepMind describes specification gaming as behavior that satisfies the literal objective without achieving the intended outcome, and notes a counterintuitive consequence: as optimizers become more capable, correctly specifying the objective can become more important because the system becomes better at finding loopholes. [4]
The same principle applies outside reinforcement learning. If we select software primarily for:
- passing the test suite,
- winning a benchmark,
- raising conversion,
- closing a ticket,
- reducing latency,
- or looking correct to a reviewer,
then increasingly capable agents will become better at satisfying those criteria. That does not guarantee conceptual simplicity, physical correctness, maintainability, privacy, security or alignment with the user's real interests.
Recent work on reward hacking in language-model agents demonstrates the same structural problem in controlled environments: optimizing observable reward can widen the gap between the measured objective and hidden safety objectives. [5]
Software can evolve to become worse
Evolutionary software, then, should not be assumed to mean improving software.
Evolution produces effective but messy solutions: local optima, historical accidents, redundancy, parasitism and complicated workarounds. Software already looks like this because organizations themselves are selection environments. Automated generation can accelerate the process.
One risk stands out: software may become better at appearing trustworthy faster than it becomes trustworthy.
A human reviewer has limited time. A generated system can cheaply provide a polished interface, plausible charts, confident explanations, hundreds of passing tests, benchmark results and a long architecture document. All of those things are useful evidence. None of them independently prove that the underlying model is correct.
The old Excel/VB program eventually produced a plausible-looking answer. Its AI-generated descendant may produce the answer, the test suite, the explanation for the answer and the review summary explaining why the answer should be trusted.
The engineering problem moves upward
This does not make engineering irrelevant. It changes where engineering effort is most valuable.
If variation becomes abundant, engineers increasingly design the selection environment: specifications, invariants, test strategies, independent evaluators, physical models, formal constraints, sandboxes, provenance, staged deployment and feedback mechanisms.
The question changes from:
How do we implement this solution?
toward:
What pressures must exist so that useful, correct and safe solutions are more likely to survive than convincing but harmful ones?
That may prove the harder problem.
Software engineering becomes ecosystem engineering
We therefore expect a gradual transition rather than a clean break. Humans will still write code. Traditional design will remain important. Many systems will not use explicit evolutionary optimization. But the large-scale economics change once agents can continuously generate alternatives and other agents increasingly participate in evaluating them.
The unit we engineer may shift from an individual program toward an ecosystem of generators, evaluators, constraints and feedback loops.
And once that happens, security changes too. Malicious software is subject to exactly the same pressures. A malicious agent that is detected disappears; one that avoids detection persists. Defenses improve; attacks adapt. The security problem begins to look less like maintaining a static wall and more like maintaining a healthy population under continuous evolutionary pressure.
That is the subject of the second article in this series: An Immune Ecosystem for the Agentic World.