AI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI
Loading slides…
Read the talk
AI Writes the Code. So What’s Left for Engineers?
Laurie Voss follows the bottleneck from code generation to review, operations, and the harder task of defining what people actually need—then asks how engineers should work and learn when agents do the typing.
From a talk by Laurie Voss
At a glance
Ideas worth remembering
Cheap code generation moves the delivery constraint into verification. More code produced is useful only insofar as it becomes software worth shipping.
A maintainer’s merge decision includes context beyond passing tests. Specs, rules, rubrics, and evals can make more of that judgment available to agents.
“Keep track of my orders” leaves the essential requirements unresolved. Product engineering discovers what that request means for a particular customer and defines a useful result.
Replacing junior implementation tasks also weakens an apprenticeship. Teams need deliberate training in customer conversations, specification, and judgment.
Measure shipped work, useful review findings, maintainer acceptance, and production behavior; concentrate human attention on consequential changes.
Eight times the code does not mean eight times the shipping
Writing plausible code has become dramatically cheaper. Establishing whether to trust it still takes work. That mismatch drives Laurie Voss’s forecast for engineering: accelerate implementation, and the work piles up at review; accelerate review, and another constraint takes over. Voss, head of developer relations at Arize AI and co-founder of npm Inc., extends the rise of the AI engineer into a question about every software engineer’s job: what remains when agents can handle more of the delivery process?
Voss cites two figures for developers using autonomous coding agents: 741% more code and 30% faster shipping. He does not give the team sizes, observation windows or definitions used to measure either outcome. The figures motivate a practical question for a team adopting agents: when a change is ready sooner, how long does it still wait for review and release?
Human review cannot simply expand to match that output. Voss cites an older Cisco study in which review accuracy fell sharply beyond about 450 lines per hour. His concrete warning is a 10,000-line agent-generated diff: inspecting it carefully becomes a substantial block of work, which he describes as about a week. Generating the diff quickly has transferred the effort to the person who must understand it. Larger output alone has not removed that effort.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Passing tests leaves out the maintainer’s decision
Automated review is the obvious response, and Voss describes it already taking a significant share of the work. He reports that GitHub Copilot performed more than one in five GitHub reviews in the preceding month and describes OpenAI’s internal workflow as permitting pull requests without mandatory human review. These examples establish the direction of adoption. They do not establish that machine review catches everything that matters.
A comparison of four AI reviewers exposes a useful distinction between finding bugs and providing dependable coverage. In Voss’s account, all four found real issues, but they found different issues and agreed on no individual bug. Each reviewer contributed something useful; any single reviewer left findings uncovered that another could detect. That is his reason for recommending multiple reviewers to teams taking automation seriously.
Skipping review altogether has produced painful reversals in the cases Voss recounts. Dexter Horthy abandoned advice to stop reading generated code after systems needed to be ripped out and replaced. Voss also describes Amazon responding to agent-related outages by requiring senior engineering sign-off, and an Anthropic postmortem about bugs that escaped both human and machine reviewers. The failures differ, but they all challenge the assumption that an agent has completed a change correctly on its first attempt.
Cognition’s benchmark asks maintainers whether they would merge a proposed patch. Voss reports 54% under that criterion and contrasts it with 87% for passing tests. He gives no common task count for the two percentages. His separate 80% figure describes a model on a standard coding benchmark. A maintainer also considers style, side effects, external systems and project history. A patch can pass the available tests and still violate one of those expectations.
That missing information is Voss’s first version of “what good means.” A change can satisfy an executable test suite and still be wrong for the system it must join. Improving review therefore requires more than a stronger bug detector: it requires making the reasons for accepting or rejecting a change available to the reviewer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
After review, the constraint moves again
Eliyahu Goldratt’s theory of constraints supplies the model for what happens next. In Voss’s account of The Goal, improving a part of a system that is not its bottleneck does little to improve total throughput. Once the bottleneck is relieved, another part becomes limiting. Faster code generation has brought review into focus; successful automated review would move that pressure elsewhere.
The proposed next constraints concern running the software: deploying it, scaling it, and making it secure and private at scale. Voss explicitly makes a leap here. He assumes progressively better agents will eventually handle these tasks because humans have accumulated substantial knowledge about them. This is a forecast with thin evidence for some stages, not an account of a completed autonomous delivery system. He expects progress on review over the following year, while placing operations, scaling, and security on a five-to-ten-year horizon.
A second assumption determines whether cheaper software supports more work for humans: demand must remain effectively unlimited. The everyday examples are bad websites at a hairdresser, a bank, or a government department. Better versions are possible, but the people who need them cannot always afford the engineering effort, and capable developers are occupied elsewhere. If production gets cheap enough, previously uneconomic improvements become possible. Voss’s prediction of a larger profession depends on that unmet demand turning into more software.
His resulting ordering separates implementation work from decisions about the product. Writing code has already become cheap in his framing; review and maintenance are improving; deployment and scaling come later. The work he expects to remain with people is deciding what to build, defining success, and making the result delightful. The rest of the talk explains why those decisions might resist the same economies of scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
“Keep track of my orders” is not a specification
A customer says, “I need to keep track of my orders.” Thousands of products fit that sentence. Choosing or building the right one requires discovering what the sentence leaves out. A bakery, an auto parts store, and a large manufacturing business all track orders, but they need different software. The initial request names a category of activity; it does not describe how that activity works in this particular business.
The next step is conversation: what does an order mean here, what must the system do, and how does the business operate? The necessary knowledge belongs to the person running that business. A model’s general knowledge of order management cannot supply facts that this customer has never expressed. Voss’s deliberately bodily description—“flapping your meat parts”—lands on a practical task: talk to the customer until the request becomes specific enough to build.
That discovery work transfers poorly between products. A calendar app and a scheduling app address closely related needs, yet their requirements and definitions of success differ. Adding configuration options does not guarantee a fit either: Voss points at Jira as the familiar experience of software that can be configured endlessly and still fails to do what someone needs. Cheap implementation makes this specificity more consequential. Producing another system is easy only after someone has described the right system.
Fred Brooks’s distinction helps name the remaining difficulty:
- Accidental complexity: Difficulty introduced by tools, languages, syntax, and the mechanics of implementation.
- Essential complexity: Difficulty in the problem itself—understanding what must happen and what a satisfactory solution means.
Voss uses the modest shipping gain beside the large code-output gain to illustrate why reducing implementation effort does not remove the whole problem.
Sean Grove’s description of code as a “lossy projection of the specification” pushes the value upstream. The specification contains intent that may not survive fully in the implementation. Whoever can express that intent precisely—a product manager, engineer, lawmaker, or marketer—can direct what gets built. In Voss’s forecast, writing down what good means becomes a central engineering skill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Written judgment becomes something agents can use
There is a tension in treating judgment as the remaining human advantage: once people express it clearly enough, automation can use it. Cognition’s mergeability benchmark attempts to capture more of the maintainer’s acceptance criteria so models can train against them. Voss also describes Cursor’s Bugbot turning team thumbs-down feedback into rules, reporting roughly 44,000 rules. In both examples, knowledge moves from an individual’s judgment into a reusable representation.
This is why Voss expects operations and security to become more automatable: their requirements can increasingly be written down, checked, and used to improve agents. His proposed human role keeps producing those definitions. The baker’s unstated needs must first be discovered; after they become a specification, agents have something concrete to implement. The scarce work lies in getting from a person’s situation to a useful definition of success.
That shift also changes the division between product management and engineering. Voss cites Andrew Ng describing a team’s proposal for twice as many product managers as engineers, then favoring a closer combination of the jobs: an engineer who understands users can decide what to build and build it directly. The historical comparison is the systems analyst, whose job was to talk to the business and turn its problem into requirements. Product engineering recombines that discovery work with delivery.
Voss calls the combined role the product engineer: someone who finds out what people need, defines what good looks like, and ships it with agents doing the typing. Gergely Orosz’s earlier description of the product-minded engineer supplies a recognizable predecessor. Voss’s stronger claim is that product judgment moves from a desirable trait, often associated with senior engineers, to an essential skill for engineers generally.
Forward deployed engineers provide his market example of a related role. He reports postings rising 800% over nine months of 2025 and average compensation of $240,000, interpreting that pay as recognition of senior engineering judgment. The labels remain unsettled: product engineer is his chosen name for a cluster of responsibilities, rather than a standardized title shared across employers.
This forecast makes software development a broader activity than programming. It also carries an objection Voss raises himself: people discussing AI often conclude that everything can be automated except the part they personally enjoy. Discovering the problem is his favorite part of software work. His bet that it will remain central deserves that qualification; it is a reasoned forecast about missing customer knowledge, not proof that agents can never participate in discovery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The old training task is becoming the agent’s task
The traditional junior assignment was a well-specified ticket: implement this behavior, then have a senior engineer explain what needs correcting. Repeated implementation and review helped beginners acquire judgment. Agents now receive that same kind of assignment. If employers stop hiring people for those tickets, they also remove a route by which people learned to become the senior engineers they still need.
Voss says employment fell for software developers aged 22–25 and rose for those aged 41–49. His companion analysis charts US software developer employment in ADP payroll data, indexed to October 2022 = 1.0 for each age group. The younger group falls below its baseline while the older group rises above it. Employment counts alone cannot tell us how hiring changed or how much AI contributed. Voss’s concern is how new engineers will acquire the judgment employers value in experienced ones.
Existing product training programs suggest a possible shape for the replacement, but Voss argues that their scale is inadequate. He describes Google’s Associate Product Manager program as producing about 50 people a year, estimates about 1,000 annually across comparable programs, and calls for at least 10,000. The demand figure is his estimate of what the transition requires. The mismatch he wants leaders to address is deliberate training in customer understanding versus the much larger pipeline that previously learned judgment through programming work.
His proposed exercise returns to the bakery: sit with a baker for a week and come back with a specification. Data structures and React teach useful implementation skills, but that assignment trains someone to discover and express a business problem. Voss points to IBM redesigning entry-level roles around customer conversations and specifications as an example of the direction. Engineering leaders should build such learning into real jobs, rather than wait for product judgment to appear as an accidental byproduct of the old apprenticeship.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Loving the craft does not mean loving the replacement job
Some programmers chose the profession for a clean abstraction, a stubborn bug finally yielding, or the pleasure of making a machine obey. Interviewing a baker is a different attraction. A forecast of more software and more developers does not compensate someone whose preferred work becomes a smaller part of the paid job. Voss treats that as a real loss and offers no easy consolation.
For some people, though, the satisfying moment may have come just before the typing: when a vague problem resolved into a repeatable system in their head. That activity remains central to his product engineer. Programming as a craft can also survive independently of its market position, as woodworking survived furniture factories. His analogy separates the desire to keep making things from the question of which tasks employers will pay large salaries to perform.
The transition could still be ugly. Tools are arriving after the load they must handle, and replacement training is arriving after the junior ladder has weakened. Voss looks to compilers, frameworks, and open source as earlier changes that removed large amounts of typing while expanding what software developers could accomplish. His ten-year prediction is a larger profession with comparable pay, organized mostly around product engineering. It remains conditional on the automation and demand assumptions that opened the argument.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make quality explicit, then measure what ships
The immediate recommendations concern where to spend human attention and how to make automated work verifiable:
- Identify code with a high blast radius. Write down which changes can cause widespread harm and concentrate human review there as automated review improves. This is the direction Voss recommends preparing for; his earlier failure examples explain why it cannot rest on assuming current agents always get changes right.
- Delegate work that is cheap to verify. Ask how to determine automatically whether an output is good, then give that check to the agent. A clear success test can support repeated automated attempts without requiring a person to inspect each one from scratch.
- Express expert judgment in specs, rubrics, and evals. Turn criteria that live in experienced people’s heads into artifacts that agents can use to build and assess the result.
The measurements should follow the outcome rather than celebrate generation. Track how much software actually ships, how often review comments lead to action, and the gap between passing tests and earning a maintainer’s approval. Each answers a different question: is work getting delivered, are review findings useful enough to change the code, and are automated checks capturing the acceptance criteria that matter? A 741% increase in code output cannot answer those questions.
“The last reviewer is production.” Deployed behavior encounters conditions that earlier checks missed. Voss describes another model grading that behavior as an eval, extending quality assessment beyond the pull request. This makes the definition of good operational: it must be useful for judging what the system actually does, as well as what its tests say it can do.
The product engineer’s work is to turn a sense of the problem into a specification precise enough for agents to build from, while keeping the result connected to the people who need it. Finding those needs, writing down what good means, and making something people like are the responsibilities Voss expects to spread across engineering. His closing line gives the forecast its bluntest form: “We are all product engineers now.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Voss’s companion analysis develops the age-based employment comparison, discusses alternative explanations, and links its labor-market sources and data.
- LLM Evaluation BasicsReference
An introductory course on evaluating language-model applications.
Voss’s workshop slides on evaluations for agentic applications.
Related talks
- The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Voss refers listeners to his earlier talk for a closer look at the code-review bottleneck.
- Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Full transcript
Loading transcript…