AI Engineer Code 2025
Leadership in AI-Assisted Engineering
Loading slides…
Read the talk
Leadership in AI-Assisted Engineering
AI adoption can rise while delivery gets worse. Effective engineering leadership connects usage to quality, protects learning time, and targets the constraints that actually limit throughput.
From a talk by Justin Reock
Before you start: Familiarity with pull requests, software delivery, and AI coding assistants will help you follow the examples.
Why can AI feel faster while making work slower?
What should an engineering leader conclude when one AI productivity study looks positive and another looks negative? Justin Reock opens with that problem, offering his AI strategy playbook for senior executives as the longer reference behind the discussion.
Reock contrasts a Google headline reporting 10% higher engineer productivity with METR’s early-2025 developer study, which measured 19% longer task completion times with AI access. The Google figure’s measurement conditions are not established here. METR studied 16 experienced developers completing 246 tasks in familiar, mature repositories, using early-2025 tools, primarily Cursor Pro with Claude 3.5 Sonnet and Claude 3.7 Sonnet. Longer completion time is the measured result; a productivity percentage is not interchangeable with a time percentage.
The more unsettling result is the gap between experience and measurement. Participants collectively estimated that AI had saved them time even after the experiment showed a slowdown. Reock describes this more strongly as every engineer feeling more productive, and suggests that an induced sense of flow might explain the mismatch. He also questions the study’s design without specifying the flaws. The practical problem remains: feeling productive does not establish that a workflow is faster.
Industry averages offer another incomplete view. DORA’s generative AI research associates a 25% increase in AI adoption with 7.5% higher documentation quality and about 3.4% higher code quality. These are estimated associations, not guaranteed causal gains, and the report separately finds worse delivery throughput and stability. Better process artifacts do not by themselves establish better delivery.
DX’s aggregate data initially looks similarly reassuring. Reock reports about a 2.6% increase in change confidence, a positive average movement in perceived code maintainability, and a 0.11% reduction in change failure rate. Change confidence measures the share of respondents who feel confident about the changes they put into production; maintainability is also a qualitative measure here.
Breaking those averages into company results changes the interpretation. Reock shows companies with roughly 20% increases in change confidence alongside companies with roughly 20% decreases. Maintainability and change failure rate also vary sharply. His failure-rate example compares a 4% baseline with a two-percentage-point increase: moving from 4% to 6% is a 50% relative increase in the failure rate. He describes that as shipping more defects, but the rate alone does not establish a defect count. The reported DX changes otherwise do not consistently distinguish percentages from percentage points. The relevant question is where your organization sits in the distribution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A usage mandate can succeed without improving anything
Organizations seeing positive outcomes differ from those struggling with adoption or quality. A mandate for universal AI adoption is easy to satisfy superficially: an engineer can use an assistant to update README.md every morning and count as compliant. The utilization target moves; the delivery constraint may remain untouched. Turning on a tool without education, support, or a measurement strategy creates the same disconnect.
Reock uses DORA’s Bayesian posterior chart to shift attention toward the conditions surrounding adoption. The distributions represent uncertainty about the effects of different initiatives. In his reading, probability mass on the yellow, right-hand side favors the desired direction, while a narrow peak indicates a more concentrated estimate. Clear AI policies and time to learn stand out as useful supports. Giving people documents is different from giving them protected time to experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the purpose of adoption explicit
Start with the whole software development lifecycle, not just code completion. For many organizations, writing code is not the bottleneck. Faster code generation therefore has a limited effect unless the surrounding workflow can absorb and deliver that work. Access barriers also require design: Reock points to Bedrock and Fireworks AI as infrastructure options to consider when data exfiltration concerns prevent experimentation. The hosting arrangement still needs to meet the organization’s requirements.
Explain why metrics are being collected, what the organization intends to improve, and which early successes are worth sharing. Reock frames AI as a way to augment engineers and increase business throughput. Compliance, trust, and employee development belong in the same conversation: these tools introduce skills that leaders can help employees acquire. His warning is that competition may come from people who use AI well, even where the technology cannot replace an engineer.
Psychological safety makes that learning possible. Reock invokes Google’s Project Aristotle, begun in 2012, describing how psychological safety emerged as more consequential than the expected recipe of talented individuals, experienced managers, and abundant resources. People need to be able to admit uncertainty and discuss failures without treating every experiment as a threat to their standing.
Capability claims should support that conversation honestly. Reock cites a SWE-bench result of roughly one-third of tasks completed without human intervention, emphasizing the unresolved remainder. No system, benchmark split, or evaluation configuration is identified for that figure, so it describes the example in this talk rather than a current capability ceiling. Communicate the intended use proactively: better developer experience and greater throughput through augmentation, before employees have to infer the purpose from tool rollouts or dashboards.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure speed and quality through different kinds of evidence
Speed and quality must move together. PR throughput and velocity matter, but generating work that creates technical debt simply moves the bottleneck downstream. Pair those speed measures with change failure rate, perceived quality, change confidence, and maintainability.
No single data source explains all of those outcomes. Reock distinguishes three complementary approaches:
| Evidence | Example | What to watch |
|---|---|---|
| Telemetry | Suggestions and acceptances | An IDE event is not retained code |
| Experience sampling | AI-use questions on a PR | Capture the experience near the work |
| Surveys | Developer experience across teams | Participation and question design matter |
Acceptance telemetry illustrates the limitation. In the workflow Reock describes, the API records acceptance when an engineer clicks the IDE’s accept control. It does not establish whether the engineer later rewrote every suggested line. A PR form can add context by asking whether AI was used and whether using it was enjoyable. Surveys then provide a broader view of the system in which those individual experiences occur.
Reock describes effective surveys as achieving participation above 90% and asking about developer experience as a systems problem, not a people problem. He reinforces that stance with an attribution to W. Edwards Deming that places 90–95% of organizational productivity output in the system rather than the worker. The numerical attribution is not needed to operationalize the principle: use the results to identify friction in the work environment rather than to rank individual developers.
Utilization tells you what is happening with the technology; established developer experience and delivery measures tell you whether the initiative is working. Reock points to Microsoft’s adoption metrics and its work on a bad developer day, which combines telemetry about difficult working conditions. He describes Dropbox as pairing daily and weekly active usage with quality measures such as change failure rate, and Booking as following a similar approach.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move from utilization to impact and cost
The DX AI Measurement Framework organizes this evidence into utilization, impact, and cost. Reock describes it as inspired by DORA, SPACE, and DevEx, as is DX’s Core 4 metric set, and calls it first to market. Its useful structure is a progression in the questions a team can answer.
- Establish utilization. Determine who uses the tools, what share of PRs are AI-assisted, and how many tasks are assigned to agents. Experience sampling can supply information that product telemetry misses.
- Connect usage to impact. Examine how utilization relates to velocity and quality. Increased activity becomes meaningful when it connects to delivery outcomes.
- Account for cost. Include the expense of achieving those outcomes. Reock compares the challenge with cloud-cost optimization, which remained an active industry long after cloud adoption began. He also recounts anecdotal reports of $2,000 a day in token spending.
This progression turns an adoption dashboard into an evaluation of what the organization receives for its investment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose variation to fit the task
Shared instructions govern what the assistant should do; temperature influences variation when sampling its output. At each generation step, a model has candidate next tokens with different probabilities. Temperature changes the sampling distribution, affecting how strongly generation favors the more likely candidates. Reock presents a zero-to-one range and advises avoiding the endpoints. Treat that as guidance for his demonstration: supported ranges and sampling behavior depend on the model and API, and low temperature alone does not guarantee identical results.
At temperature 0.0001, Reock’s repeated task produces character-for-character identical output. At temperature 0.9, his gradient-generation prompt produces two substantially different approaches: CSS and JavaScript canvas. He calls both relatively valid, but only one supplies the JavaScript method he requested. That distinction matters: producing a plausible solution and satisfying the requested interface are separate requirements.
A small JavaScript canvas function makes the requested kind of output concrete:
javascript
function drawGradient(canvas, startColor, endColor) {
const context = canvas.getContext('2d');
if (!context) throw new Error('A 2D canvas context is required');
const gradient = context.createLinearGradient(
0, 0, canvas.width, 0
);
gradient.addColorStop(0, startColor);
gradient.addColorStop(1, endColor);
context.fillStyle = gradient;
context.fillRect(0, 0, canvas.width, canvas.height);
}
This illustrates the canvas approach visible in the comparison. A CSS gradient can address the visual goal while failing the explicit request for a JavaScript method. Decide whether the task benefits from exploring alternatives or from greater consistency, then evaluate both correctness and instruction adherence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach useful tasks and make experimentation possible
Reock suggests Docker Model Runner, Ollama, and LM Studio as places to experiment with model behavior. Access to tools needs to come with education and adequate learning time. To identify useful material, DX sampled developers who reported saving at least an hour per week and asked them to rank their five most valuable use cases. The resulting Prompting guide for AI-assisted engineering combines prompting and code examples to help engineers develop useful habits. These are developer-ranked opportunities based on perceived savings.
Reock says the guide became required reading in some engineering groups, but reiterates that distributing it is insufficient without time to practice. In the sampled developers’ ranking, stack trace analysis came first. That is an interpretive task: helping an engineer understand existing evidence rather than generating more code. It broadens the search for valuable uses beyond completion and code synthesis.
Unblocking usage also means examining the actual compliance constraints. Consider self-hosted or private models, involve compliance partners from the beginning, and test assumptions about what is prohibited. A restriction that engineers believe exists may differ from the organization’s actual policy. Resolving that uncertainty can open useful workflows without bypassing governance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the constraint beyond code generation
Eli Goldratt’s Theory of Constraints supplies the final test for choosing where to apply AI: does the intervention relieve the bottleneck? Saving effort elsewhere may not increase total throughput. Reock describes data spanning almost 140,000 engineers in which annualized AI time savings are eclipsed by context switching, interruptions, and meeting-heavy days. Faster individual tasks cannot compensate for every source of organizational friction.
Legacy reverse engineering: Morgan Stanley’s DevGen AI examines COBOL, mainframe Natural, and Perl to produce specifications that developers can use for modernization. The intervention removes some of the work of reconstructing what an existing system does before replacing it. Reock reports about 300,000 hours saved annually at Morgan Stanley, citing public press coverage.
Onboarding: Reock reports that Zapier’s bots and agents help make new engineers effective in two weeks, compared with his cited benchmarks of about a month for strong onboarding and roughly 90 days in the middle. He describes Zapier’s response as faster hiring: if each incoming engineer becomes useful sooner, increasing the team can create more value. The objective is expanded organizational capacity rather than preserving existing output with fewer people.
Incident response: Reock describes Spotify gathering context when incidents are detected, including runbook steps and documentation, and pushing that material directly into SRE channels. The assistant reduces the time responders spend establishing what happened and finding the relevant response procedure. His account describes faster response, although he says MTTR increased; that wording does not support a clear claim about the direction of the MTTR metric. The concrete mechanism is the earlier delivery of relevant operational context.
Put these ideas into an operating cycle: distribute the guide as a reference for integrating AI into existing workflows, choose a method for evaluating impact, and track adoption alongside delivery outcomes. Use that relationship to refine best practices and select the next use cases. Reock closes by showing the guide access again, returning to the leadership task behind the rollout: keep checking whether increased use is producing the outcomes the organization intended.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Justin Reock's leadership guide to AI enablement, trust, measurement, and adoption across software development. Includes a public summary and a download form.
A practical guide covering prompting techniques, developer-ranked use cases, and leadership support. The public page summarizes the guide and provides a download form.
A randomized trial of experienced developers completing tasks in familiar open-source repositories, comparing completion time with and without early-2025 AI tools.
Research on associations between AI adoption, software quality, developer experience, and delivery performance, including uncertainty estimates.
Full transcript
Loading transcript…