AI Engineer World's Fair 2026

From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization

May Walter· Co-Founder and CTO, Hud22:46

Loading slides…

Read the talk

From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization

A reliable optimization agent needs more than plausible code changes: it needs production context, a repeatable investigation method, runtime verification, and a reason to deserve a human’s attention.

From a talk by May Walter

Before you start: Familiarity with pull requests, production telemetry, SQL, and coding agents will help you follow the workflow.

How long will fixing the slow page take?

Jenny says a page is slow. Engineering agrees it could probably be optimized, but someone would have to investigate, so the work waits. A few weeks later, the page is too slow to ignore. Now Jenny needs an estimate: somewhere between an hour and a week. The investigation has to happen before anyone can narrow that range.

A perf-backlog chat shows Jenny asking how long optimization will take, Ben answering somewhere between an hour and a week, and Jenny asking how to put that on a roadmap.Open full source frame
The investigation trap: estimating a fix before investigating it.

Who can investigate? Only Dave—the one person who still understands code whose other authors left a decade ago. The bottleneck is not necessarily implementing an optimization. It is discovering which optimization is needed. Teams repeatedly find performance and stability improvements when they look, yet the uncertainty of that search makes proactive work difficult to justify.

May Walter, co-founder and CTO of Hud, approaches this as a runtime-context problem. Hud supplies coding agents with function-level production information and deeper forensic evidence about important events. The continuous optimization workflow built on that data aims to make investigation happen routinely, before another slow page becomes an emergency.

0:010:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Plausible code still needs verification

An agent writes a pull request. Another agent reviews it and approves. Both outputs look reasonable, but the engineer still wants to verify the change before merging it into production. Agreement between agents has not supplied the missing evidence that the change works.

Walter connects that hesitation to a broader gap between individual effectiveness and reliable team delivery. She describes stronger individual productivity, limited throughput gains, and worsening stability, attributing the comparison to “DORA metrics for 2026.” That year label needs care: DORA’s research chronology distinguishes its March 2025 generative-AI study from its September 2025 annual report, and its March 2026 discussion describes the latter as finding increased throughput alongside increased instability. The useful engineering question is how to turn faster coding into dependable delivery. Walter’s proposed next step is to use AI to accelerate repair as well as development.

1:502:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Make investigation a recurring job

The usual performance backlog behaves like a leaky bucket: ignore an issue, let it degrade, repair it in emergency mode, then return to ignoring the backlog. Investigation perpetuates that cycle because its cost arrives before its value is known. An engineer might spend an hour or weeks simply determining what could be improved.

Automating discovery changes what can be prioritized. A weekly process can inspect real production behavior, identify opportunities, and score the changes that combine meaningful impact with modest implementation effort. This makes the occasional performance sprint a recurring activity without requiring someone to stop feature work just to find out whether useful work exists.

3:263:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:26 · section reference included

A weekly workflow with replaceable parts

The infrastructure had to remain flexible across compute providers, agent harnesses, and models. It also needed secure tool calls, explicit permissions and authentication, and both scheduled and event-driven triggers. A weekly sweep handles proactive investigation; a webhook could initiate an investigation after a service-level objective breach. Maintainability mattered just as much: expectations rise after a workflow starts working, so its instructions and feedback loop must be easy to revise.

The team chose GitHub Agentic Workflows. The workflow describes the job, its goal, and the analysis to perform over a repository and its production data. In the setup Walter presents, GitHub Actions runs weekly, Claude Code performs the investigation, Hud exposes runtime intelligence through MCP, and the resulting report goes to Slack.

Diagram places Claude Coding Agent and Hud Runtime Intelligence inside GitHub Actions, connected by MCP, with a Sends Report arrow pointing to Slack Weekly Report.Open full source frame
A weekly GitHub Actions workflow connects Claude and Hud to a Slack report.

Those choices establish a working deployment rather than a required vendor combination. Walter names Cursor or Copilot as alternative agents, and Teams or email as alternative destinations. The operational requirement is that the investigation runs unattended and its output reaches somewhere engineers already pay attention.

4:545:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:54 · section reference included

Deliver a diff and evidence

Production traces, queries, and latencies give the agent a starting point, but finding something optimizable is not enough. Walter uses a flow that runs every three weeks and a possible 20-millisecond reduction as examples of opportunities that may not deserve attention. Frequency and business relevance determine whether an apparent inefficiency matters.

The intended loop carries an opportunity all the way to a reviewable change:

  1. Identify the slowdown in production evidence.
  2. Establish its root cause and why the affected flow matters.
  3. Implement a code change and produce the diff.
  4. Rerun tests and verify the impact on the specific flow.
  5. Hand the change and supporting evidence to a human reviewer.

The deliverable is a verified improvement proposal, not merely an optimization suggestion. Walter illustrates the business context with a flow running 7,000 times a week. That context gives the reviewer a reason to care about the evidence accompanying the diff.

7:167:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:16 · section reference included

Three ways a reasonable-looking fix fails

Putting a human review gate at the end did not make the preceding automation reliable out of the box. Three recurring failures exposed the need for a more explicit investigation method:

  • Plausible but unverified: The agent identified something that could theoretically cause a slowdown, but verification showed it did not explain the production problem.
  • Complex queries: Hud uses ClickHouse, a columnar database whose behavior does not always fit familiar SQL patterns. Access to a query tool did not ensure the agent would ask the right question.
  • The lazy fix: Catching an exception could suppress the visible failure without explaining why the exception occurred or why results were lagging.

The response was to encode the playbook a senior engineer would follow: gather the right evidence, investigate the cause, and verify the proposed repair. More data alone would not supply that discipline.

8:218:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:21 · section reference included

Connect production behavior to functions

Production context can be simultaneously excessive and incomplete. A large volume of low-signal telemetry makes reasoning difficult, while a missing log, trace, or metric leaves room for an unsupported explanation. Service-level CPU and memory measurements, or an endpoint’s P90 latency, also describe the system at a different level from the functions and files a coding agent edits. The agent needs a connection between those views.

Hud calls that connection prod-to-code: explain production behavior at the function level where the agent reasons. Function-level context is linked back to the endpoint, event consumer, or cron job that initiated the work.

Slide titled Prod-to-Code states that agents’ context is on a function level, above an editor screenshot showing code and a function selection menu.Open full source frame
Prod-to-code: agents work with function-level context.

For an endpoint that sometimes takes seven seconds, the investigation can ask where that time went: a local function, a database call, an LLM request, or another microservice. Walter describes a function invocation map that records where calls originate and how frequently they execute. Deeper forensic evidence is captured selectively for requests exceeding P99 or another configured threshold, providing a concrete slow request to investigate rather than requiring equally detailed capture everywhere.

The heads-up display also places runtime information over the code. That interface makes the same structural point as the data model: production evidence becomes easier to use when it is attached to the code responsible for the behavior.

9:5910:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:59 · section reference included

Layer investigation skills over queries

The base layer is HudQL, which Walter describes as ClickHouse queries over functions, endpoints, and forensic data. Query access sometimes sufficed, but repeatedly getting the agent to formulate the right query introduced variance in evaluations. Skills added reusable investigation procedures above that access layer.

InvestigationProcedure
HTTP 500Trace the error to its origin.
Memory spikeIdentify work on the affected pods at that time; compare with a baseline.

These procedures specify what evidence to retrieve and what comparison to make. Walter reports that this more methodical approach produced more consistent results. Automations sit above the skills: repair issues as they arise, investigate apparently unused code, or run the recurring performance sweep. Her unused-code example looks for functions not observed running for 60 days; absence in one observation scope is a candidate for investigation, not proof that no other service calls the function.

Three-tier blue pyramid labels HudQL with Clickhouse at the base, Skills for HTTP 500, performance and memory spike in the middle, and Automations for auto-fix and dead code at the top.Open full source frame
Runtime Intelligence MCP layers automations and skills over HudQL.

A performance skill can search for concrete patterns: artificial delays such as sleeps and timeouts, N+1 queries, missing indexes, and sequential asynchronous operations. For example, if an application fetches an order count separately for each customer, a candidate batched SQL query is:

sql

SELECT customer_id, count(*) AS order_count
FROM orders
WHERE customer_id IN (101, 102, 103)
GROUP BY customer_id;

The application must preserve the original behavior, including representing customers with no orders as having a zero count. Runtime evidence then determines whether those repeated calls are actually a bottleneck and whether batching improves the affected flow. The pattern supplies a search target, not a diagnosis by itself.

Walter’s example is a codebase with two decades of history and hundreds of contributors: accumulated inefficiencies are unsurprising, and some are straightforward to remove once found. Static analysis can identify useful candidates too. For an automation expected to run repeatedly and earn trust, however, production context helps establish which candidates explain real slow endpoints.

12:5113:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:51 · section reference included

Reviewer attention is part of the cost

Once evaluations looked good, the workflow could find real slow endpoints, score opportunities, test changes, and verify them each week. Automatically opening pull requests seemed like the natural next step. But Walter’s image of 80 small PRs arriving at once captures the problem: verified work can still overwhelm the people expected to review it. The team instead started by surfacing one opportunity at a time, building both appetite and habit.

Prioritization combines three considerations:

  • Hot-path relevance: How frequently does the flow execute, and how critical is it?
  • Business impact: Payments and signup flows deserve particular attention. Would a product manager prioritize this problem?
  • Change risk: A migration or another risky modification increases the effort required from the reviewer.

The target is highest return on engineering attention, not simply the largest possible latency reduction. Automating investigation makes it practical to compare impact and risk before asking a person to spend time on an opportunity.

15:3115:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:31 · section reference included

Make the report worth opening

Instead of a flood of pull requests, the team produces a short report that explains why an issue deserves attention. Walter’s report example describes an endpoint that usually takes about 200 milliseconds but intermittently takes 45 seconds. She attributes the slow requests to using DISTINCT rather than Mongo’s search index. That is a case-specific diagnosis: MongoDB’s distinct() documentation explicitly allows index use, so distinct operations and indexed execution are not inherently alternatives.

The report pairs a brief explanation of the problem with the proposed fix. The recipient can create a ticket, request a PR, or inspect the finding. Walter reports that these small, readable summaries made a substantial difference because they respect the human’s role: confirming both that a fix is good and that the issue is worth fixing.

The feedback loop continues after deployment. Walter describes the endpoint’s latency becoming flat again once the change is deployed. Seeing that outcome makes the next report easier to trust and more likely to receive attention. The account supplies no measurement window or reproducible before-and-after benchmark, so it supports the adoption mechanism rather than a general speedup claim.

17:2117:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:21 · section reference included

Automate work that was not happening

Scoring and guardrails begin with defining what matters. Cheaper fixes are still not free: each carries implementation cost, reviewer attention, and the risk of changing production behavior. A workflow that generates more changes without accounting for those costs can create work instead of removing it.

The distinctive gain here is that weekly opportunity discovery previously did not happen in engineers’ day-to-day work. The automation does more than accelerate an existing task. It makes a neglected phase routine, surfacing improvements that would otherwise remain undiscovered.

Walter’s principle is “context over cleverness.” Relevant runtime information and well-defined skills make the agents more useful. Although newer models improve the results, she considers existing models sufficient for most of this automation. The remaining responsibility is to steer them with domain knowledge: which issues matter, which changes are worth making, and which latencies materially affect customers. There need not be one universally correct optimization when business priorities determine the value of the work.

18:4618:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:46 · section reference included

Unattended engineering has a higher trust bar

Interactive coding assistance benefits from an engineer who is present, understands the context, and can redirect a mistake. Walter uses an agent working 80% of the time in an IDE as an illustration of that tolerance. An unattended workflow cannot depend on continuous correction. If it strays, it can waste review time and gradually make people stop trusting its output.

The hardest requirement was reaching confidence in two things before the handoff: the issue deserves repair, and the fix has been verified in runtime. A human still reviews the result. Walter’s closing language about 80–90% trust is an informal aspiration, not a measured reliability threshold. The operational standard is concrete: when the automation asks for attention, it should already have done enough investigation and verification to make that attention worthwhile.

20:5821:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:58 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…