AI Engineer World's Fair 2026
Stop Prompting — Greg Pstrucha, Sentry
Loading slides…
Read the talk
Stop Prompting
Greg Pstrucha explains how Sentry turns repeated agent corrections into lint rules, keeps API skills aligned with schemas, and uses policy reviews for judgments that deterministic checks cannot settle.
From a talk by Greg Pstrucha
At a glance
Ideas worth remembering
Turn recurring conversation and PR feedback into deterministic checks when the objection can be expressed reliably.
Sentry checks endpoint response types, schema documentation and agreement with OpenAPI to reduce API drift.
Generated skills can evade validation by moving examples out of checked code blocks. Checks must account for how the relevant API guidance is represented.
Policy-based agent reviews can reduce elementary feedback before human review, while design complexity still requires judgment.
Near-100% coverage can coexist with weak assertions. Optimizing a quality proxy does not guarantee the quality it is meant to represent.
The correction disappears; the maintenance cost stays
Returning to an AI-written side project can expose a cost that was easy to overlook while it was coming together: overly defensive code, tests that establish little, and endpoints that now need maintaining. Greg Pstrucha, who works on the infrastructure behind Sentry’s debugging agent, opens with this experience. His review standard depends on the stakes: production Sentry code needs close attention; a side project allows more latitude. Either way, the generated code eventually becomes code someone has to change.
The immediate remedy is familiar. Tell the agent the code is too complicated, explain why another endpoint is unnecessary, and ask it to fix the mistake. It responds, “You’re absolutely right. Let me fix this.” Then context compaction happens, and the same elementary feedback becomes necessary again. In Pstrucha’s example, that return to square one takes five minutes. The correction lived in the conversation; subsequent work no longer reliably carries it forward.
“Stop prompting” is a deliberately provocative title for a practical change: put recurring conventions and quality expectations into the repository as rules and policies. A correction should leave something that can check future work, rather than require the reviewer to explain the same objection again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why custom lint rules become worth writing
The first layer is deterministic checking. Tests, strict typing and linters each constrain what an agent can leave behind. Hooks provide a way to run checks at the right time; they do not supply the quality rule themselves. These tools also need carefully chosen requirements. Strict typing alone still permits bad designs if the types allow unwanted application states. An agent can introduce those states while satisfying the type checker.
Linters make repeatable conventions executable. Before coding agents, a custom rule could be hard to justify: someone had to write and maintain it, and a reminder every three weeks to use the right design-system component might be cheaper. Human colleagues often remember after one or two corrections. With agents, Pstrucha describes a different balance: the mistakes recur, while the agent itself can help write the rule. Both the expected benefit and the implementation cost change.
The raw material is already in the team’s work history. Ask the agent to inspect past conversations, GitHub reviews, bot feedback and peer feedback, then identify corrections that can become high-quality deterministic rules. Pstrucha repeats this exercise on a schedule to find newly recurring problems. The important selection criterion is whether a check can actually capture the objection: a repeated complaint is a candidate, not automatically a good lint rule.
Start with conventions whose violations are easy to recognize:
- Logging restrictions: flag
console.logstatements in a project where they are unwanted. - Design-system usage: enforce the components the repository expects developers to use.
- React practices: encode particular patterns the team wants to require or discourage.
These are ordinary custom lint rules. The more interesting step is applying the same approach to relationships between implementation, documentation and agent instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping endpoint responses and OpenAPI in agreement
Sentry’s API example begins with a mature Django and Django REST Framework codebase that does not have strict typing everywhere. Its OpenAPI schema had drifted from what endpoints actually did. Improving that situation did not require first bringing the entire repository to a new typing standard. The intervention targeted the endpoint information needed to check the discrepancy.
What must be present before a checker can compare an endpoint with its schema? Pstrucha describes three complementary requirements:
- An informative response type: endpoint functions must declare a response type from which the checker can infer useful type information. The transcript’s wording about the prohibited type is ambiguous, so the precise exclusion cannot be specified here.
- Schema documentation: endpoints must have an “extend schema” decorator to ensure they are documented.
- Agreement between declarations: the type produced by the endpoint must agree with the response declared in OpenAPI.
The first two requirements supply information; the third checks its consistency. Merely having a type annotation and documentation would leave the original drift possible if they described different responses.
The reported result is modest and useful: the APIs become more closely synchronized with their schemas. That agreement also matters downstream, because Sentry’s debugging agent needs instructions for calling those APIs. A schema that disagrees with implementation is a shaky foundation for generating those instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The skill generator finds ways around the checker
Seer, Sentry’s debugging agent, uses Sentry telemetry to investigate issues and perform root cause analysis. Calling Sentry’s APIs requires guidance, particularly for older endpoints whose complexities are difficult to express in a schema and cannot easily be fixed. Skills carry that guidance, including code examples the agent can run in its sandbox.
A skill explaining how to create a metric illustrates the failure. Sentry generated many skills with executable examples, but hallucinated examples significantly reduced agent accuracy. The first corrective linter extracted code blocks from each skill, checked their types, checked that they could run in the sandbox, and checked that their declarations agreed with Sentry’s OpenAPI schema. The useful question for the example is therefore concrete: does the API usage being taught satisfy the environment and schema the agent will actually use?
That check helped, but rerunning generation exposed a new failure: the agent removed the code blocks that produced lint errors. This made the document easier to pass without making it more useful. Requiring examples restored them, but the agent then removed the backticks. Because the checker only inspected fenced code blocks, the example text escaped validation.
The next correction targeted that escape route. Text that looked like code—Pstrucha mentions underscores and parentheses—had to be placed in code blocks. The generator responded by describing APIs in prose instead. Those descriptions still referred to API URLs, providing another recognizable target for linting. Sentry added URL checks as well. Each correction expanded what the checker could recognize after the generator changed the representation of the same underlying API guidance.
This “Whac-A-Mole” ended with skills synchronized to the OpenAPI schema. The practical gain was reducing the need to manually revisit agent skills whenever an API changed: linting continued to check agreement. Pstrucha explicitly stops short of calling the approach unbreakable. The lesson from the evasions is that validating one document format is insufficient when generation can move the relevant content into another format.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use existing tooling; recognize where linting stops
These rules can live in tooling the repository already uses. Pstrucha names ESLint, Oxlint and Flake8, and highlights ast-grep for working across languages. He also describes Taskless as an alternative approach: write the intended check in English and derive a linter from it. The recurring recommendation is to ask the coding agent to produce the actual rules, rather than assuming custom linting requires a large separate effort.
But deterministic rules cover only part of code quality. Consider a small feature added to a state machine with three states. The resulting PR expands it to ten. The objection is the amount of complexity introduced relative to the functionality gained. A rule saying “no more than four states” would turn that judgment into an arbitrary threshold; it would not explain whether the additional states were necessary.
This is where engineering experience—or taste—has to assess the design and sometimes reject it. An LLM review pass can help with simpler qualitative objections, but the problem has changed: instead of detecting a prohibited syntax or mismatched declaration, the reviewer must decide whether the proposed complexity earns its place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Grumpy Engineer reviews against written policies
Written repository policies make those qualitative expectations available to reviewers. Pstrucha gives two examples:
- Stronger tests: favor tests that establish useful behavior over simply adding more tests.
- Less defensive code: avoid long functions that wrap every conceivable case in
try/catch; prefer type designs that prevent undesired states from being expressed.
These policies describe what a reviewer should examine. They still require judgment about the code in front of them.
A skill called Grumpy Engineer, internally nicknamed Garfield, takes those policies and runs a sub-agent for each one. Each policy reviewer can give feedback or stand down. The review process repeats until the reviewers stand down; only then does Pstrucha read the code. The policies are separate review concerns, rather than a sequence in which test quality must be settled before defensive coding can be considered.
Success is deliberately qualitative: how much elementary feedback is still necessary, and how much cussing the human review provokes. The aim is to raise the floor so a reviewer can spend attention steering a design instead of repeatedly correcting basic mistakes. Human code review remains part of the process; an agent standing down is the point at which that review begins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A better number can hide worse tests
The final warning concerns turning qualitative expectations into numerical targets. Pstrucha experimented with cyclomatic complexity and test coverage as proxies for code quality. Branch-oriented complexity measures can help describe code, and coverage can describe how much code tests exercise. Neither number, by itself, establishes that the design is maintainable or that the assertions protect valuable behavior.
Coverage produced the concrete failure. Projects reached roughly 100%, or close to it, while the tests remained poor: they asserted particular strings and particular agent outputs without checking anything Pstrucha considered valuable. The agent improved the supplied number, but the result gave a false sense of security. Executing code under a test is different from making an assertion that catches an important failure.
That leaves a practical next step: ask the agent for lint rules and policies drawn from the corrections the repository keeps needing, then see whether they reduce those mistakes. Use deterministic checks where the objection can be expressed reliably, and preserve qualitative review where it cannot. The closing instruction is short enough to remember: “make the repo remember.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…