[
  {
    "startMs": 13180,
    "text": "All right. Hello, everyone. I'm gonna try to speak as loud as I can so it's reasonably good with all the, all the commotion. Uh, my name is Greg. I'm an AI engineer at Sentry. I work on the infrastructure that powers Sentry's, uh, debugging agent. And I have a little rage baity title of the presentation that calls to stop prompting, but I do not mean loop engineering. I'm going to get, uh, a little bit deeper here. Uh, who still reads code that AI generates?"
  },
  {
    "startMs": 43740,
    "text": "Anyone? It's not a trick question, and I'm not trying to shame you. I, I think it's contextual. I read code when it's Sentry code, and I need to make sure that we are not breaking the actual production for Sentry. But at the same time, if I am working on a side project, it doesn't really matter, uh, to read every line of code, and I can, I can be a little bit more lenient with it. Uh, but then every now and then, I go back to the actual code. I want to make improvements and, um,"
  },
  {
    "startMs": 73680,
    "text": "it's bad. It's, it's overly defensive code. Its, uh, tests are crap. They are not testing anything. Um, I would not want to maintain or work in that code base anymore. So then what you do is you, you talk to the agent. You say, \"Hey, you've done a mistake. This is overly complicated. The code's too defensive. Um, you don't have to create endpoints for everything you're doing, otherwise I have to maintain it forever.\" Uh, and I had to remove a lot of swearing from this slide because in my actual transcript, there is going to be a"
  },
  {
    "startMs": 103720,
    "text": "lot of, a lot of rage. Usually, those are very elementary mistakes that the agents are making if you don't put any guardrails in place and, um, and that just outrages me. So then you do that. The agent says, \"You're absolutely right. Let me fix this.\" And it's going to run and try to fix the, the mistakes, and five minutes pass, and compaction of context happens, and we're back to, back to square one."
  },
  {
    "startMs": 134380,
    "text": "And you would have to now repeat yourself, re-prompt your fixes, re-prompt the agent on the right track. And my entire thesis for this talk, my entire claim here is you should stop prompting. You should start codifying the actual rules that the agent should be following, um, rules and policies that you want to establish inside your code base that are going to capture both convention and the quality of code that you are expecting and would make you proud."
  },
  {
    "startMs": 165940,
    "text": "And so when I say you would want to add deterministic checks or rules, what are you thinking of?"
  },
  {
    "startMs": 174440,
    "text": "Yes."
  },
  {
    "startMs": 177240,
    "text": "What's that? Evals. Evals, yes. Hooks. Hooks, yes. Um, the three that I'm thinking of before I even get to hooks and evals, hooks and evals are important. Tests are absolutely the class of that. Hooks are how you run those things at the right time. Um, to me, the three things that I want to optimize on the deterministic level are making sure that I have tests, making sure that there is strict typing, making sure that there is lin- there is linters. Those are not total statements, and they are not enough to actually get your, uh, agent to write good"
  },
  {
    "startMs": 207180,
    "text": "code. As an example, if you just take from this that you have to put strict typing in place, you will still be in a bad place. Um, if you don't discourage or ban, um, a way of typing that removes edge cases you don't really wanna see, then y- the behaviors that... or the states that are representable in the app are going to lead the agent to introduce them. But the thing that I want to predominantly"
  },
  {
    "startMs": 237000,
    "text": "focus on in this talk is linters. Linters are our way to codify, um, the rules in a deterministic manner that we want to, um, remove from the, from the code base. Historically, like, before AI, when you think about linters, um, the way that this worked is the trade-off wasn't clear. Writing an actual linter took a lot of work. Um, it, it was a maintenance cost that you had to sort of sign yourself up for, and you had to consider how often the"
  },
  {
    "startMs": 266760,
    "text": "issue that you tr- try to lint away comes up. Is it even worth it? And if it's, if it's pre-a- pre-agents, and those are humans that are working in my code base, and the only thing I'm doing is once every three week telling people to use the right component from the design system, then I don't necessarily need the linter for that, right? And people have memory, so if I tell them that once or twice, they will usually remember that. But then this shifts entirely into this new world where the"
  },
  {
    "startMs": 296680,
    "text": "things that are writing code do not have any memory and are going to make the same repeated mistakes over and over again. And on the, on this-- at the same time, to write a new linter is very cheap and very easy. You can just have the agent do that, and it's going to be much less frustrated with the actual lint rules that you're trying to, to write. So my claim for this is simple, and if you take one thing from this entire talk that I'm gonna yap about a little bit more, is you can, after this talk, take your laptop,"
  },
  {
    "startMs": 326720,
    "text": "go to your agent and ask, \"Hey, look at all my past transcripts. Look at my GitHub reviews, the feedback from the bots, the feedback from my peers, and try to codify which ones of those are actual high-quality lint rules that are deterministic and will be able to capture issues that we otherwise re-prompt all the time.\" Um, that's something that I do now, and I happen to do that even more often, uh, sort of on the schedule to try to see whether there are new things that I haven't captured."
  },
  {
    "startMs": 358060,
    "text": "So here are some examples of what I mean when I say- ... custom lint rules. Those are, uh, lint rules that try to capture conventions that you want in your code base. There is like a whole simple class of them. Those are the, uh, ones that you know and have written for a long time. Those are things like, I wanna lint out console log statements from a particular project, or I want to make sure that if I have a design system or React practices that I wanna follow or discourage, those are codified and are"
  },
  {
    "startMs": 387840,
    "text": "linted away. Those are the simple things. But I think with agents, we are getting into more interesting complexity. We can push this a little bit further. So one example here. This is a little bit of a tricky example, but, um, I'll go through it. Sentry is a pretty old code base. We are on Django. We are on DRF. We don't have strict typing everywhere. This is a very common case for any mature code base, where you're not gonna be at the top leading edge of the technology. Um, but we still want to improve the state of"
  },
  {
    "startMs": 417800,
    "text": "the world for, for the agents. So concretely here, um, there is... There was a lot of drift of what the OpenAPI schema for Sentry said it's going to do and what it did. And so we started introducing l- lint rules for that. First lint rule is if something is a endpoint, endpoint, a function for an endpoint, it needs to return a response type, and within that response type, there cannot be any types, so that we can infer the actual information of the type. Second, if it's a"
  },
  {
    "startMs": 447840,
    "text": "endpoint, you get that little extend schema decorator so we can ensure that it's documented. And then third, enforce that the actual type being produced here agrees with what the OpenAPI schema at the end of the day, um, declares. And that helps us because now APIs are a little bit more in sync, and that gets us to even more insane examples of, um, of linters. So Seer Agent is a"
  },
  {
    "startMs": 477600,
    "text": "debugging agent, and it has to talk to Sentry a lot. It uses Sentry telemetry to debug and RCA issues and provide, uh, root cause analysis solutions, all of that. Because it has to talk to ope- uh, to, um, Sentry API, we oftentimes have to guide it, and the way we do it is through skills. Oftentimes, those are old endpoints that we are not able to, um, easily fix, and they have some complexities that are hard to describe on the level of schema itself. So instead, we guide that through"
  },
  {
    "startMs": 507340,
    "text": "skills, which is a pretty common pattern in the agentic world. But here is an example of a skill that tells how a metric should be created at Sentry. We generated a lot of those skills and examples with those code blocks as the, as the examples of code that can then be run in the sandbox by agent. And in the original attempt to do that, we lost the accuracy of the agent significantly because lots of those examples were hallucinated. So the first linter we created was a linter that"
  },
  {
    "startMs": 537200,
    "text": "says if there is a code block inside a skill, you gotta extract those code blocks. You have to make sure they are type-checked, they can run in our sandbox, and whatever they declare agrees with OpenAPI schema from Sentry. And that helped, but what, what the fir- the first thing that the agent did when we tried to rerun the skill generation with that rule is, \"Oh, there are lint rule errors. Let me remove the code blocks.\" And that wasn't very desired"
  },
  {
    "startMs": 567220,
    "text": "by us, so we said, \"No, no, no. Examples have to be put in code blocks.\" So it put the examples back, but it removed the back ticks because only the back ticks are linted. So let's, let's fix the problem, right? So we're like, \"No, no, no. If it looks like a code, smells like a code, if it has underscores and parentheses, push those into code blocks.\" So it's like, \"Okay. So you want me to put code into code blocks? I'm going to describe the APIs.\" So now in prose it started"
  },
  {
    "startMs": 596800,
    "text": "describing what the APIs should do, but it did one mistake, which is it referred to the URLs, uh, of the APIs. So then we linted away the URLs. And we played this little Whac-A-Mole for a little while, but at the end of the day, we ended up with skills that are synced to our OpenAPI schema, and that wasn't that expensive to do. And we have..."
  },
  {
    "startMs": 620600,
    "text": "I'm not gonna say unbreakable model, but we have a model where when API changes, we actually don't have to follow up on the agent skills all the time. It's, it's, it's kept in sync through the, through the linters."
  },
  {
    "startMs": 633020,
    "text": "And when it comes to the actual movements or the actual how do we do this, you can use any sort of tool that you're using already. ESLint, Oxlint, Flake8, Kleppy, ast-grep. ast-grep's great, uh, because it's language agnostic, so it kind of, you know, unified tooling. Warm, fuzzy feelings about that. Um, there is a bunch of new tools that are popping up that are trying to approach it from a different, uh, perspective. One example is Taskless, where is instead of writing the actual linter, you write what you want the"
  },
  {
    "startMs": 662920,
    "text": "linter to do. You write your intent in English language, and then the actual linter is, um, derived from it. Um, and then the last thing that I, uh, that I will repeat is try asking the agent for the actual lin- linters. You will be surprised, uh, what it can, what it can generate. But that's only half the picture. What if there are rules that are deterministic cannot express, um, cannot be expressed through linters? Th- there is plenty of those, right?"
  },
  {
    "startMs": 693180,
    "text": "Those are all types of qualitative measures. Like, you have a state machine with just three states, and I created a small PR to add small functionality and ballooned up the number of states into 10. I don't know how to lint against that. I don't think I reliably can lint against that. I can, like, create small tactical things like no more than four states, but it's all bullshit. It's all made up. This is a, this is a situation where we're talking about qualitative measures and,"
  },
  {
    "startMs": 723420,
    "text": "um, this is where you are engaging your engineering experience or if Twi- as, as Twitter would say, your taste. Um, and you just have to decline it straight up. Or you can have an LLM agent that is going to do a pass and for simpler thing, help you with that."
  },
  {
    "startMs": 742448,
    "text": "Concretely here,"
  },
  {
    "startMs": 745488,
    "text": "I've, uh, I've started to create policies for the repository, and those policies are such as, I don't want too many tests, I want stronger tests, or I do not want a code that's too defensive. I'm sure you've seen those where every single possible case is try caught by the agent and it's, it's just sloppy, long, long functions. I would rather have code that in its type system does not allow to express states that are"
  },
  {
    "startMs": 775508,
    "text": "undesired. Um, so I created those policies, and then I created a skill that I called Grumpy Engineer. We call it Garfield internally. And the way it works is it takes all the policies and runs sub-agents for each of the policies, and the sub-agent can either give feedback or stand down, and we loop that. And this is the closest you will hear me doing loops, by the way. Uh, we will loop that until the agent stands u- stands down and says, \"I-"
  },
  {
    "startMs": 805688,
    "text": "I'm okay with that. I validate that. This is fine.\" And only then I read the code. And sort of my metric of success here is how much I'm cussing at the agent, how much when I'm reviewing the code, I'm like, \"Oh yeah, this is not ideal, but I can, I can steer it here, and I don't have to do very elementary feedback all the time.\" It is still qualitative, but the goal here isn't to remove the code review quite yet. You know, these things will change as models get better, but more so to raise the floor and make it"
  },
  {
    "startMs": 836008,
    "text": "easier to, um, to work with the agents, uh, as you're, as you're going forward. There is also one thing that I wanna mention, that is, um, beware of a snake oil. We are talking about the tricky part being qualitative measures, and I've done experiments where I tried to capture a qualitative measure in a quantitative metric. So two concrete examples. One of them is, um, cyclomatic"
  },
  {
    "startMs": 865568,
    "text": "complexity, and the other one is test coverage. Cyclomatic complexity, uh, is a measure that says how maintainable and testable the code is, and it tries to base that on how many branches happen in the code and how deep the call stack goes. And, um, and test coverage, you know, test coverage tries to tell you how much of the code is covered by tests. Um, and the problem with that is when you give a number to the agent, the agent will optimize the shit out of that number."
  },
  {
    "startMs": 896908,
    "text": "And it did, and I, I got code base... I got projects where I got to like a hundred percent or close to a hundred percent of test coverage. But the actual tests are bad. They are like asserting particular string and particular output of the agent. They are not measuring anything of value. Um, and more so, they are giving you a false sense of security. So beware of that. Beware of, um, the complexities, uh, of, of actual qualitative metrics. Um, and that's it. That's, that's my call."
  },
  {
    "startMs": 926848,
    "text": "My call to action for you is after this, try to ask your agent for linters and policies and see whether that helps you. Um, try things and make the repo remember, and that's, that's my time. Thank you."
  }
]
