[
  {
    "startMs": 160,
    "text": "Hi everyone, my name is Rajat Kotial."
  },
  {
    "startMs": 2720,
    "text": "I'm part of the automated reasoning"
  },
  {
    "startMs": 4400,
    "text": "group at AWS. And today I'm going to"
  },
  {
    "startMs": 7200,
    "text": "talk about a project I worked on"
  },
  {
    "startMs": 9519,
    "text": "recently called KO specification"
  },
  {
    "startMs": 11920,
    "text": "analyzer. And um we use certain"
  },
  {
    "startMs": 14960,
    "text": "techniques which involves formal methods"
  },
  {
    "startMs": 16880,
    "text": "and automated reasoning to prove"
  },
  {
    "startMs": 19760,
    "text": "specification that goes to your coding"
  },
  {
    "startMs": 21680,
    "text": "agent before it writes code."
  },
  {
    "startMs": 30720,
    "text": "So let's take a certain bug right"
  },
  {
    "startMs": 34719,
    "text": "this is a specification"
  },
  {
    "startMs": 37760,
    "text": "uh snippet and there are two rules for"
  },
  {
    "startMs": 40480,
    "text": "example and uh the rule two says when an"
  },
  {
    "startMs": 43840,
    "text": "order is submitted and this is for a a"
  },
  {
    "startMs": 46640,
    "text": "general e-commerce website let's say"
  },
  {
    "startMs": 49200,
    "text": "when an order is submitted and the"
  },
  {
    "startMs": 50960,
    "text": "inventory is unavailable we place the"
  },
  {
    "startMs": 53600,
    "text": "order as a back order which makes sense"
  },
  {
    "startMs": 55760,
    "text": "when you read it individually and"
  },
  {
    "startMs": 57680,
    "text": "there's also another rule in the same"
  },
  {
    "startMs": 59120,
    "text": "specification file uh which you hand to"
  },
  {
    "startMs": 61520,
    "text": "the coding agent which says u no order"
  },
  {
    "startMs": 64080,
    "text": "shall be placed as a back order and this"
  },
  {
    "startMs": 66799,
    "text": "is a rule we've set so individually read"
  },
  {
    "startMs": 70320,
    "text": "these two lines make sense right but in"
  },
  {
    "startMs": 74000,
    "text": "when an LLM reads this reads both the"
  },
  {
    "startMs": 76799,
    "text": "rules together both cannot hold true at"
  },
  {
    "startMs": 79119,
    "text": "the same time so we need to make sure"
  },
  {
    "startMs": 82640,
    "text": "the we resolve this ambiguity that the"
  },
  {
    "startMs": 84880,
    "text": "LLM is observing."
  },
  {
    "startMs": 88479,
    "text": "And uh"
  },
  {
    "startMs": 91280,
    "text": "the scope and the reason we"
  },
  {
    "startMs": 96320,
    "text": "uh put an effort into this project is"
  },
  {
    "startMs": 98880,
    "text": "based on public research and recent"
  },
  {
    "startMs": 101119,
    "text": "research, we've observed there's a 20 to"
  },
  {
    "startMs": 102880,
    "text": "40% drop with unclear prompts on the"
  },
  {
    "startMs": 106720,
    "text": "accuracy the LLM uh generates on the"
  },
  {
    "startMs": 109600,
    "text": "code accuracy, right? and 60 to 90% of"
  },
  {
    "startMs": 112720,
    "text": "valid code is still wrong after it"
  },
  {
    "startMs": 114479,
    "text": "writes it. And also in that case we"
  },
  {
    "startMs": 117759,
    "text": "observe two times the more more"
  },
  {
    "startMs": 119759,
    "text": "regressions when we uh deploy that code."
  },
  {
    "startMs": 125759,
    "text": "So where do bugs like these hide? Right?"
  },
  {
    "startMs": 128720,
    "text": "We want to detect these bugs before they"
  },
  {
    "startMs": 130959,
    "text": "reach the codebase. And the way only way"
  },
  {
    "startMs": 133760,
    "text": "to do that it's similar to how we write"
  },
  {
    "startMs": 135440,
    "text": "design docs. Similar to how we draw"
  },
  {
    "startMs": 138160,
    "text": "diagrams and flowcharts to make sure the"
  },
  {
    "startMs": 140720,
    "text": "flow is set before we write any code. We"
  },
  {
    "startMs": 142959,
    "text": "want to verify that without a human or"
  },
  {
    "startMs": 146480,
    "text": "anyone reading these files as the first"
  },
  {
    "startMs": 149280,
    "text": "reviewer"
  },
  {
    "startMs": 151200,
    "text": "in one sentence. Let's say the ambiguity"
  },
  {
    "startMs": 154800,
    "text": "and wording wrong level of detail but"
  },
  {
    "startMs": 157760,
    "text": "across sentences we can observe"
  },
  {
    "startMs": 159599,
    "text": "conflicts and gaps in the specification"
  },
  {
    "startMs": 162160,
    "text": "or rules. Right? sentences need reviews"
  },
  {
    "startMs": 165519,
    "text": "that need search in between them and"
  },
  {
    "startMs": 168000,
    "text": "ambiguity searches."
  },
  {
    "startMs": 170560,
    "text": "So we have a three-step process to do"
  },
  {
    "startMs": 173360,
    "text": "this where the LLM will first refine it."
  },
  {
    "startMs": 176400,
    "text": "It'll try to clean up the wording on its"
  },
  {
    "startMs": 178000,
    "text": "own and u if it succeeds then well and"
  },
  {
    "startMs": 181680,
    "text": "good but if it cannot we then formalize"
  },
  {
    "startMs": 184879,
    "text": "the um English language into SMT"
  },
  {
    "startMs": 188640,
    "text": "formulas. So, SMT lib is a certain set"
  },
  {
    "startMs": 192720,
    "text": "of type of function which we pass to a"
  },
  {
    "startMs": 195280,
    "text": "SAT solver a satisfiability checker"
  },
  {
    "startMs": 197840,
    "text": "where it's going to give out a boolean"
  },
  {
    "startMs": 199519,
    "text": "saying this expression that you've given"
  },
  {
    "startMs": 202239,
    "text": "me is valid true or false invalid"
  },
  {
    "startMs": 207120,
    "text": "and then once the solver gets back to us"
  },
  {
    "startMs": 209599,
    "text": "if that expression is valid then we"
  },
  {
    "startMs": 211760,
    "text": "don't do anything. But if it detects any"
  },
  {
    "startMs": 213440,
    "text": "ambiguity between two sentences or"
  },
  {
    "startMs": 215519,
    "text": "within a sentence where there's two"
  },
  {
    "startMs": 217840,
    "text": "meanings to a sentence, then we detect"
  },
  {
    "startMs": 219760,
    "text": "that and tell the LLM to recheck that"
  },
  {
    "startMs": 223680,
    "text": "sentence and it's going to reiterate on"
  },
  {
    "startMs": 225840,
    "text": "that."
  },
  {
    "startMs": 227760,
    "text": "For example, the rules we saw"
  },
  {
    "startMs": 231360,
    "text": "we've seen when order is submitted and"
  },
  {
    "startMs": 233440,
    "text": "inventory is unavailable, we shall place"
  },
  {
    "startMs": 235920,
    "text": "the order as a back order. And in SMT"
  },
  {
    "startMs": 239680,
    "text": "format, it kind of looks like this. This"
  },
  {
    "startMs": 241360,
    "text": "is like a math formula, right?"
  },
  {
    "startMs": 248560,
    "text": "Once we go deeper, we"
  },
  {
    "startMs": 251280,
    "text": "once we detect this, we also do a"
  },
  {
    "startMs": 254480,
    "text": "three-step process again where we sample"
  },
  {
    "startMs": 257199,
    "text": "the translations again and again. We ask"
  },
  {
    "startMs": 259759,
    "text": "the LLM to translate it seven times or"
  },
  {
    "startMs": 261759,
    "text": "10 times and to understand whether this"
  },
  {
    "startMs": 264880,
    "text": "LLM is behaving this way uh based on its"
  },
  {
    "startMs": 269840,
    "text": "inherent capabilities or is something"
  },
  {
    "startMs": 271680,
    "text": "affecting it to behave this way and then"
  },
  {
    "startMs": 274080,
    "text": "based on logical equivalence of these"
  },
  {
    "startMs": 276720,
    "text": "seven or 10 um different set of"
  },
  {
    "startMs": 279520,
    "text": "interpretations by the LLM we cluster"
  },
  {
    "startMs": 281759,
    "text": "them on logical equivalence which sets"
  },
  {
    "startMs": 284639,
    "text": "which uh um entries came under which uh"
  },
  {
    "startMs": 289680,
    "text": "assumptions that the LLM made for itself"
  },
  {
    "startMs": 292960,
    "text": "and then we accept or ask the user to"
  },
  {
    "startMs": 295919,
    "text": "clear out the ambiguity or abstain from"
  },
  {
    "startMs": 298479,
    "text": "implementing that in the codebase."
  },
  {
    "startMs": 305759,
    "text": "This is an example of asking the human"
  },
  {
    "startMs": 309199,
    "text": "um what you want to do with that spec"
  },
  {
    "startMs": 313120,
    "text": "rule that was um perhaps contradicting."
  },
  {
    "startMs": 316960,
    "text": "remove the record is very a very vague"
  },
  {
    "startMs": 318880,
    "text": "statement. The LLM cannot can interpret"
  },
  {
    "startMs": 321199,
    "text": "anything it wants but we shouldn't allow"
  },
  {
    "startMs": 323919,
    "text": "that to happen and we either give two"
  },
  {
    "startMs": 326720,
    "text": "options we keep it as is the record is"
  },
  {
    "startMs": 328639,
    "text": "deleted or we retain it and but hide it"
  },
  {
    "startMs": 331440,
    "text": "from the users which one does the user"
  },
  {
    "startMs": 333520,
    "text": "prefer."
  },
  {
    "startMs": 337600,
    "text": "Um"
  },
  {
    "startMs": 339440,
    "text": "a conflict is when the examples we"
  },
  {
    "startMs": 341919,
    "text": "example we saw rule one is conflicting"
  },
  {
    "startMs": 343840,
    "text": "with rule three and a gap is where no"
  },
  {
    "startMs": 346960,
    "text": "rule covers an idle order in this"
  },
  {
    "startMs": 349600,
    "text": "example where the LLM detected an actual"
  },
  {
    "startMs": 353520,
    "text": "sorry"
  },
  {
    "startMs": 355199,
    "text": "process or pipeline detected there's no"
  },
  {
    "startMs": 358720,
    "text": "um entry for this rule where you are"
  },
  {
    "startMs": 361199,
    "text": "missing something"
  },
  {
    "startMs": 366160,
    "text": "all in Well, it cannot do a lot of"
  },
  {
    "startMs": 368560,
    "text": "things. End of the day, we want to make"
  },
  {
    "startMs": 372080,
    "text": "sure there is no oracle uh for for"
  },
  {
    "startMs": 375360,
    "text": "specification correctness other than the"
  },
  {
    "startMs": 377039,
    "text": "user. User needs to make sure to check"
  },
  {
    "startMs": 380319,
    "text": "every part of the snippet part of the"
  },
  {
    "startMs": 383280,
    "text": "rule book before sending it to the LLM."
  },
  {
    "startMs": 385919,
    "text": "And um we as a tool only can help them"
  },
  {
    "startMs": 391440,
    "text": "and the user is more important here"
  },
  {
    "startMs": 394000,
    "text": "because of implicit domain knowledge."
  },
  {
    "startMs": 396400,
    "text": "The LLM doesn't know a lot of domain"
  },
  {
    "startMs": 398000,
    "text": "knowledge and there's no guarantee I"
  },
  {
    "startMs": 400160,
    "text": "mean LLM only uh user can give that"
  },
  {
    "startMs": 404160,
    "text": "guarantee right and there's thin"
  },
  {
    "startMs": 406319,
    "text": "evidence right now for formalization and"
  },
  {
    "startMs": 408319,
    "text": "how it can be used in agents we are"
  },
  {
    "startMs": 410720,
    "text": "emerging in this technology"
  },
  {
    "startMs": 416960,
    "text": "what do we take back from this um talk"
  },
  {
    "startMs": 420319,
    "text": "check the spec always before you pass it"
  },
  {
    "startMs": 422720,
    "text": "down to the agent uh cluster the"
  },
  {
    "startMs": 425599,
    "text": "translation We what we do is we cluster"
  },
  {
    "startMs": 427919,
    "text": "the translation by logical equivalent to"
  },
  {
    "startMs": 431039,
    "text": "determine the different LLM"
  },
  {
    "startMs": 433360,
    "text": "interpretations and where it's genuinely"
  },
  {
    "startMs": 435599,
    "text": "going wrong. The LLM translates, the"
  },
  {
    "startMs": 438479,
    "text": "solver checks and the human decides. We"
  },
  {
    "startMs": 441759,
    "text": "give the human uh options and if there's"
  },
  {
    "startMs": 444960,
    "text": "no answer to that, we use LLM judges or"
  },
  {
    "startMs": 447759,
    "text": "abstain from implementing that piece."
  },
  {
    "startMs": 452080,
    "text": "Here are the references and u this work"
  },
  {
    "startMs": 455440,
    "text": "has been majorly contributed by my team"
  },
  {
    "startMs": 458639,
    "text": "um Andrea Jin Stefan and Delmos. You can"
  },
  {
    "startMs": 462319,
    "text": "check out their blog in the website link"
  },
  {
    "startMs": 465120,
    "text": "provided and um del deeper into this."
  },
  {
    "startMs": 469039,
    "text": "Thank you."
  }
]
