[
  {
    "startMs": 12560,
    "text": "Thank you for joining this session. Uh, it's a bit of an emotional one. Um, my name is Nathanael, and I'm a non-evaluator. Uh, it's been a lot of time since the last, uh, agent I've been shipping without any evaluation. And today, I want to tell you the story, how I quit this addiction of shipping an agent without any evaluation."
  },
  {
    "startMs": 39240,
    "text": "So I'm working at Air Canada, so compared to the previous talks, more like focused on specific business. Uh, I joined Air Canada in June, uh, of this year. And Air Canada is a huge company. We work in operation, corporate, commercial, and loyalty. Each of those segments have their own challenges. So my team is in, in charge of delivering, uh, AI solution for all of those. And the problem when we tackle s-- uh, a, a task or we create an"
  },
  {
    "startMs": 69280,
    "text": "agent for a task that is so specific, we don't have any public benchmark we can rely on. And that's the main, main problem we-- I'm trying to address in this presentation. More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain expert. And those domain experts sits in different business units, and they are very busy in their own task,"
  },
  {
    "startMs": 99220,
    "text": "and we have a limited access to that. So both of those things went together in a situation where it's very hard for us to build any evaluation or any, any, uh, benchmark for our task. So b-before going to Air Canada, I was working for a different company. Um, it was a company th-- And, and the task was to build a, um, a Q&A assistant for telecommunication engineer. So I don't know if you're familiar with the domain. Um, telecommunication is a huge"
  },
  {
    "startMs": 129160,
    "text": "domain. There is about ten thousand documents, a lot of vendor documents on top of that. There's a logic to know which document you should, you should found in order to answer the question. So it's, it's a bit of a complex task. And in twenty twenty-four, everybody wanted to build RAGs, so we started to do that. Uh, on LinkedIn, it seems super easy. It's not. So, um, the thing is, when we tackle this project, the same problem as at Air Canada, we don't have"
  },
  {
    "startMs": 159060,
    "text": "access to expert. We have limited access to experts. And we, we could not actually grade the answer ourself. So that's the, the problem I, I want to talk today. Funny enough, uh, we were in competition with other teams, um, two external, one internal. It's not like in parallel, but in sequence. And our solution actually worked. What mean worked is actually delivering the value. The people liked the solution, actually used it."
  },
  {
    "startMs": 189160,
    "text": "Uh, but to be honest, I don't know why we actually made it work compared to the other, the other team. So today, I'm just gonna show what I actually did or my team actually did to make it work. Um, so they, they all have the same document, same task. Everything is the same, but our solution somehow worked. So for those that are familiar with RAGs, um, the principle is very-- is, is to ask a question to the system. The system found the,"
  },
  {
    "startMs": 219120,
    "text": "the relevant documents. Based on this relevant document, formulate an answer. It's something that exists for a long time now. And for those that actually tried it, if you found the right page or the right documents, ninety, ninety percent of the time, you have actually the right answer. So the challenge to find the right documents, at least in this task."
  },
  {
    "startMs": 240700,
    "text": "So we created a very complex pipeline that gets complexified over time. If you, uh, publication-- If you found publication back in the time, there's everybody has an amazing technique to improve the, the solution. We try them. It actually didn't improve the solution. So on paper, it worked for them, for the publication, but for us it didn't really help. And more importantly, when we add complexity, we found that, like, we break something else. So something that used to work now is not working anymore. And"
  },
  {
    "startMs": 270140,
    "text": "that's actually the, the, the real problem. So we can add and more i-- we can get more ideas on-- into solving these, uh, different problems, but we, we keep continu-- we keep breaking things over and over. So complexity doesn't always help. And the other problem is this one. Uh, we can change an agent in few minutes, add a certain layer of, uh, query writing or whatever, reasoning step. But to grade that, we had"
  },
  {
    "startMs": 300020,
    "text": "this crazy meeting once a week with experts. We put a gi-giant screen with five different chatbot, ask the same question to all of them, and ask the, the expert to grade them. \"I prefer this one. I prefer this one.\" And funny enough, they're not always agreeing. It's very hard  when you, when you put expert-- two expert in the room, you get three opinions. So that's the, the, the problem we were facing. And so our budget was we can try one solution a week, and this was, uh,"
  },
  {
    "startMs": 329980,
    "text": "not enough to go fast."
  },
  {
    "startMs": 333280,
    "text": "So the-- Before, uh, going further, so the, the, the, the process was the following. We ask, we ask a question, the agent answer. We ask the expert to read. Sometimes they don't know the answer. They have to look up. It takes a lot of time. Um, and we had the budget of one trial a week."
  },
  {
    "startMs": 351380,
    "text": "Um, so there is nothing new here. This binding that is well known is we reverse the problem. So we start with a page. We ask an LLM to write a question about this page, which is, uh, an easy task. We ask the agent to search for this answer and compare to what the agent found, compare to the, to the page that has been used for-- to generate the question. We can actually use code to grade that. And no longer opinions, no longer"
  },
  {
    "startMs": 381080,
    "text": "anything. We can automate this task, and that allow us to go much, much faster. So for those that work in RAGs, there is plenty of implementation of this thing. I didn't invent it. I just found it very, very, uh, very smart, and it stick to me. Like why this principle cannot be applied to other tasks? So the impact of that is we went from one evaluation a week to dozens a day, and we start to see the,"
  },
  {
    "startMs": 411960,
    "text": "the accuracy or the, the, the grading from the expert going up and up and actually removing a lot of complexity help. Like, we didn't need a lot of those. And focusing on the right, uh, embedding model, the right chunking, the right, uh, alpha in the BM25 are, is the right thing to do. And we, we saw the, we saw the, the accuracy increasing over, over weeks. So I don't know why we succeeded, but I'm pretty sure the other team didn't have any test set. So they were using this kind of long"
  },
  {
    "startMs": 441920,
    "text": "loop with expert, maybe a much more expert than we had access to, but we had, uh, we had this test set."
  },
  {
    "startMs": 451040,
    "text": "And my learning from this, uh, back in 2024 is if you cannot grade your agent, uh, you cannot make it better. So that stick to me until this day, has been two years now, and I want to go back to Air Canada now, where we have a different project. So in 2026, the project is about, uh, analyzing, um, supplier contract. So you can imagine Air Canada has a lot of"
  },
  {
    "startMs": 480800,
    "text": "suppliers. We get a lot of this contract. There's experts looking at them one by one, checking each of those clauses, and they have to analyze it and provide the feedback. Like, this clause is not compliant. This clause needs to be rewritten. Like, there is a lot of work to do that. It's a very long task and very costly because if we need the, if we need some supplier to deliver right now, and we don't, then it cause a lot of delay in the, in the chain. And if we miss a"
  },
  {
    "startMs": 510680,
    "text": "clause, it's also very costly because at, at the end of the day can backfire to us. So it's a tricky problem. Um, and they came to me, uh, I mean to the, to the data science team, to build an agent that can actually do that for them. So can we automatize this task? And obviously, there is a lot of documents, a lot of rules, a lot of things to consider when you do that. Uh, experts are super busy. You can imagine the price of, of a lawyer"
  },
  {
    "startMs": 540740,
    "text": "by hour, and there is no benchmark for that. We cannot go on the web and find documents, uh, uh, find an existing benchmark we can apply. So go back to the same problem as I was in 2024. Uh, cycle were super slow, and we knew that we want to be there in a s-- in a, in a, in a shape where we can go fast and try and try and try. Um, and I knew that if we, if we learn to this"
  },
  {
    "startMs": 570440,
    "text": "situation, we can have a better trajectory in term of accuracy, and that was the challenge for this, uh, for this project. So the, the problem is creating the answer from documents is a well-known and defined things in the RAGs, but for this specific project, there is no guy in the world that say, \"Hey, you should use this technique. You should use this technique.\" So we try to think how we can reapply the same mechanism from what I learned on RAG to this specific project, and we have"
  },
  {
    "startMs": 600240,
    "text": "plenty of project at Air Canada. So we're using the same concept of getting the answer first and then try the agent to find it back. So going into the details, uh, it's a bit simplified, but actually the, the agent has four steps. Uh, we need to spot the clause in the contract. We need to find the relevant documents to support an amendment. Uh, we need to write the, the amendment. And then at the end, there is a classification system that tells you should we keep"
  },
  {
    "startMs": 630240,
    "text": "it or not, uh, this amendment. So the first step, for example, spot is classifying all the section of the contract into specific domain, domains, and then find if there's a defect or there's a change to be, to be done in this section. So we did have this trick from the RAG, so we know how to find the document. So this is covered. Uh, but for the other, other section of the, of the agent, we didn't have anything."
  },
  {
    "startMs": 659400,
    "text": "So I'm gonna present five different techniques that we use in this project, or we try to use in this project that is trying to address this specific problem. My goal is not to show off. My goal is to get you inspired in how to reapply those techniques in your own, uh, challenges."
  },
  {
    "startMs": 680340,
    "text": "So the first one is to use past decision and answer. So we, we-- you can retrieve past contract, compare with the one that has been, uh, changed. The second one is to choose something you're looking for, write the contract around or generate the contract. The third one is, uh, take a good contract that has been signed and break it. Um, the fourth one is about things that should not change. No matter what you do, you should not get more flags. And"
  },
  {
    "startMs": 710380,
    "text": "the fifth one is actually the most powerful. It's very hard, is to turn an example into many using the concept of templates."
  },
  {
    "startMs": 720340,
    "text": "So immediately, we jump into this approach. Uh, we ask the expert to provide us the original contract, the one that has been signed We're gonna take that, do a diff like you do in the code, compare what has been changed, and the agent should, should provide the same exact flags. That would be perfect. However, for many reasons, our, our team do not keep all the contract. Um, they don't keep the original one, they just keep the last one. They don't have all the versioning of the"
  },
  {
    "startMs": 750400,
    "text": "contracts. It sometimes goes into negotiation. There is a lot of things. So we could not have any of those pairs of contracts, so this thing doesn't work."
  },
  {
    "startMs": 762200,
    "text": "Okay, second one, we took a rule. So that's changing a bit the topic. Uh, instead of creating, um, uh, flags or creating an agent that can provide flags, we start from rules. And the rule is, for example, we have the, the cap above one million dollars. So that's every contract at Air Canada should follow this rule or ma-many of those in specific domains. So you know the rule. You create, uh, a clause"
  },
  {
    "startMs": 792140,
    "text": "without, uh, breaking this rule and generate the contract around, and you should find, um, the rule back-- the, the flag, which is, uh, breaking the rule. So that's simple. It doesn't require a lot of, uh, of, of work. A lot of things can be generated. The problem is you need to know the rule. So lucky enough, we could actually get some of those from either the expert or the documentation they had, uh, but cannot guarantee the coverage of everything. So we came up, we also with our own rules,"
  },
  {
    "startMs": 823100,
    "text": "et cetera. So we generate at the end, uh, two hundred seventy-five tests that we know can run, and we know the answer. So similar to the RAG, we invested-- we invented the problem, we planted the problem, and we tried to find it back."
  },
  {
    "startMs": 839060,
    "text": "Another one, which is very hard actually, uh, we took an example of a contract that has been signed. We remove a clause, and the agent is supposed to say, \"Hey, there's something missing here.\" Um, it's a very hard task because some supplier has different templates, have different kind of content. Knowing what should be in what contract require a b- a bit of thinking. Uh, and we mix those with existing contract that should not show any"
  },
  {
    "startMs": 868980,
    "text": "flags. So that's the most powerful approach with that we have today. Um, using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing, uh, most of the clause that has been removed. So that's a very, very strong benchmark."
  },
  {
    "startMs": 886280,
    "text": "Okay, uh, fourth technique. Um,"
  },
  {
    "startMs": 891720,
    "text": "you don't change-- You change something that should not affect the agent or shouldn't-- should not change anything in the result. For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags. Um, you can reorder the clause, so some randomness should also lead to the same results. Uh, and you can run the same, uh, contract again and again. All of those are things that we check to make sure that the, the output is"
  },
  {
    "startMs": 921420,
    "text": "consistent over all of those. In this example too, everything can be checked using code. You don't need to do any LLM-as-a-judge. The previous speaker, I'm sure, like, it's a very hard task. So we stay away from, uh, LLM-as-a-judge as much as possible. The fifth one, um, it's the concept of template. Our expert have different kind of templates with blank. So I don't know if you ever write a contract. Those blanks should be filled"
  },
  {
    "startMs": 951320,
    "text": "by the expert. And instead of trying to fill them, we generate, um, an array for each of those blanks to be filled with. So that allow us to generate plenty of contracts. Some tuples break some rules, some tuple doesn't. We know that. And so we know what actually should be flagged as a, as a, as a breach or not. Again, this is a code check. We don't need to have any LLM involved. Um, what, what-- The good"
  },
  {
    "startMs": 981280,
    "text": "thing with this one is our expert already used templates, so we can reuse them to generate some contracts."
  },
  {
    "startMs": 989240,
    "text": "So the cycle was a bit slower in this case. So four weeks to get a, uh, a feedback, and we can get a script to run and get the, the evaluation back in five minutes. So I just want to highlight one point, and we went through all of those, um, kind of categories of grading at the, at the bottom. Exact match or code are very fast. As soon as we step into LLM-as-a-judge, uh, it's can be very fancy, it can be very useful. It's so hard. Like, we got"
  },
  {
    "startMs": 1019280,
    "text": "so much noise from the LLM-as-a-judge compared to the noise with the, with the agent that we're building. Solving this problem is actually almost as hard as solving the, the, the right problem. So finding the right prompt or the right harness for the LLM-as-a-judge is a very hard task. There is some padding you can do, task, uh, you can do, uh, fact-checking, you can do a bunch of stuff, but actually it requires so much work, and we could not find the right one for this project."
  },
  {
    "startMs": 1049480,
    "text": "There is plenty of other techniques. I won't go over all of them. For example, trajectory. We have another project that require the agent to hit some endpoints. So trajectory, you can, um... You, you know what the endpoint should be in some order. You generate a story about it. You ask the LLM to fill, uh, to use the agent and look at the trace to, to see if the, the trajectory is correct. Text-to-SQL, you might already know it. Doesn't matter what your agent is, if you know how to pick the answer, if you"
  },
  {
    "startMs": 1079500,
    "text": "know what you're looking for, and I showed a couple of examples, um, generate the input and then try, try to find it. It doesn't apply to everything, and sometime we had to re-engineer our solution to fit this paradigm. Even if we move away slightly from the original design of the solution, we could, uh, with the original design, we could not evaluate it. We switch to something that we can actually evaluate because of that."
  },
  {
    "startMs": 1107960,
    "text": "Um, one thing to highlight is not, uh, at the end of the day, you generate a test, you get a better agent, you get some real feedback, so expert can provide some feedback, and we don't use this feedback to fix the agent. We use this feedback to generate better test, better prompt, better ideas how to remove clauses, et cetera, et cetera, and one loop at a time. Uh, uh, that can be done by AI Rise. There is a lot of tool there."
  },
  {
    "startMs": 1138080,
    "text": "Thanks for the invitation. Um, and that's my conclusion slide. Write the answer first. Um, I think I, I explained that. Um, when you generate those tests, it's not enough to know if you can ship the product. It's just a starting point. It's a tool to help and support the developer. It's not the go/no-go for the, for the solution. So I hope you like this, um, this presentation. I know"
  },
  {
    "startMs": 1168100,
    "text": "that the twenty-four hour ship is the hardest one if you want to quit, uh, uh, releasing agent without evaluation. And, um, I would be happy to support you and have your own feedback. Thank you very much."
  }
]
