[
  {
    "startMs": 1920,
    "text": "Right. Hello everyone. So my name is"
  },
  {
    "startMs": 3919,
    "text": "Rohan and this is a quick demo on the"
  },
  {
    "startMs": 6960,
    "text": "optimization agent that what has been"
  },
  {
    "startMs": 9360,
    "text": "recently built. So a quick intro into"
  },
  {
    "startMs": 12960,
    "text": "what the purpose is and what the goal is"
  },
  {
    "startMs": 14880,
    "text": "of this agent and then I'll get into a"
  },
  {
    "startMs": 17199,
    "text": "brief demo and results that uh from a"
  },
  {
    "startMs": 19199,
    "text": "previous run that we just did. So what"
  },
  {
    "startMs": 22000,
    "text": "the goal is here is to build an agent"
  },
  {
    "startMs": 24800,
    "text": "that helps um find the best VLM serving"
  },
  {
    "startMs": 29199,
    "text": "config for any given model. Uh the"
  },
  {
    "startMs": 32239,
    "text": "purpose is to reduce the time it takes"
  },
  {
    "startMs": 34160,
    "text": "to manually um find the best configs and"
  },
  {
    "startMs": 38239,
    "text": "the VLM serving recipe. And if we can"
  },
  {
    "startMs": 40960,
    "text": "automate this and have it run in the"
  },
  {
    "startMs": 42640,
    "text": "background and um have the agent find"
  },
  {
    "startMs": 45280,
    "text": "the recipes and the optimized parameters"
  },
  {
    "startMs": 47920,
    "text": "faster, then it will save a lot of time."
  },
  {
    "startMs": 50399,
    "text": "So that's the goal of the agent and a"
  },
  {
    "startMs": 52719,
    "text": "little bit more into how the agent"
  },
  {
    "startMs": 54399,
    "text": "actually does this. So the strategy that"
  },
  {
    "startMs": 57440,
    "text": "the agent uses is called Beijian"
  },
  {
    "startMs": 59120,
    "text": "optimization and bian optimization is a"
  },
  {
    "startMs": 62320,
    "text": "technique um that's used to find the"
  },
  {
    "startMs": 65360,
    "text": "most optimal numbers um in any given"
  },
  {
    "startMs": 68799,
    "text": "situation. So in our situation it's to"
  },
  {
    "startMs": 71040,
    "text": "find the optimal numbers in this uh VLM"
  },
  {
    "startMs": 74080,
    "text": "recipe. Uh it uses mathematical steps"
  },
  {
    "startMs": 76720,
    "text": "that essentially minimizes the search"
  },
  {
    "startMs": 79280,
    "text": "space. So by minimizing the search space"
  },
  {
    "startMs": 81600,
    "text": "it uh reduces a lot of the time that it"
  },
  {
    "startMs": 83759,
    "text": "takes in finding the optimized uh"
  },
  {
    "startMs": 86560,
    "text": "numbers in this uh recipe during the"
  },
  {
    "startMs": 88720,
    "text": "run. So B uh basian optimization is a"
  },
  {
    "startMs": 91840,
    "text": "strategy that is used but optuna is the"
  },
  {
    "startMs": 94960,
    "text": "tool that uses a strategy in the search"
  },
  {
    "startMs": 97520,
    "text": "loop. So optuna uses beijan optimization"
  },
  {
    "startMs": 100799,
    "text": "it feeds it the current recipe the"
  },
  {
    "startMs": 103600,
    "text": "starting recipe. uh Beijian optimization"
  },
  {
    "startMs": 105840,
    "text": "runs on that and over many iterations or"
  },
  {
    "startMs": 108399,
    "text": "many trials, it eventually finds the"
  },
  {
    "startMs": 110720,
    "text": "optimized recipe and Optina basically"
  },
  {
    "startMs": 113280,
    "text": "takes each recipe after every trial,"
  },
  {
    "startMs": 115360,
    "text": "assesses it, gets a score, um uses a"
  },
  {
    "startMs": 118479,
    "text": "strategy to change the recipe, improve"
  },
  {
    "startMs": 120479,
    "text": "it and then it reruns it within the uh"
  },
  {
    "startMs": 123040,
    "text": "search loop um until the trials is done."
  },
  {
    "startMs": 125920,
    "text": "So let's go a little bit into how uh we"
  },
  {
    "startMs": 128959,
    "text": "do this. So I'm currently in a droplet"
  },
  {
    "startMs": 131280,
    "text": "here and this droplet for example has"
  },
  {
    "startMs": 133440,
    "text": "GPT OSS uh 12B model loaded up. So what"
  },
  {
    "startMs": 138720,
    "text": "we can do is uh we can look at the"
  },
  {
    "startMs": 143040,
    "text": "current"
  },
  {
    "startMs": 144800,
    "text": "um recipe. So what we see here is four"
  },
  {
    "startMs": 147599,
    "text": "regimes right that we're testing it on."
  },
  {
    "startMs": 149680,
    "text": "So there's chat short, there's chat"
  },
  {
    "startMs": 151840,
    "text": "long, there's rag and batch classify. Uh"
  },
  {
    "startMs": 154319,
    "text": "they have their own set of ISL OSL"
  },
  {
    "startMs": 156080,
    "text": "tokens that we're feeding it through. Uh"
  },
  {
    "startMs": 158160,
    "text": "all these um can be changed depending on"
  },
  {
    "startMs": 160560,
    "text": "what uh we want to see. But essentially"
  },
  {
    "startMs": 163280,
    "text": "this is the starting uh config starting"
  },
  {
    "startMs": 165599,
    "text": "recipe that you would get when you"
  },
  {
    "startMs": 167360,
    "text": "download the weights of the model. Um so"
  },
  {
    "startMs": 170959,
    "text": "moving once you have the model"
  },
  {
    "startMs": 173040,
    "text": "downloaded in your droplet and you're"
  },
  {
    "startMs": 175200,
    "text": "ready to run this agent you paste in"
  },
  {
    "startMs": 178319,
    "text": "this command uh which essentially tells"
  },
  {
    "startMs": 180480,
    "text": "you you know where the where the weights"
  },
  {
    "startMs": 182400,
    "text": "are coming from. um activate a virtual"
  },
  {
    "startMs": 185280,
    "text": "um environment and have um uh or not"
  },
  {
    "startMs": 189920,
    "text": "have but run the script called tuner.py."
  },
  {
    "startMs": 192080,
    "text": "Uh this script essentially tells you or"
  },
  {
    "startMs": 194000,
    "text": "it has a couple of things you have to"
  },
  {
    "startMs": 195200,
    "text": "tell it. So you you want to give it the"
  },
  {
    "startMs": 197200,
    "text": "name of the study. You want to tell it"
  },
  {
    "startMs": 199599,
    "text": "the starting uh YAML file which we just"
  },
  {
    "startMs": 202879,
    "text": "saw above here. You want to tell it the"
  },
  {
    "startMs": 204720,
    "text": "hardware that it's running on and you"
  },
  {
    "startMs": 206640,
    "text": "want to tell it the number of trials"
  },
  {
    "startMs": 208080,
    "text": "that you want. So uh in this example we"
  },
  {
    "startMs": 210640,
    "text": "have 24. Um, briefly I'll show you"
  },
  {
    "startMs": 213360,
    "text": "another example of a past run that was"
  },
  {
    "startMs": 216159,
    "text": "run uh that had only nine trials. So um"
  },
  {
    "startMs": 220799,
    "text": "uh anyway uh this is the command that's"
  },
  {
    "startMs": 223519,
    "text": "that's run. Uh typically these commands"
  },
  {
    "startMs": 226159,
    "text": "take um a couple hours to run. I think"
  },
  {
    "startMs": 229519,
    "text": "for the GPTO it says it took about four"
  },
  {
    "startMs": 231680,
    "text": "or five hours and then for the other"
  },
  {
    "startMs": 234239,
    "text": "model it took a couple hours as well. So"
  },
  {
    "startMs": 236480,
    "text": "we can go into the HTML report that's"
  },
  {
    "startMs": 239760,
    "text": "generated after the uh agent finishes"
  },
  {
    "startMs": 243920,
    "text": "its work. Um so this one is for the"
  },
  {
    "startMs": 246000,
    "text": "GPTOSS model that I mentioned. Uh what"
  },
  {
    "startMs": 248480,
    "text": "what we see in the HTML report is which"
  },
  {
    "startMs": 250560,
    "text": "trial number was the best trial out of"
  },
  {
    "startMs": 252879,
    "text": "the total of how many trials. So in this"
  },
  {
    "startMs": 254400,
    "text": "case it was 50 trials. uh you get so at"
  },
  {
    "startMs": 257120,
    "text": "the end of the run you get an optimized"
  },
  {
    "startMs": 259199,
    "text": "uh YAML file which includes all the"
  },
  {
    "startMs": 261359,
    "text": "numbers that we currently look for in"
  },
  {
    "startMs": 263440,
    "text": "the BLM serving recipe. So in this case"
  },
  {
    "startMs": 266400,
    "text": "we found or the agent found that 256 was"
  },
  {
    "startMs": 268639,
    "text": "the optimized number uh so on and so"
  },
  {
    "startMs": 270720,
    "text": "forth and the way that it's getting the"
  },
  {
    "startMs": 272560,
    "text": "accuracy is the amount of goodput tokens"
  },
  {
    "startMs": 275040,
    "text": "that's being generated. So uh overall we"
  },
  {
    "startMs": 277600,
    "text": "found that from the first config we had"
  },
  {
    "startMs": 280080,
    "text": "an improvement of 17.5% in terms of uh"
  },
  {
    "startMs": 283360,
    "text": "tokens generated and this one as I"
  },
  {
    "startMs": 287120,
    "text": "mentioned is for GPOSS. We have another"
  },
  {
    "startMs": 289199,
    "text": "HTML report here which is for the"
  },
  {
    "startMs": 291120,
    "text": "deepseek R1 distill uh model. In here we"
  },
  {
    "startMs": 294960,
    "text": "found that or in here we had uh nine"
  },
  {
    "startMs": 297919,
    "text": "trials that was run and out of the nine"
  },
  {
    "startMs": 300000,
    "text": "the ninth one was the best. uh we found"
  },
  {
    "startMs": 302160,
    "text": "that token throughput uh doubled and ITL"
  },
  {
    "startMs": 305280,
    "text": "was improved by almost half aka it was"
  },
  {
    "startMs": 307680,
    "text": "reduced by almost half. So this is the"
  },
  {
    "startMs": 310800,
    "text": "quick intro and brief demo on the"
  },
  {
    "startMs": 313039,
    "text": "commands you run and what the output"
  },
  {
    "startMs": 314720,
    "text": "looks like in in in general you get a"
  },
  {
    "startMs": 316639,
    "text": "HTML report that tells you um how each"
  },
  {
    "startMs": 319440,
    "text": "trial went and then you also get um a"
  },
  {
    "startMs": 322960,
    "text": "YAML file that has the optimized recipe"
  },
  {
    "startMs": 325120,
    "text": "at the end of the day. So hopefully that"
  },
  {
    "startMs": 326960,
    "text": "was helpful. Thank you."
  }
]
