[
  {
    "startMs": 12640,
    "text": "My name's Chris Vela. Uh, I work at Runpod. I'm a forward deployed engineer. Uh, today I'm gonna be talking about, um, benchmarking vLLMs, uh, kind of like the struggle that people have when like... Especially like a new model comes out. Um, just some like learnings that we've had, like internally at Runpod and with like customers. Uh, so today I'm gonna be showing you, um, s- essentially like a dashboard that, uh, is built on top of like vLLM. Um, it's using our like pods actually. So to give you like a little more context of like what Runpod does, is, uh, we're in"
  },
  {
    "startMs": 42300,
    "text": "Neocloud. Um, we make it really, really easy for you to rent GPUs, um, starting from like R&D to like more serverless production, uh, workflows. Uh, a lot of times what I see with customers, especially with vLLM, is like someone wants, you know, the latest model, they don't know how to like spin it up, or they, they wanna start going to production and they don't know the, the knobs to like touch. They, they're just using whatever, like the base config. And I'm guessing a lot of you've probably seen like vLLM recipes before. Um, you know, it, it's a good start, uh, especially"
  },
  {
    "startMs": 72280,
    "text": "in like... But like, especially in this world, we're always moving fast. Um, so how do you get to more a production level, uh, style of code? And at the same time, not just like vibing it, it's more like, how do you actually have data? How do you show, you know, your manager or like your staff engineer that, \"Hey, this is actually better. This is actually working correctly\"? So"
  },
  {
    "startMs": 95640,
    "text": "really what I'm trying to say is like, I've seen a lot, everyone's just guessing. Like, you know, you're... A lot of times I see people are using the wrong vLLM version. They might-- They're probably not using like the nightly version. Um, they have the wrong size for their context window. Um, it, it really depends. Like, like I, I take it always a step back. It's like, what are you trying to achieve with your, your LLM model? Are you-- Is it gonna be a coding agent? Is it more kind of like you're just, um, you know, doing summaries, like very, very basic tasks? That's usually the first question I would be"
  },
  {
    "startMs": 125320,
    "text": "asking before you, uh, you know, getting Kimi K3, having a, you know, a million, uh, long context. Like, that's not always necessary. Uh, and at the same time it's like, how many users are gonna be using this? Um, a lot of times people are like going... They're-- Everyone's trying to always get like the, the smallest amount of GPU for the, the cheapest price. But if you're gonna be going to have more concurrent users, you need to start thinking about like, how am I gonna be utilizing the KV cache? Uh, so all this is... There's"
  },
  {
    "startMs": 155280,
    "text": "all these little like optimization knobs that you need to like figure out before you go to production, and a lot of times we're trying to move fast. So, um, yeah, like, like I said, like Runpod, like we like have very much like, you know, we make it very, very easy. It's like a VM, uh, it's Dockerized. Uh, you can bring your own image, but at the end of the day, like, um, like who here has used like vLLM before? Okay. So, you know, like when you're looking for like, like especially like Kimi K3, like if you, you"
  },
  {
    "startMs": 185260,
    "text": "use like a version of like, you know, 0.27, it's not gonna be working. Like the, like we're moving so fast, the code is not getting as fast. Like you need to have the right version. And a lot of times that turns into kind of like a CDI-- CI/CD process that, um, especially if people who don't understand what's happening, uh, just start struggling. They, they start seeing errors and they're like, \"Why is this not working?\" You know, it-- The, the layers within it are not... They haven't been written or they're... It's very in development code. Uh, and at the same time that's like actually kind of a problem, is like"
  },
  {
    "startMs": 215220,
    "text": "Kimi K3 comes out, you know, the next day I would not put that on production just yet. Uh, you're gonna be seeing a lot of probably hallucinations. Uh, it's just not ready in that state, I would say. Um, so what I propose, um, so, uh, I created this thing called, it's called BenchPod. It's gonna be a template. It hasn't come out yet. Um, so it's essentially built on top of like, uh, vLLM. Um, it's a whole metric system. Uh, you can benchmark very easy, and a lot of you who are familiar with like vLLM, you can"
  },
  {
    "startMs": 245220,
    "text": "benchmark already. But this kind of, um, centralizes everything so you can be able to see everything, be able to change the config very quickly. Uh, that's like a very important thing that I, I notice is that, uh, especially with people who use like our, uh, our pods, is you set up a config, but then you have to restart the image. You have to wait for cold starts to happen. It's all this loading and you're not moving as fast. You're waiting like thirty minutes, like for Kimi K3 to like download the weights, load into"
  },
  {
    "startMs": 274920,
    "text": "GPU. Uh, it's kinda wasted time. I mean, if you have an agent, you know, you really don't care, but, um, that is a pain point, especially when we're trying to move fast. And at the same time is like the cool thing about Runpod is you can find these machines. So if you have that question, it's like, \"I wanna run Kimi K3 on a B300,\" we have B300s. And you can start going from like, not just like reading about it, you can actually start testing it and seeing if, you know, there's truth to all these metrics. So yeah."
  },
  {
    "startMs": 304800,
    "text": "So like benchm- like, like I said, like BenchPod is more like, um... It's, it's built like of, of, um, a Docker image of, um, um, vLLM Open API. We're gonna be having every version already. Um, we just have a, a little like, like payload that like pretty much creates this whole FastAPI application. Uh, you could use it for your agents. Uh, the cool thing is you can serve, uh, you can pretty much specify which, uh, models you wanna have. We have a whole watchdog too, so it like self-heals too. So, uh, if you are actually starting to go to more"
  },
  {
    "startMs": 334700,
    "text": "production levels, uh, you can actually, uh, you don't have to worry about it. And like I'll show you, uh, like this past Sunday, uh, Kimi went down on our instance and it, uh, self-healed. And, you know, we have, we have right now like four nodes running it, so none of like our internal users saw any problem. It was production ready, um, didn't have to like think about it too much. And at the end of the day, yeah, like it's, it, it makes it much more sim, uh, simple process to, uh, have a"
  },
  {
    "startMs": 364580,
    "text": "production worklo- workflows. So I'm guessing a lot of people... I'm, I'm gonna be showing Kimi K3 because we've been using that a lot. Um, I'm guessing a lot of people here are very familiar with Kimi K3. Um, so, you know, one, 1 million in context. Uh, one thing that's interesting on one B300, uh, can't get to the 1 million, you gotta use half of it. But one, one thing I've noticed especially with our team, like we're using it a lot for our coding, uh, agents, um, that is very much, um, enough"
  },
  {
    "startMs": 394396,
    "text": "to do long tasks. Um, it's utilizing KV cache, uh, very well too. So, um, in terms of speed of like time to first token, um, it is very, very, very quick. And like in the demo, I'll show you too with like my coding agent running this Kimi, uh, model on a Runpod. So this is like our dashboard. Um, has a lot of metrics. I know it can, it look, looks like a lot, but uh, essentially, you know, we're always thinking about like the GPU, like, you know, have a lot of like GPU, uh,"
  },
  {
    "startMs": 424476,
    "text": "metrics, knowing what CUDA version you're on, which is very, very important. You know, if you're running Kimi, it doesn't work on a 12.8 CUDA, but it works on 13.0. Uh, and really it's... What we have is like the bench- We can, we can look at benchmarking, look at all our data. We can like play around with it. So like s- before you even put it into your like coding agent or, uh, just hitting the OpenAI, uh, response, you can start testing real quick if this is working. And just a lot of like more CI/CD processes of like, you know, how long are my cold starts"
  },
  {
    "startMs": 454256,
    "text": "happening? Um, how are, how are my metrics looking, um, through actual traffic? So I'm gonna show you guys the demo, so..."
  },
  {
    "startMs": 465296,
    "text": "So really like, I'm gonna show you real quick, like Runpod itself. So, um, this will-- So we don't have this templated up yet. Like I said, um, it'll be up probably at the end of the month. Uh, but really all we have is just like a Docker image. We can be able to, uh, put that in right there, if it loads correctly. But, uh, yeah, so right now, uh, there's not B300s, but, uh, I have one running right now.  That's the fun thing about working in this industry. Uh, you know, uh, supply can be, uh,"
  },
  {
    "startMs": 496156,
    "text": "dependent. Uh, but-  But yeah, s- In, in the sense of that, like, you know, we make it really, really easy. You can put storage volume. Uh, you deploy it, you know, uh, we have a part two, so you can deploy when you're available. So, you know, if you were seeing the screen like this, it will deploy whenever, uh, it get, it gets started. Uh, so that's like literally the process. You click that button, BenchPod like starts up. Um, and I can show you like right here."
  },
  {
    "startMs": 527096,
    "text": "I have it on the wrong one, but I'll show you guys here. So like, this is like, this is a- our actual like Kimi running currently. Um, so a cool thing is like, you know, you, you start... Like it has a whole error scan, so you have actual data to know like what is happening. Um, cool thing is we have live metrics too, as like things are running. Uh, all of this is also like backed up in like storage data, so like it's all, um, um, backed up no matter what. Um, it's all face- FastAPI calls, so your agent could easily just"
  },
  {
    "startMs": 557116,
    "text": "call all this data. You can actually just serve everything. Like you don't even have to use the UI. I, I think I'm a big proponent of like, as we go to agents, it's like I still wanna have like some awareness of what's happening, and having this gives me a good like, like eagle's-eye view of like what is happening with my, uh, Kimi K3, K3 instance. Uh, but yeah, like you can see here, like I was gonna show you, like, uh, this past weekend we had a failure with a GPU. Uh, we were able to fix it, but, uh, being able to just do a cold start time of"
  },
  {
    "startMs": 587076,
    "text": "like nine minutes on it, like K3 is pretty, pretty good. Um, and the cool thing here is that we have the serve. So you can start out experiments and essentially like apply them. It'll restart real quick. Um, you can... It's literally just like the config YAML file from like vLLM. And um, so like for this like little demo, what I tested was we were just ch- changing the max number of batch tokens, uh, that were sent through. Uh, and I kinda wanted to show you guys like, like, you know, what, what kind of like speed"
  },
  {
    "startMs": 616976,
    "text": "ups would happen with this. Um, I'll go right here. So I ran these before because we're not gonna spend, uh, like 20 minutes just looking  at, uh, benchmarks. So these benchmarks are running like concurrency, so they're, they're running, um, you know, starting with like just one concurrent user and then eight, 64, 192. Um, and the cool thing is like we can be at-- we can actually compare multiple runs against each other. Um, you can"
  },
  {
    "startMs": 646976,
    "text": "kinda see like even just increasing the, the match, the, the number of batch tokens doesn't really change that much the like time to first token. Uh, but interesting thing that I did notice of increasing that is like the inter-token latency actually decreases quite a bit when you increase that, um, uh, batch tokens. Uh, which means it's like the request itself, if it's especially running like a long, um, uh, like job essentially, uh, it's actually much"
  },
  {
    "startMs": 676536,
    "text": "faster finishing the job than the, the base like config that like you find on vLLM Recipe. Um, the cool thing is, is like you can add this to an agent and just start going crazy with it, and you can start getting those answers much, much quicker, uh, have the data, and move from this. If you have it like on a pod, you can start moving to, you know, a multi-node, uh, cluster. Uh, you can move to serverless too. So if you're gonna be like scaling up and down, um, that, that--"
  },
  {
    "startMs": 706516,
    "text": "this is a perfect way of like actually getting data. And just to give you like a context on our serverless, like serverless, a lot of people will do the same thing. They, they just put whatever the base config is and then are struggling as like they're waiting for workers to start up. Um, this actually gets you the real data. You're now ready to go to production. Um, makes it really, really simple, um, and just gives b- It's... I, I like I always say, it's basing everything in facts. And, um, I can like show you guys right here, like this is"
  },
  {
    "startMs": 736516,
    "text": "actually like Kimi running. So it's just like, what is a BenchPod? Misspelled it, sorry"
  },
  {
    "startMs": 744712,
    "text": "But that, this is little Kimi instance running through it. Um, yeah, and, um, you know, it is pretty, pretty quick. Like m- like, I would say, like, there's like probably like 50 users right now on our internal one, and we... most of us are getting off Claude and just using this. And, uh, we have other instances with like GLM, uh, DeepSeek, uh, Qwen 3.8. Um, but yeah, like, w- I think in the future, we're looking to more at like smart routers, like how, how do we host all this without being reliant"
  },
  {
    "startMs": 774672,
    "text": "on someone else? Um, yeah, so let me go back to our, uh,"
  },
  {
    "startMs": 782972,
    "text": "thing real quick. So some like important data that we were just talking about if anyone wants to screenshot that. But, uh, like I said, like, you can, you can specify the experiment yourself too. So if you were like, \"I'm gonna have like 500 people using this all the time,\" like, \"I wanna like see how well this is gonna be.\" Um, but at the end of the day, like I said, is how do we go from R&D to production? And I think this is where like BenchPod will make it a lot more easier for people to use."
  },
  {
    "startMs": 812992,
    "text": "Um, so like at the end of the day, like what I want you guys to like know is like, um, you know, pick a pod, get like a GPU machine. You have your model, compare configs. Don't just start with like the vLLM recipe. The vLLM recipe is proven, but it's not production ready. Um, and then test. Have your agent test it, and then whatever, you know, based on what you need, if it's ITL, if you need TTF, TPOT, like metric, like optimize on"
  },
  {
    "startMs": 843052,
    "text": "that, and then ship that winner. Go to serverless, scale out, and now you can like, um, now you have a working LLM model that, uh, can be used internally, can be used for your customers. Um, yeah, it's, it, it just makes it very, very simple. Um, but yeah, um, one thing I would say, especially with like Kimi that I've noticed is, um, it's not always necessary for a team. Like, like a"
  },
  {
    "startMs": 872912,
    "text": "big thing that I think like most people don't think about is that, um, you can use a smaller model. You know, it, it's a B300 right now, like a node that's... Like, on our, on our platform, like that's gonna cost you 40K a month, essentially. And it might go up, it might go down a little, but you know, it's a very powerful machine. And I think when you're thinking about like how you're gonna be, um, serving stuff if you wanna be doing this internally or for yourselves is like"
  },
  {
    "startMs": 903592,
    "text": "what makes sense the best? And just because it's the, the, the frontier model, maybe it doesn't make sense. Um, like I said, like, like I run a DeepSeek on a four H200s, and it does a really good job. Uh, Kimi is more for like long tasks that I'm trying to solve something like very, very detailed. Um, but like I said, the other thing is like making sure your like tooling tools like on there. That's a big thing. Um, context, like I, I'm not gonna be going through too much about like context, uh,"
  },
  {
    "startMs": 933512,
    "text": "like wikis and all that, but the ability to have all that, uh, and like A- AI governance is the next step that we are gonna be going with Runpod. But just something to be, always be thinking about, especially if you're gonna be self-hos- um, hosting your, um, your LLM models. But yeah, is there any questions, like... And we'll, we'll also be at the booth, so like downstairs. But yeah, add me on LinkedIn, uh, if you wanna talk more. I'm always exploring every new model like every week, so uh,"
  },
  {
    "startMs": 964072,
    "text": "always have some like, uh, gotchas that I've been learning through the, through the time I've been doing all this work. And, uh, yeah, also scan the car- uh, QR code so you can get, uh, um, you'll get some credits and, um, you can test out Runpod. And like I said, the template will be out like at the end of the month, so uh, you can go play around with it."
  }
]
