Large clusters for small models — Daniel Svonava, Superlinked

All right. I think you guys can hear me. I can certainly hear myself. Um, whoever came closer gets a T-shirt. I, I meant it. There's like a bag full of T-shirts over here. Uh, and also for questions, maybe there will be some questions at the end. If you ask a question, you'll get a T-shirt as well. And if you can guess what is on the background of this slide, you get a, uh, you get a T-shirt as well. Any guesses?

What does that visualize, this picture in the background?

No? Anybody has seen a transformer model? Spatial coding. Uh, yeah, positional encoding. Yeah. Very good. You get a T-shirt, sir. All right. So today we'll discuss, uh, basically small open source models and how they're pretty good now, and how they create unique challenges when you want to serve a bunch of them in your own cloud. Uh, everything we'll discuss is kind of open source,

uh, do-it-yourself. This is the kind of stuff you can just, uh, you know, run a command and own the stack. So there is no proprietary, uh, you know, pieces of the puzzle here. Uh, let's get this, uh, underway. Oh, this works. Okay. So small models. What do we mean by small models? Um, you know, depending who you ask, the way I think about it is basically models that you can run on two, three generations of old NVIDIA

hardware. The whole model fits into one GPU. Um, and therefore, they are easy to serve. Those GPUs are available, and they are affordable as well. Um, and then most people think, okay, small models, um, there will be some kind of trade-off in terms of, um, you know, quality of the results, and hopefully I'll be able to do a good job in this talk to convince you that actually, for specific tasks, you can be at frontier or beyond frontier

performance and get all the other obvious benefits, right? Uh, orders of magnitudes of, uh, kind of cost savings and, uh, potentially quite big, uh, latency or throughput im-improvements, of course. Um, so this is kind of one of the charts we like to show. This is the Artificial Analysis Intelligence Index over time, and what they typically don't show you is that there is like a breakdown of the open source models you should think about,

right? There is the GLM 5.2 and so on, those, uh, kind of frontier open source models with, let's say, seven hundred and fifty billion parameters. Um, but then there are the small open source models kind of trailing the big ones and trailing the frontier. You can see the frontier is kind of getting diminishing returns these days, and the small models are catching up, right? So, uh, you see this kind of convergence, saturation on top and, and kind of growth of the, of the small models. Um, and, you know,

let's say Qwen 3.6-27B, somewhere around the performance of GPT-5.1. So if you have a workflow, if you have a pipeline that, uh, you know, can run with GPT-5.1, now you can move that to a small model and, uh, you know, get all the, all the benefits we discussed. So small models, not dumb anymore. Um, now it is also about how you use the small models, right? So you can't just read that twenty-seven billion parameter Qwen 3.6 as

your kind of totally generalized, I can prompt it to do anything kind of model. No, you need to adopt the approach where, um, you basically figure out slice of tasks from the generalized model workload and then per task you figure out which model in the open source fits the task the best. You run some evals, maybe some adaptation we'll discuss, um, and then, you know, that's how you kind of reach the, the right quality, uh, to actually push this into production. So here is some example of a contract review

agent that uses, you know, nine different models. This is the kind of shape that you will see in your workloads and your agents as you move to using small models for your, um, setup. You'll start to see that, okay, instead of kind of hammering one API with bunch of different requests, uh, or one model, you would rather use a fleet of models and then your problem is, okay, how do I serve all of these different things in a way that my infra people don't go crazy, right? And this is just one of the agents that you might be running and there

might be, you know, ten of these in your company. So how do, how do we sort of, you know, that's the kind of expansion of, uh, infrastructure scope, let's say. Um, now all of those different tasks that I mentioned, there is an open source model that's, uh, sitting there waiting to be used, um, from, you know, OCR to question answering on top of documents to labeling images, generating SQL, you know, reviewing code. Uh, there are open source models fine-tuned and trained for those

tasks. Uh, you know, if you use an open source model that's trained to do OCR on receipts in Vietnamese, that project has seen the most receipts in Vietnamese, right? There's somebody who like took the time to gather as much data as possible, and on that task, that model will outperform pretty much anything else. Um, and there is, you know, hundreds of thousands of models on Hugging Face that, uh, look like that, right? So it's just, uh, it's all sitting there and it's all free, basically. Mostly quite permissive licenses. So the

models exist, you know. That- that's not the bottleneck. And, you know, we have been talking about, like, open source AI since 2024 and, uh, it's so far still not really happening, and to the extent it's happening in companies, it basically equals, like, open source AI equals AWS Bedrock. Uh, except when you look at the model catalog in Bedrock, it's, like, very, you know, restrained in model types that are available. These models are old, uh, often, you know, two, three

years behind the state of the art. Um, and when you do any kind of fine-tuning in Bedrock, you don't actually own the fine-tuned or trained artifacts, so you can't, you know, use it as an actual advantage i- in your business. It kind of stays serving from the Bedrock infra. Um, so that's on the proprietary. Now, if you do small model serving on open source infrastructure, vLLM, SGLang, different solutions, just know that these things are not tuned for any specific model or any specific hardware model

combination. You'll have to do the tuning, right? This is the do-it-yourself. All of these tools ship with guides on how to actually do the tuning, the parameter sweep, tailoring to your traffic, and so on. This is a kind of open-ended research project every time you try to adopt one of these tools. So this is not really something that sort of you, you take it and it's like an engineering project, and a week later you have a high performance serving infrastructure. It doesn't work like that. Um, and that's kind of the typical

problem with open source tools, right? It's kind of like a little bit too much do yourself. Um, and then on top of this not being kind of pre-tuned for small models, the small model workloads and traffic that uses a bunch of different models kind of flips the equation for inference, uh, kind of clusters, right? So normally when you try to serve one big model, um, your problems are how do I share that model across multiple GPUs? How do I have a router sitting on top that

understands the state of all these workers, you know, the KV cache state and so on, and then makes a top-down routing decision of, okay, this request goes to this worker or this group of workers and, and so on, right? It's very top-down set up. But if you have small and fast requests, and you have many of them, the, this, this sort of top-down routing becomes the bottleneck, right? Because the router has a little bit obsolete version of the worker state, and it's just really hard to saturate the workers if you have that kind of

upfront decision on top that's, that has to get it perfectly right in terms of, um, you know, balancing the local queues on each of these workers 'cause there is many small requests, right? Um, and you know, like, we have experimented with the vLLM and SGLang routers for small models and this sort of traffic, and it's very hard to get your GPU utilization beyond 20, 30% under constant load. And the problem is that those batches are just not correctly sized,

basically, because you have that routing bottleneck. Um, and then the third problem is that with small models, you benefit a lot from LoRAs and just model adaptation in general. And so the traffic that you have to serve, you know, contains, you know, people coming to you and saying, "Hey, I have 10 LoRAs. How do I, you know, use this with our serving stack?" Um, or, "I have this custom fine-tune I made last night, you know, um, I, I want to serve this in production." And this conversation between the AI

engineer and the infrastructure person in getting those, you know, LoRAs up there, custom models up there, that's the thing that takes time, and basically that's like the main killer in organizational, uh, velocity is talking, right? Like, ideally you would want the infrastructure engineers to do their job, and you would want to, the, those AI engineers to do their job, and they don't have to talk to operate on the day-to-day mode. So, so, you know, they, they are not blocking each other basically. Um, and this kind of model

adaptation desire around small models kind of breaks that and creates a lot of back and forth and, uh, that, that's a problem, right? Uh, so these are some challenges related to, okay, we have a bunch of small models. How do we have a cluster? How do we serve this efficiently? Um, so we have been playing with this problem for a while. Uh, I'm Daniel actually from Superlinked. I kind of skipped the intro. Um, so we are, you know, VC-backed company out of, uh, SF, and we've been building AI-powered search and document processing

systems and, and agents for the last couple of years, and our main pain point has always been inference, um, specifically these problems that I have described. And so we have iterated, and iterated, and explored different topologies for clusters for, uh, running, you know, large, wide fleets of small models in different environments because sometimes you need to deploy together with some platform in some environment where who knows what is available there. Um, you know, the small models make it

easier because in whatever environment you can get some L4s or some kind of small GPU quota is much easier. So this kind of, uh, I'll describe a little bit about the topology of the cluster that we have kind of converged to. And by the way, this whole thing is Apache 2.0, completely open source. Um, you guys can just take it and wrap it, and now you are an inference startup. Um, this is open source from kind of the control plane all the way down to the thing that runs on the GPU. Um, so

we, we didn't pull any, any punches. Uh, and the topology is basically there is a gateway, and instead of having a router that kind of pre-decides what goes where, there is a gateway that parses some of the request and attaches some metadata to the request, inserts that request into a shared queue and into some side channels. I'll go a little bit into that. And then the workers pull from tho- from that centralized queue instead of kind of pushing the data down to the workers, and this way they can saturate themselves

better. And then the worker, um, setup, I think I have a slide for that, uh, will describe how, uh, we basically absorb the complexity of different model architectures into kind of a coherent set of workers that, you know, don't have, like, competing Python requirements and stuff like that. Um, so that's kind of the overall topology. Um, and this is kind of life of a request. So maybe just, uh, I'll call out a couple of things from here. Um,

we, you know, one of the things we don't like about the OpenAI kind of API standard is the Base64 encoded kind of JSON. Not good for small models, not good for high throughput. So we use a MessagePack throughout, like a binary format. Um, this way we can also push all the multimodal data through the actual API gateway, so there is no like, "Hey, you know, binary data over here," and then request over here, and then the cluster needs access to your cloud storage to start loading some bin- some

binary data, images, or videos. We kind of encode it all and we push it through the gateway. Um, and then the gateway kind of separates some of these heavier pieces to not clog the internal queue and defers it on cloud storage kind of in flight while the request is in queue. So it kind of splits up some of these requests that are, let's say, over a megabyte, and then uses cloud storage in the backend. Um, but as a user, you push all your bits and bytes into the API layer, and it's kind of clean interface because of that.

Um, basically the whole stack is REST, so gateway REST, the, the worker is REST. And then over a socket, locally, it kind of attaches to different runtimes, and we have basically PyTorch, Candle, and SGLang on the a- as, as a runtime. Um, and, and then w- when we do the optimization, I'll kind of go into that on how we make sure that whichever runtime we are using and whichever code is running in that runtime is the most, uh, efficient one. We

have a autoresearch loop for that basically. Um, but yeah, so, so life of a request kind of looks like that. And like one tidbit is that you really wanna make sure that the gateway that's kind of the first thing that's hit by the request doesn't do too much work, 'cause then it becomes a bottleneck, right? So you don't even want to parse the whole request. You want to be able to kind of look at the packets and, uh, figure out the general shape of what's coming, do the annotation, and then you have the

workers, however many workers you have, hundreds of GPUs that, that, uh, look at the queue state and then pull from there. And, and the queue use, uh, NATS JetStream, and that thing can do, you know, million requests per second. Like th- th- that's very hard for that to become a bottleneck. Um, so yeah, like ideally you don't want to serialize, deserialize as you go through all of these different components. That's basically the kind of obvious thing. Um, this is a little animation that

shows the idea behind the centralized queuing, right? So instead of the top-down router trying to, uh, you know, fill in the local queues just right, which is basically impossible, um, you know, the, the whole idea is, hey, can we somehow centralize the queuing and can the workers, uh, rather pick up the task of forming their own batches with their own prediction of the cost of the batch, uh, and, and then, you know, become much more efficient? Now one tidbit and kind of

side note, once you kind of start working on these things, uh, you realize that it's actually really hard to predict, um, how many things to pick up from that shared queue for the batch to be really, like really the optimal size. And so you would want some mechanism that sort of allows you to put some things back into the queue if you figure out, uh-oh, like I pulled a little bit too much, and that's a network hop, right? So that's a problem. And we have special optimization for that for machines that have multiple GPUs locally, right? So there is a,

there is additional kind of machine local queuing element that takes advantage of the fact that the local processes that run on the m- multiple GPUs on one machine can kind of negotiate with the queue a little bit back and forth, which over the network, you know, there is like milliseconds extra that that would add. And so, um, we d- we don't do it over the network, only when we co-locate the, um, workers on multi-GPU machines. And you know, the, I, I mean, we are not talking about like 5% differences here, right? So like you

centralize the queue and now you get double the throughput of the cluster. So this is, this is, uh, significant. Uh, I mentioned three different runtimes. So basically it's either, you know, we write, let's say for models that are encoder-only, we write the, the PyTorch code and we kind of optimize it and we have a autoresearch loop that optimizes it. Same for Candle. We started to play with Candle not too long ago. We still can't get it to perform anywhere near the PyTorch performance, so it's a little bit more of a research

project. It's just the dependency like, you know, the worker Docker image with PyTorch is like 12 gigabytes and the worker do- basically binary, statically linked binary with Candle is, uh, maybe like 10% of that, right? And if you care about, uh, kind of waking up from the cold state and loading these images on batch of different machines, the, you know, going from 12 gigs to a gigabyte or something like this makes a, makes a huge difference. So that's kind of the motivation behind Candle. It's just

the, the getting the same performances from PyTorch is, is, is really hard. And then SGLang we have there as a kind of a go to baseline, like we should perform as at least as well as, as SGLang with the optimal tuning of all of those parameters that I mentioned that you have to do the tuning. Um, here is some numbers. So for example, when we wrap SGLang with the socket and with our kind of Rust sidecar, um, actually we can improve on the bare SGLang performance

just because we kind of, uh, do something on the batching side that natively SGLang doesn't do and probably you can make it to do that if you do like, if you develop custom plugins into SGLang and stuff like that, like probably you can match our performance because, you know, you can just push that same logic into the SGLang core server. Uh, but now you are developing custom code that only works with SGLang. And the whole lesson here from small models is that the runtimes are super diverse,

right? You don't wanna necessarily get stuck with any one particular runtime because there is, you know, we have, I think, on the order of 50 different adapters now that, that we parameterize for the different models. And so you need to somehow deal with this kind of underlying complexity, and it's probably not by building a bunch of plugins for one specific runtime. It's probably some kind of abstraction, uh, which in our case is this, uh, Rust sidecar concept and then the socket. Um,

now I'll talk about a couple different numbers, but in terms of like language around benchmarking, you know, the knee is this concept of like when you ramp up traffic on a server, uh, when you sort of request more and more throughput from it, and it gives you more and more throughput, that's when you go kind of linearly up. And then some point, you hit this point where you kind of ask for more and more is not coming, so you kind of flatten out and the latency goes up. So we call that the, the knee, and it's, it's like a useful concept in, in benchmarking, uh,

because that's kind of the point of saturation, right? That's, that's kind of the maximal performance without hurting latency. Um, so just to give you some ideas of what is possible on relatively small hardware, right? And different types of small models. So this is measured on the RTX PRO 6000. We, we kind of work with NVIDIA L4, you know, A100, RTX PRO 6000, H100, that sort of range. Um, again, those

GPUs are much more readily available, kind of on-demand in any cloud, basically. Most continents have quota, you know. Um, and on this kind of stuff, uh, you can basically get, uh, for embedding models even up to, let's say, uh, hundreds of millions of parameters. You can get hundreds of thousands of tokens per second encoded into the embedding, right? So imagine you are sitting there now, like hitting your text embedding three on OpenAI API.

Instead, you could be like having one GPU and push half a million tokens per second into the thing and get the vectors out, right? Like, is this like connecting, right? You have half a million tokens that you are pushing into a single GPU that's not even that big per second, and you are getting out vector embeddings for your search system, as opposed to like pushing all of that into a managed embeddings endpoint somewhere and paying like orders of

magnitude more money, right? And you can get latencies like, you know, low tens of milliseconds for these calls, right? Like if you use, uh, you know, Cohere, OpenAI APIs and so on, these are hundreds of milliseconds, right? And this is not rocket science. You, you know, you can have just like massive cost saving, massive latency improvements and relatively eas- easy operation, um, with, with like handful of GPUs and some, some infra around them, right?

So this like really low-hanging fruit. If you start anywhere with, uh, open source models, small models, embeddings are like no-brainer, right? Uh, but it doesn't end there. So let's say, um, you want to look at, uh, named entity recognition. You want to look at, let's say, multi-vector search, um, even generation, right, uh, of text or structured outputs and so on. Um, you, you can be getting, you know, thousands of tokens per second output from, uh,

you know, task-specific generative models as well per, uh, like, let's say half a thousand per second for, for one GPU there at the bottom. Um, and so let's say you are generating synthetic data. You are generating annotations for your fine-tuning, for your evals. You know, don't do that on a, on a managed endpoint. That's a perfect task because you have it k- kind of under control. You can survey the quality. That's a perfect task for,

uh, open source model on your own infra.

Um, and then you're like, if the infra you have around those GPUs is like reasonable, you'll get linear scaling with, with the number of those GPUs. Um, now another sort of, uh, idea if you are into small model serving, uh, is that you don't, you know, normally, um, you have kind of worker pool per model, right? You have a set of w- uh, workers, set of nodes. Uh, they have GPUs. You kind of bring those up. You preload the models. The models load

for tens of minutes because there are hundreds of billions of parameters. Uh, and so you are happy. Okay, they finally loaded. Now I have a worker pool. This mentality doesn't really work with small models. Yeah, yeah. Quickly. How, how-- What's the time left? Six minutes over. Oh, six minutes over. Okay. All right. So pack models on the same GPU is faster. Um, this is a story of how you still want to pin some models, but you want to also do, um, basically, uh, lazy

loading and eviction, uh, as a kind of function of memory pressure. You want to figure out how to combine the two. Um, there is a little bit about kind of autoresearch. We have autoresearch loops for, uh, adding support for new models and for their performance. Um, we build a lot of internal tooling to do the measurement to feed into those autoresearch loops to basically push the numbers forward. Um, and maybe perhaps most importantly, when we ship support for a model, it has all the tuning

done, right? So there is no, "Okay, let's do a parameter sweep." We bundle basically a config for end-to-end the whole cluster. Um, this is a setup for the autoresearch loop that is like a meta loop that builds the harness that then runs the loop, and there is a dashboard on top that helps you understand how it works. Um, we have custom UIs for that, and one of the outputs of that was a LoRA that took eighty cents to train, and it improved eighteen perc- It

improved quality of retrieval on German legalese text as a proof of concept by eighteen percent, and that's it. So small models are good. They are relatively easy to serve. Uh, they are actually much cheaper, faster, is as smart, and that QR code goes to the GitHub repo of RCluster that I just described. Give us a star and, uh, happy self-hosting. Thank you.