# Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean

Rohan Pandey, DigitalOcean · Balaji Varadarajan, DigitalOcean

[Talk page](https://ai.engineer/talks/cWrYEBlWpcY-stop-shipping-llm-settings-that-only-win) · [Video](https://www.youtube.com/watch?v=cWrYEBlWpcY)

## Stop Shipping LLM Settings That Only Win on Paper

DigitalOcean’s tuning agent tests vLLM recipes on inference hardware and measures how many tokens the server can deliver within its latency targets. Two completed studies show what a winning configuration changed and why the metric matters.

### How a measured trial changes the next recipe

[0:01](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=1s)

Downloading a model gives an engineer a starting recipe for serving it. Finding settings that work well on a particular machine can take hours of trials. In Rohan Pandey’s demo, DigitalOcean moves that search into the background. An engineer supplies the starting configuration, and the agent takes over the work of changing settings and measuring what happens.

The agent uses Optuna to run a Bayesian optimization study. Each trial pairs a configuration with a measured score. Optuna uses those observations to choose which combination of settings to try next, so the agent can spend its trial budget on promising recipes without testing every combination. When the budget runs out, the winner is the highest-scoring configuration the study tried.

DigitalOcean’s description of the production setup follows a proposal onto its AMD MI355 inference servers. The agent rolls out a candidate vLLM and Kubernetes configuration, measures throughput against latency targets, then keeps the winner or restores the previous setup. Pandey calls the scoring measure *goodput*. A setting can increase total tokens per second yet do little for goodput if more requests miss their latency target.

### Four workloads define the test

[2:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=128s)

The demo opens on a Droplet with a **GPT-OSS** model already loaded. What traffic will exercise it? The starting recipe defines four workloads — short chat, long chat, retrieval-augmented generation (RAG) and batch classification. Each has an input sequence length (ISL) and an output sequence length (OSL), both measured in tokens. Read the pair together to see how much text the server must process for a request and how much it must generate in reply.

Those lengths are adjustable. A short prompt followed by a long reply gives the server a different mix of work from a long input followed by a short classification. The starting YAML thus fixes two things for the experiment. It gives the tuner an initial serving configuration and describes the traffic under which a change will be judged.

### Launching a study with a trial budget

[2:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=173s)

What does the agent need to turn that YAML into a run? With the weights on the Droplet, the launch command points to their location, activates a virtual environment and runs `tuner.py`. The script takes a study name, the starting YAML file, the hardware it will run on and the number of trials to perform. The example requests 24 trials. That last argument tells the search when to stop.

These runs take long enough that Pandey switches to completed studies to show the results. He recalls roughly four or five hours for GPT-OSS and a couple of hours for the other model. Each completed report comes from a run with its own trial budget.

### GPT-OSS gained 17.5% over its starting recipe

[3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s)

The completed GPT-OSS study ran 50 trials. Its HTML report identifies the best trial in that field. The companion YAML records the serving parameters for the selected configuration, so an engineer can use the winning recipe without reconstructing it from the trial history.

Pandey points out one selected value, `256`, without naming its YAML key. The comparison applies to the recipe as a whole. Measured by generated tokens that counted toward goodput, the winning configuration improved on the first configuration by 17.5%.

![Serving parameters and goodput tables for short chat, long chat, retrieval and steady batch traffic.](https://ai.engineer/api/talk-essay-media/7a4dc4498e8e4e826367648cb7c9f7e82959cd53595ace71dd5b5f7c9986dba7)

The tuning report breaks goodput down by traffic type, so an aggregate improvement can be checked against each workload.

### DeepSeek’s ninth trial improved throughput and token timing

[4:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=289s)

The second report covers a **DeepSeek R1 Distill** model. The ninth of nine trials scored best. What changed for the server and for someone reading a streamed response?

- **Token throughput doubled.** The server generated twice as many tokens per unit of time in the reported comparison.
- **Inter-token latency fell by almost half.** Once generation was underway, the wait between successive tokens was roughly half as long.

At the end of either study, the HTML report shows how each trial went and the YAML gives an engineer the settings to run again. That distinction matters for a close result. DigitalOcean’s recording description gives an example from its four studies in which an apparent 4% gain turned out to be run-to-run noise. The same description identifies the team’s work in progress: extending the agent to disaggregated serving, where prefill processes the prompt and decode generates successive tokens on separate serving resources.

![The highlighted results compare the baseline inter-token latency with the fastest of ten trials.](https://ai.engineer/api/talk-essay-media/7ee38886bbd4cab143f1d32e733ee4f7c16dfec4a2cfc231714c989416f2dc81)

The per-trial report puts latency beside throughput to show what the ninth trial changed.

## Key takeaways

- Read a winning recipe alongside the workload that selected it. Input and output token lengths determine the mix of work per request, and goodput counts the throughput that meets the latency targets.
- Keep the metric attached to the gain. GPT-OSS improved goodput by 17.5% over its first configuration in a 50-trial study. DeepSeek’s best trial doubled token throughput and cut the delay between successive tokens by almost half.

## Slides

### Configuration optimizer

[3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) [Configuration dashboard with an objective of 1,586 weighted tokens per second and the serving settings from trial 47.](https://assets.ai.engineer/sites/aiecode2025/sha256/f58876226495195942c7f61b9d176e512ba182c079e0cf4ca5ea6e3ed6d4e005.webp)

Configuration Optimizer. Best objective: 1,586 weighted tok/s under SLO. Improvement: +17.5% vs first config. Best trial #47 of 50. Four traffic types jointly optimized.

### Goodput for each traffic type

[4:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=270s) [Serving parameters and goodput tables for short chat, long chat, retrieval and steady batch traffic.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a4dc4498e8e4e826367648cb7c9f7e82959cd53595ace71dd5b5f7c9986dba7.webp)

Optimized serving config (deployable). Goodput per traffic type — best config. Traffic type, goodput (tok/s), concurrency, p99 TTFT, p99 TPOT, SLO (TTFT/TPOT). chat_short, chat_long, rag, batch_steady.

### Latency and throughput in a tuning report

[4:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=288s) [The highlighted results compare the baseline inter-token latency with the fastest of ten trials.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ee38886bbd4cab143f1d32e733ee4f7c16dfec4a2cfc231714c989416f2dc81.webp)

vLLM tuning report. Baseline ITL p50: 76 ms. Best ITL p50: 37 ms. ITL improvement: 51%. Throughput: 2.0x. Best trial #9.

### Which tuning settings changed?

[4:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=296s) [Tuning table highlights settings changed from the baseline, marks reverted runs, and shows the winning row in green.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bf73678132cfd2e5199d3500971a29956a85c4cb4d1735d7c59c49b9a46d16a.webp)

BEST. Highlighted cells = knob differs from baseline. Crashed trials failed to boot and were auto-reverted.

## Full transcript

[0:01](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=1s) Right. Hello everyone. So my name is

[0:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=3s) Rohan and this is a quick demo on the

[0:06](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=6s) optimization agent that what has been

[0:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=9s) recently built. So a quick intro into

[0:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=12s) what the purpose is and what the goal is

[0:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=14s) of this agent and then I'll get into a

[0:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=17s) brief demo and results that uh from a

[0:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=19s) previous run that we just did. So what

[0:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=22s) the goal is here is to build an agent

[0:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=24s) that helps um find the best VLM serving

[0:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=29s) config for any given model. Uh the

[0:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=32s) purpose is to reduce the time it takes

[0:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=34s) to manually um find the best configs and

[0:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=38s) the VLM serving recipe. And if we can

[0:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=40s) automate this and have it run in the

[0:42](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=42s) background and um have the agent find

[0:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=45s) the recipes and the optimized parameters

[0:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=47s) faster, then it will save a lot of time.

[0:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=50s) So that's the goal of the agent and a

[0:52](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=52s) little bit more into how the agent

[0:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=54s) actually does this. So the strategy that

[0:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=57s) the agent uses is called Beijian

[0:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=59s) optimization and bian optimization is a

[1:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=62s) technique um that's used to find the

[1:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=65s) most optimal numbers um in any given

[1:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=68s) situation. So in our situation it's to

[1:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=71s) find the optimal numbers in this uh VLM

[1:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=74s) recipe. Uh it uses mathematical steps

[1:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=76s) that essentially minimizes the search

[1:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=79s) space. So by minimizing the search space

[1:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=81s) it uh reduces a lot of the time that it

[1:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=83s) takes in finding the optimized uh

[1:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=86s) numbers in this uh recipe during the

[1:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=88s) run. So B uh basian optimization is a

[1:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=91s) strategy that is used but optuna is the

[1:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=94s) tool that uses a strategy in the search

[1:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=97s) loop. So optuna uses beijan optimization

[1:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=100s) it feeds it the current recipe the

[1:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=103s) starting recipe. uh Beijian optimization

[1:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=105s) runs on that and over many iterations or

[1:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=108s) many trials, it eventually finds the

[1:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=110s) optimized recipe and Optina basically

[1:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=113s) takes each recipe after every trial,

[1:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=115s) assesses it, gets a score, um uses a

[1:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=118s) strategy to change the recipe, improve

[2:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=120s) it and then it reruns it within the uh

[2:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=123s) search loop um until the trials is done.

[2:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=125s) So let's go a little bit into how uh we

[2:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=128s) do this. So I'm currently in a droplet

[2:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=131s) here and this droplet for example has

[2:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=133s) GPT OSS uh 12B model loaded up. So what

[2:18](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=138s) we can do is uh we can look at the

[2:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=143s) current

[2:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=144s) um recipe. So what we see here is four

[2:27](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=147s) regimes right that we're testing it on.

[2:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=149s) So there's chat short, there's chat

[2:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=151s) long, there's rag and batch classify. Uh

[2:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=154s) they have their own set of ISL OSL

[2:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=156s) tokens that we're feeding it through. Uh

[2:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=158s) all these um can be changed depending on

[2:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=160s) what uh we want to see. But essentially

[2:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=163s) this is the starting uh config starting

[2:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=165s) recipe that you would get when you

[2:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=167s) download the weights of the model. Um so

[2:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=170s) moving once you have the model

[2:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=173s) downloaded in your droplet and you're

[2:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=175s) ready to run this agent you paste in

[2:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=178s) this command uh which essentially tells

[3:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=180s) you you know where the where the weights

[3:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=182s) are coming from. um activate a virtual

[3:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=185s) um environment and have um uh or not

[3:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=189s) have but run the script called tuner.py.

[3:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=192s) Uh this script essentially tells you or

[3:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=194s) it has a couple of things you have to

[3:15](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=195s) tell it. So you you want to give it the

[3:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=197s) name of the study. You want to tell it

[3:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=199s) the starting uh YAML file which we just

[3:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=202s) saw above here. You want to tell it the

[3:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=204s) hardware that it's running on and you

[3:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=206s) want to tell it the number of trials

[3:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=208s) that you want. So uh in this example we

[3:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=210s) have 24. Um, briefly I'll show you

[3:33](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=213s) another example of a past run that was

[3:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=216s) run uh that had only nine trials. So um

[3:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=220s) uh anyway uh this is the command that's

[3:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=223s) that's run. Uh typically these commands

[3:46](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=226s) take um a couple hours to run. I think

[3:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=229s) for the GPTO it says it took about four

[3:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=231s) or five hours and then for the other

[3:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=234s) model it took a couple hours as well. So

[3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) we can go into the HTML report that's

[3:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=239s) generated after the uh agent finishes

[4:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=243s) its work. Um so this one is for the

[4:06](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=246s) GPTOSS model that I mentioned. Uh what

[4:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=248s) what we see in the HTML report is which

[4:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=250s) trial number was the best trial out of

[4:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=252s) the total of how many trials. So in this

[4:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=254s) case it was 50 trials. uh you get so at

[4:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=257s) the end of the run you get an optimized

[4:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=259s) uh YAML file which includes all the

[4:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=261s) numbers that we currently look for in

[4:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=263s) the BLM serving recipe. So in this case

[4:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=266s) we found or the agent found that 256 was

[4:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=268s) the optimized number uh so on and so

[4:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=270s) forth and the way that it's getting the

[4:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=272s) accuracy is the amount of goodput tokens

[4:35](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=275s) that's being generated. So uh overall we

[4:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=277s) found that from the first config we had

[4:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=280s) an improvement of 17.5% in terms of uh

[4:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=283s) tokens generated and this one as I

[4:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=287s) mentioned is for GPOSS. We have another

[4:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=289s) HTML report here which is for the

[4:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=291s) deepseek R1 distill uh model. In here we

[4:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=294s) found that or in here we had uh nine

[4:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=297s) trials that was run and out of the nine

[5:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=300s) the ninth one was the best. uh we found

[5:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=302s) that token throughput uh doubled and ITL

[5:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=305s) was improved by almost half aka it was

[5:07](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=307s) reduced by almost half. So this is the

[5:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=310s) quick intro and brief demo on the

[5:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=313s) commands you run and what the output

[5:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=314s) looks like in in in general you get a

[5:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=316s) HTML report that tells you um how each

[5:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=319s) trial went and then you also get um a

[5:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=322s) YAML file that has the optimized recipe

[5:25](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=325s) at the end of the day. So hopefully that

[5:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=326s) was helpful. Thank you.
