AI Engineer World's Fair 2024

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Dmytro (Dima) Dzhulgakov· Co-founder and CTO, Fireworks AI18:55

Loading slides…

Read the talk

Customizing open-model inference for production

Production inference means matching model capabilities and serving infrastructure to a workload, then connecting models to the tools and context that make an application useful.

From a talk by Dmytro (Dima) Dzhulgakov

Before you start: Familiarity with language-model inference, fine-tuning, and basic API calls will help; the article explains the roles of RAG, LoRA, and function calling as they arise.

What does your application actually need from a model?

What does it take to turn an open model into a production service tuned for a particular application? For Dmytro (Dima) Dzhulgakov, that question follows naturally from building the tools used to train models. He describes Fireworks AI’s founding team as former PyTorch leads at Meta and Google AI veterans, and his own background as five years maintaining PyTorch. The precedent is an open toolchain that developers can adapt; Fireworks applies that approach to model customization and production inference.

An audience poll finds substantial open-model use in the room, although it says nothing about the market as a whole. The choice still has a real trade-off: proprietary frontier models offer strong capabilities across many domains, making them attractive starting points for production applications.

Slide linking GPT-4 to chat, code generation, and agents, with the statement “Best quality off-the-shelf, but high costs and poor UX in production.”Open full source frame
Big models: broad capabilities, but high costs and poor production UX.

The mismatch appears when both the model and its serving infrastructure are broader than the task requires. Batch inference over a narrow domain and a real-time voice assistant can receive the same serving treatment despite needing different things. GPT-4 and Claude can handle many tasks, but a customer-support chatbot does not need Pokémon trivia or poetry-writing ability. It needs reliable answers within its support domain. Model capability and serving configuration are separate things to customize.

0:320:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:32 · section reference included

Repeated calls multiply latency and cost

Large-model response times are especially visible in voice assistants, where a delay interrupts the interaction. Agentic applications make the problem more pronounced: reasoning and tool use can require repeated model calls before the user receives a result. A delay that seems tolerable for one call can become costly across a sequence of dependent steps.

Dzhulgakov says smaller Llama or Gemma models can match or improve quality in a narrow domain while running up to 10× faster. His concrete example is a fine-tuned Llama 3 function-calling model with similar performance on a Berkeley benchmark and a claimed 10× speed advantage. The accompanying slide labels GPT-4 and Llama 3; the talk does not establish the exact benchmark snapshot, prompt and output lengths, concurrency, or timing definition. Treat this as a particular specialization claim, not a general speed ratio between the model families.

Latency chart shows GPT-4 at 1.97 seconds and Llama 3 at 0.18 seconds, with a 10× annotation and Berkeley Function-Calling Leaderboard source. Text links slow processing to low user engagement and product adoption.Open full source frame
The latency slide compares GPT-4 and Llama 3, highlighting a 10× difference.

Traffic creates a second multiplier. The illustrative workload uses 5,000-token prompts, 10,000 users, and 20 calls per user per day—one billion input tokens daily. Dzhulgakov estimates roughly $10,000 per day, or several million dollars per year, for a large-model service. That is a rough total rather than a bill derivable from the stated inputs: output length is unspecified. The economic motivation remains clear even without adopting the estimate as a price quote: unnecessary model capacity becomes expensive when purchased repeatedly at scale.

3:063:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:06 · section reference included

Open weights bring flexibility—and operating responsibilities

Domain adaptation is the stronger reason to consider open models. Language adaptations of Gemma, code models, and medical models illustrate how specialization can change what a model is useful for. Hugging Face offers a broad collection of variants, and access to weights lets developers continue tuning for their own task instead of accepting a fixed general-purpose model.

That flexibility introduces three distinct responsibilities:

  • Setup and maintenance: Find GPU capacity, choose a framework, download the model, and optimize it. New model releases can require repeating that work.
  • Workload optimization: Choose settings for the actual combination of latency, throughput, and prompt length. A configuration suitable for short interactive requests need not suit long-context processing.
  • Production operation: Handle unreliable GPUs and add the scaling, telemetry, and observability needed beyond an experiment.

A model that produces a good answer locally has solved only part of the deployment problem.

4:384:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:38 · section reference included

Tune the stack around a latency requirement

Fireworks’ approach spans custom CUDA kernels, the inference runtime, deployment, and service orchestration. The purpose of controlling those layers is to tune them together for a customer’s workload, cost target, and latency requirement. The optimization target is therefore an application configuration, not just an isolated model benchmark.

Long-prompt retrieval-augmented generation, or RAG, makes the distinction concrete. Repeated prompt content creates opportunities for caching. Runtime and deployment settings can also be adjusted to increase throughput while maintaining latency. Dzhulgakov cites Fireworks as the fastest provider under Artificial Analysis’s Long Prompt selection at the time; that historical ranking is not a promise about every prompt distribution or today’s providers.

The same serving focus extends beyond language models. Dzhulgakov describes Fireworks as the fastest SDXL provider and the sole provider serving Stable Diffusion 3, with Stability’s API routing to Fireworks servers. These are his claims about the offering at the time. Stability’s API announcement confirms the serving partnership, but does not establish exclusive hosting or a fastest-provider result.

Minimize cost subject to the application’s latency constraint. That is a more useful objective than maximizing speed without regard to expense. Dzhulgakov’s customer example asks for a specified amount of generated text in under two seconds; the token count is not supplied. Once that cutoff is known, settings across the stack can be tuned to increase the amount of work completed while still meeting it. He reports multiple-fold throughput improvements from such tuning, without a universal multiplier. For a fixed workload, higher throughput means less GPU capacity is required, which lowers serving cost.

6:386:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:38 · section reference included

Serve specialized variants without duplicating everything

The model catalog described in the talk includes Gemma, Llama, speech recognition, and text-to-speech models. Fireworks also works directly with model developers; Dzhulgakov gives Yi-Large serving in the United States as a recent example. Alongside those models, the company produces function-calling fine-tunes and vision-language model fusions. Customization can use a dataset brought to the platform or feedback collected from live serving.

Fine-tuning creates another infrastructure question: how should all the resulting variants be served? With LoRA, variants can share the base model rather than each requiring a separate full-model deployment. Dzhulgakov describes deploying thousands of LoRA variants on a shared GPU, enabling per-token serverless billing without a fixed deployment charge for each variant. The talk does not specify adapter sizes, simultaneous residency, or concurrency, so the count should not be read as a guarantee that thousands of variants can all run concurrently.

8:509:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:50 · section reference included

A coordinating model connects knowledge, models, and tools

A model alone is rarely the whole product. Grounding through RAG or external knowledge bases helps address hallucination. Different modalities can require different models chained together. Tools and external APIs let an application act beyond generating text. Dzhulgakov uses the term compound AI system, whose popularization he attributes to Databricks, for this combination of components.

In the architecture he presents, a central function-calling model interprets and triages the user’s request. It can connect to domain-specialized models, a code interpreter, or external APIs. The central model supplies coordination: deciding which capability is needed and translating a conversational requirement into a tool request. Function calling, popularized by OpenAI, is the interface between the language model and those external components. Fireworks fine-tunes models specifically for that role; Firefunction is the recently released example introduced here.

Compound AI Inference Engine diagram places FireFunction, labeled JSON/Grammar mode, around two groups: Fireworks Inference with model modalities and tuned models, and Tools and APIs with database, package, and API icons.Open full source frame
FireFunction surrounds Fireworks Inference and external tools in the compound AI architecture.
10:3310:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:33 · section reference included

From cloud providers to a stock chart—and then a cat

The demonstration combines ordinary conversation with function calling. A user can ask what the assistant can do and receive a conversational description of its capabilities. Behind that chat interface are tools for querying stock quotes, plotting charts, and generating images. The model’s job is to translate an informal request into the appropriate sequence of calls.

The first substantive request asks for a bar chart of the stocks associated with the Big Three cloud providers. Producing it requires several dependent steps:

  1. Resolve the entities. Interpret the cloud providers as AWS, GCP, and Azure, then associate those services with the companies whose stocks can be queried. Dzhulgakov explicitly calls out Azure’s ownership by Microsoft.
  2. Retrieve the data. Call the stock-query tools for the relevant companies.
  3. Construct the chart. Combine the returned information and pass it to the chart-plotting API.

The request does not supply company identifiers, prices, or chart data. Those must be resolved or retrieved before plotting. Dzhulgakov reports that this sequence has completed in the background.

The follow-up—adding Oracle to the graph—tests a different capability. The assistant must resolve which graph the user means, retain its existing context, and regenerate it with the additional company. A new request for a cat image tests the opposite behavior: the earlier stock discussion should no longer determine the tool request. The conversation can remain in history without Oracle leaking into the cat image. Useful context handling means preserving relevant state and excluding irrelevant state.

The Functional Chat demo is open source and combines Firefunction with other models, including SDXL. Dzhulgakov says the function-calling model is also available on Hugging Face, so it can run locally as well as through Fireworks. The platform supplies structured generation through JSON mode and grammar mode. These constrain the form of a generated request; the demonstrated task still requires the model to choose the right entities, tools, and conversational context.

12:5913:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:59 · section reference included

Move from the playground to dedicated infrastructure

The starting point is the Fireworks model catalog and its playground. The deployment progression presented in the talk adds control as workload requirements become clearer:

Offering in the talkBilling or commitmentDeployment control
ServerlessFixed per-token pricingNo hardware configuration
On-demandDedicated hardwareMore workload-specific settings
EnterpriseDiscounted long-term contractsPersonalized hardware and tuning

In the historical serverless offering, users could fine-tune on the platform or bring a LoRA adapter and retain per-token serving. On-demand added dedicated hardware and support for custom models brought to, or fine-tuned on, the platform. Enterprise agreements added assistance with hardware selection and performance tuning. That LoRA deployment policy has changed: documentation checked in August 2026 requires dedicated on-demand deployments for both imported and platform-trained LoRAs, rather than serverless deployment.

15:2515:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:25 · section reference included

Production workloads and familiar integration tools

Dzhulgakov reports that Fireworks was serving more than 150 billion tokens per day when he last checked. He names Quora’s Poe as a chatbot example, Sourcegraph and Cursor for code-assistant functionality, and Upstage and Liner for assistants and agents. Coding assistance makes the latency requirement particularly tangible: a response must arrive quickly enough to remain useful within the developer’s ongoing work. These are the speaker’s production examples and volume report, not audited measurements.

The closing adoption slide cites LangChain State of AI 2023, placing Fireworks after Hugging Face in the displayed open-source list. The underlying report uses anonymized LangSmith user metadata, not a market-wide survey: Fireworks led its API-access open-source-provider category and ranked sixth among LLM providers overall. That scope matters when interpreting the slide as evidence of developer adoption.

Slide states that 33,000 developers have run on Fireworks since August 2023 and cites Langchain State of AI Report 2023. A ranked list places Hugging Face first, fireworks.ai second, and ollama third.Open full source frame
The adoption slide reports 33,000 developers and lists Fireworks second under “Most Common OSS Models.”

The final integration point is OpenAI API compatibility. Dzhulgakov presents it as a way to reuse familiar clients and tools, including LangChain and LlamaIndex, while changing the model and its serving configuration underneath. The application can begin with a familiar interface and add specialized models, function calling, and workload-specific infrastructure as its requirements become concrete.

16:5017:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:50 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…