AI Engineer World's Fair 2024
Making Open Models 10x faster and better for Modern Application Innovation
Loading slides…
Read the talk
Making Open Models 10x faster and better for Modern Application Innovation
Specialized models need specialized serving. Fireworks connects workload-aware inference, shared LoRA deployment, and function calling into a practical path from prototype to production.
From a talk by Lin Qiao and Dmytro (Dima) Dzhulgakov
Before you start: Familiarity with language-model inference, tokens, retrieval-augmented generation, and API calls will help you follow the serving and function-calling examples.
What does a customer-support chatbot actually need?
What should a customer-support application pay for: broad general knowledge, or expertise in its own domain? That question anchors Dmytro “Dima” Dzhulgakov’s discussion of production inference at Fireworks. Presenting in place of cofounder Lin Qiao, he describes a team drawn from Meta’s PyTorch leads and Google AI veterans, with roughly a decade of production AI experience and five years of his own work maintaining PyTorch. Open deep-learning tools provide the precedent for the team’s focus on open generative models.
An audience show of hands reveals substantial open-model usage in the room. Dima nevertheless describes the broader production market at the time as dominated by proprietary models, often with frontier capabilities. GPT-4 and Claude can handle many different tasks, but their breadth and serving infrastructure do not necessarily fit each application equally well. Batch inference over a narrow domain and a real-time voice assistant have different requirements, even when they call the same model.
The customer-support chatbot makes the mismatch concrete: it does not need knowledge of 150 Pokémon or the ability to write poetry. It needs reliable answers about the business it supports. The useful unit of optimization is the application’s task, not the model’s total range of capabilities. Paying for abilities the application never uses can make a broadly excellent model an expensive fit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeated calls amplify latency and cost
Large-model response times become especially visible in voice assistants. Agentic applications compound the problem: a system such as Devin may reason through several steps and call the model repeatedly before completing a task. Each dependent call adds another wait. Dima reports that smaller Llama or Gemma models can match or exceed quality on a narrow task while running up to 10 times faster.
For function calling, Dima cites a Berkeley benchmark and reports similar quality from fine-tuned Llama 3 at 10× the speed. The accompanying slide shows GPT-4 at 1.97 seconds and Llama 3 at 0.18 seconds, with a Berkeley Function-Calling Leaderboard credit. The recording does not specify the hardware, concurrency, request lengths, or exact leaderboard snapshot behind this comparison; those conditions matter when applying the result to another workload.
Traffic makes the economic question equally concrete. Dima’s example uses a 5,000-token prompt, 10,000 users, and 20 calls per user per day, mentioning GPT-4 and GPT-4o. The input volume is:
At GPT-4o’s May 2024 launch price of $5 per million input tokens, that is $5,000 per day for input alone; output length is unspecified. Dima’s spoken estimate of approximately $10,000 per day would amount to $300,000 per 30-day month or $3.65 million per year, rather than several million monthly. The underlying motivation remains: repeated calls at substantial traffic make smaller, customizable models economically interesting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Open weights make adaptation possible, not automatic
Open models offer two routes to domain fit: choose an existing specialization or create one. Dima points to Gemma adaptations for Indian languages, models specialized for code or medicine, and the tens of thousands of variants available on Hugging Face. Access to weights makes it possible to tune quality for the application’s particular requirements.
That flexibility brings three kinds of operational work:
- Setup and maintenance: Find GPUs, select a framework, download the model, and tune its performance. Model updates can require repeating the process.
- Workload optimization: Low latency and high throughput are different objectives. Short and long prompts also favor different settings; Dima reports multiple-fold improvements from choosing those settings effectively.
- Production reliability: Cloud GPUs need operational attention. Enterprise serving adds scaling, telemetry, and observability beyond simply getting a model to answer a request.
These are the problems Fireworks sets out to absorb for application developers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize the serving stack around a latency budget
Fireworks’ approach spans custom CUDA kernels, runtime behavior, deployment, and service orchestration. Dima describes customization across that stack for each workload’s cost and latency requirements. A RAG application with long, repeatable prompts is one example: caching can avoid repeated work, while runtime and deployment settings can raise throughput without giving up the required response time.
Dima reports that Fireworks ranked fastest under Artificial Analysis’s Long Prompt selection at the time. He also describes Fireworks as the fastest provider serving SDXL, and as the sole provider serving SD3 because Stability’s API routed to its servers. These are historical provider claims; Stability’s SD3 API announcement confirms the delivery partnership, but does not establish the separate speed ranking or universal exclusivity.
A more useful optimization target than maximum speed alone is minimum cost subject to an application’s latency constraint. Dima gives the example of an interactive application that must generate its required number of tokens in under two seconds. The output length is application-specific, so the constraint must be evaluated with that length fixed:
Dima reports that tuning the latency cutoff and settings across the stack can increase throughput multiple-fold. Higher throughput means fewer GPUs are needed to serve the same traffic, reducing cost while preserving the response-time requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Serve custom models without a GPU for every variant
The model catalog described in the talk includes Gemma, Llama, automatic speech recognition, and text-to-speech models. Dima also names Yi-Large, whose US serving on Fireworks he describes as having launched the previous week. Beyond hosting models, Fireworks produces function-calling fine-tunes and releases work combining vision and language models. Developers can bring a fine-tuning dataset collected elsewhere or build one from feedback gathered during live serving.
Training a custom variant creates another question: how should it be served economically? With LoRA, customization can be represented by an adapter associated with a base model. Dima says Fireworks can deploy thousands of LoRA variants on the same GPU. Sharing serving resources lets the platform offer per-token serverless inference without a fixed serving charge for each deployed variant, rather than requiring developers to reserve separate hardware for every customization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model becomes one part of the system
A capable model still does not constitute a complete application. Hallucinations motivate grounding through RAG or external knowledge bases. Gaps across modalities motivate chaining different model types. Applications that act on the world also need tools and external APIs. Dima uses the term compound AI system, crediting Databricks with popularizing it, for this combination of models, retrieval, and actions.
In the architecture he describes, a central function-calling model interprets and triages the user’s request. It can route work to domain-specialized models, invoke a code interpreter, or call external APIs. Function calling—initially popularized by OpenAI, as Dima notes—is the connection between the language model and those external capabilities.
Fireworks fine-tunes models specifically for that coordinating role. The demonstration uses FireFunction, the latest release of which Dima places about two weeks before the talk. Its job combines ordinary conversation with deciding when and how to request a tool operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From cloud providers to a stock chart
The demo begins with freeform conversation: the user can ask what the application can do, and it describes its capabilities. Those capabilities include querying stock quotes, plotting charts, and generating images through external tools. The interesting work is translating a loosely specified request into a sequence of concrete calls.
A request for a bar chart of the Big Three cloud providers requires several steps:
- Resolve the entities. Interpret the providers as AWS, GCP, and Azure, then connect the services to their publicly traded parent companies—for example, Azure to Microsoft.
- Retrieve the data. Call stock-price tools for the relevant companies.
- Combine the results. Pass the retrieved prices to the chart-plotting API.
Dima describes the chart appearing after this sequence runs in the background. The model supplies the interpretation and coordination; the external tools supply the stock data and render the chart.
The next request adds Oracle to the existing graph. The application must preserve the earlier chart context, resolve the new company, and regenerate the chart with the additional entry. A compact JSON representation of the requested edit could look like this:
json
{
"chart_type": "bar",
"existing_companies": ["Amazon", "Alphabet", "Microsoft"],
"company_to_add": "Oracle"
}
This represents the requested change, not retrieved prices or an executed tool result. It makes the contextual requirement explicit: the earlier companies remain part of the chart.
Then the user changes topics and requests a cat photo. Now the stock-chart context should stop influencing the task: Oracle does not belong in the generated cat image. Useful conversational memory therefore requires both retention and selective disregard. The demonstration combines FireFunction with other hosted models, including SDXL; Dima makes the application available as open-source demo code, with Fireworks Demo Apps providing a companion entry point.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect model decisions to structured interfaces
The function-calling model’s weights are available on Hugging Face, allowing developers to choose hosted Fireworks inference or local execution. The FireFunction v2 model repository distributes its weights under the Llama 3 license. Weight availability gives developers deployment choices; it should not be read as unrestricted licensing.
Fireworks also provides structured generation through JSON mode and grammar mode, which Dima compares to the capabilities discussed in the preceding Outlines talk. These modes constrain the form of generated output so that model decisions can pass through structured interfaces to the rest of the application. The architecture diagram places FireFunction in JSON/Grammar mode above two destinations: Fireworks inference—including language, audio, embedding, image, multimodal, and tuned models—and external tools and APIs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From playground to dedicated serving
The starting point is the Fireworks model catalog and playground. From there, the deployment options described in the talk progressively expose more control over serving:
| Offering | Billing and infrastructure | Customization |
|---|---|---|
| Serverless | Per-token pricing; no hardware setup | Hosted fine-tunes and imported LoRA adapters |
| On-demand | Dedicated hardware | More workload settings and custom models |
| Enterprise | Discounted long-term contracts | Personalized hardware and performance tuning |
Serverless keeps the first deployment simple without excluding fine-tuning. On-demand provides dedicated resources as workload-specific control becomes useful. At larger volumes, the enterprise offering adds help tuning the hardware and serving configuration. These are the historical product choices described in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production workloads and existing clients
Dima reports that Fireworks was serving more than 150 billion tokens per day as of his last check. He names Quora’s Poe as a chatbot example, Sourcegraph and Cursor as users of some code-assistant functionality where latency matters, and Upstage and Liner as builders of assistants and agents. These examples connect the earlier serving requirements to applications already handling production traffic.
For developer adoption, Dima cites the previous year’s LangChain State of AI material, describing Fireworks as a leading source for models after Hugging Face. His suggested entry point remains practical: try Llama or Gemma in the playground, then build with FireFunction or a fine-tune suited to the application.
The final integration point is OpenAI API compatibility. Dima describes keeping familiar clients and using frameworks such as LangChain or LlamaIndex when moving to Fireworks. That preserves the application’s existing integration approach while opening up choices over model specialization and serving configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Model weights and loading examples for the Llama 3-based function-calling model, distributed under the Llama 3 license.
Further reading
Historical release explaining the Llama 3 fine-tune, conversational tool use, benchmark results and serving measurements.
Collection of Fireworks demonstrations with an entry point to the Functional chat application.
Stability's announcement of SD3 and SD3 Turbo API delivery in partnership with Fireworks.
Full transcript
Loading transcript…