AI Engineer World's Fair 2024

Llama 3 at 1000 tok/s on the SambaNova AI Platform

Rachelle MatternPetro Junior Milan· Principal AI Engineer, SambaNova SystemsVarun Badrinath Krishna· Senior Principal AI Solutions Engineer, SambaNova Systems1:00:58

Loading slides…

Read the talk

Llama 3 at 1000 tok/s on the SambaNova AI Platform

Fast generation is only one part of an AI application: this workshop connects SambaNova’s expert-model architecture to Python inference, prompt formatting, and retrieval over a technical PDF.

From a talk by Rachelle Mattern, Petro Junior Milan and Varun Badrinath Krishna

Before you start: Basic Python, virtual environments, and API credentials are useful for following the notebook exercises; the article introduces the retrieval pipeline as it is built.

What does fast inference let you build?

What does it take to serve Llama 3 at more than 1,000 tokens per second—and then use that endpoint in an application? That is the starting question for this workshop. Rachelle Mattern, SambaNova’s director of solutions engineering, introduces the platform; principal AI engineer Petro Milan leads the practical exercises, with senior principal AI solutions engineer Varun Krishna supporting the session. The sequence moves from hardware and model orchestration to a live Python build.

Agenda slide listing prerequisites and Discord, SambaNova and 1000 tokens/sec, workshop documentation, live build and support, and Q&A.Open full source frame
Workshop agenda: SambaNova performance, project setup, live build, and Q&A.

The hands-on work assumes a laptop with internet access, a Python environment, and permission to install packages with pip. During the workshop, Discord supplies files, endpoint configuration, and API keys. Those ingredients support two increasingly useful steps: making an inference request and building an application around it.

0:170:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Integrating the stack around enterprise models

SambaNova was founded in 2017 out of Stanford. Mattern introduces founders Kunle, Chris, and Rodrigo through their experience in hardware, acquired startups, and AI companies including Snorkel and Together AI. The company builds its own chip, systems, and software; at the time of the workshop, it is on its fourth chip generation, supporting pretraining, fine-tuning, and inference. Mattern reports more than $1 billion in funding, naming BlackRock, Google Ventures, Intel, and GIC among its backers.

The intended customers include enterprises and governments pursuing differentiated applications and sovereign AI. Mattern describes a platform targeting trillion-plus-parameter scale, but the integration problem is more familiar: a team must otherwise decide which chip works with which compute system, operating system, and models. SambaNova’s full-stack approach packages those decisions together. It does not require competing with every tool at every layer; the purpose is to make the layers work together without each customer assembling the entire stack.

Diagram grouping applications, foundation models, model hubs, cloud platforms, and compute hardware beside SambaNova Suite.Open full source frame
SambaNova's chips-to-models view of the AI stack.

Samba-1 Turbo is the public demonstration of that stack’s inference performance. Mattern attributes the cross-provider Llama 3 speed comparisons to Artificial Analysis and presents the service as achieving record-setting throughput around 1,000 tokens per second. The presentation does not specify the prompt lengths, output lengths, or concurrency behind that comparison; the later live demonstration gives a more concrete view of a single interaction.

4:244:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:24 · section reference included

One application interface, many specialized models

Large general-purpose services from OpenAI, Gemini, and Claude offer broad capabilities and straightforward integration. For an enterprise, however, much of the useful knowledge lives outside their training data: proprietary records accumulated in internal systems and data lakes. Mattern describes the resulting concerns as adaptation, privacy, model ownership, and costs that grow with per-token usage. Smaller models such as Llama 3 8B can be pretrained or fine-tuned for specific tasks, but maintaining many specialists creates its own orchestration and training costs.

ApproachUseful propertyOperational pressure
General-purpose serviceBroad knowledge; easy integrationProprietary-data adaptation, ownership, usage cost
Specialized modelsTask adaptation; control over models and dataMany models to train, serve, and coordinate
Samba-1 compositionSpecialists behind a shared interfaceRouting, updates, and permissions must work together

Samba-1 treats the collection of experts as a managed service. Its example specialists cover legal, HR, and coding tasks, with different applications and groups consuming different models.

Design Philosophy slide with a large monolithic cube, a central Samba-1 cube, and a cluster labeled Mixture of Smaller Models.Open full source frame
Samba-1's composition of experts blends two model paradigms.

Four controls make that shared interface useful:

  • Secure endpoint: Applications integrate with one endpoint to reach the available experts.
  • Prompt routing: A router selects the expert best suited to an incoming prompt.
  • Independent update schedules: Experts can be fine-tuned and replaced at different cadences. Mattern’s examples are annual finance-policy updates and quarterly coding-model updates.
  • Model-level RBAC: Applications, individuals, and groups receive access to designated sets of models.

The endpoint simplifies integration, while routing, lifecycle management, and authorization preserve the distinctions between the models behind it.

8:338:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:33 · section reference included

Choosing experts and choosing how to call them

The product offers a flexible composition, where the organization chooses its models, and a pre-composed Samba-1 catalog. Mattern describes the historical catalog as containing 92 expert models spanning languages, domains, and tasks. She identifies seven foundation architectures, naming Llama 2, Llama 3, Mistral, Falcon, and BLOOM, then also mentions multimodal capabilities from LLaVA, CLIP, and DePlot. Here, an expert means a separate model with useful capabilities, rather than an internal layer of a single neural network.

Mattern says SambaNova developed or co-developed 12 of those 92 experts. The examples include Thai, Japanese, and Hungarian language adaptation, a text-to-SQL model, and BLOOMChat, which she describes at the time as the second-largest open-source model. The supplied composition is a starting point: organizations can add the expert models their own use cases require.

Evaluation follows the enterprise tasks the collection is supposed to support: information extraction, text-to-SQL, coding, and function calling. Mattern reports that SambaNova meets or exceeds GPT-3.5 Turbo and GPT-4 on its presented enterprise benchmark; the session does not supply the detailed scores or evaluation configuration. Invocation is similarly task-dependent: use automatic routing when the system should choose an expert, call a model directly when the application already knows which one it needs, or chain models for a multistep agentic workflow.

14:3814:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:38 · section reference included

Keeping experts in a three-tier memory hierarchy

The hardware behind the composition is the SN40L Reconfigurable Dataflow Unit, or RDU. Its memory hierarchy has three tiers: on-chip SRAM, high-bandwidth memory, and higher-capacity DDR. The mechanism is to keep a large collection of models in memory and move the models needed for execution toward the faster tiers. Tightly connected tiers make model placement and movement part of the execution strategy.

Mattern quotes 4 GB of SRAM, 512 GB of high-bandwidth memory, and up to 6 TB of DDR. These figures should be read as a deployment-level description, not a consistent per-chip specification: the original SN40L paper specifies 520 MiB SRAM, 64 GiB HBM, and up to 1.5 TiB DDR per socket, with eight sockets per node. The spoken SRAM and HBM figures resemble node aggregates; the exact 6 TB configuration is not established.

The practical comparison is hosting a large model collection. Mattern uses 500 models as an illustration: a GPU deployment may distribute them across systems and direct requests to the appropriate system, while SambaNova aims to retain the collection within one underlying system using its DDR tier. She claims storage for up to five trillion parameters, but the talk does not establish the precision and deployment configuration needed to interpret that capacity. The useful architectural point is the separation between retaining many experts and bringing an active expert close to compute.

18:3718:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:37 · section reference included

A workout plan, then responses while typing

Milan opens Samba-1 Turbo from SambaNova’s website. The model menu includes Llama 3 8B and Llama 3 70B, the composition of experts, Mistral, and SambaLingo. He selects Llama 3 8B and asks it to create a three-day-a-week workout schedule for an intermediate fitness level, then repeats the request to show how quickly the answer appears.

For the Llama 3 8B workout-plan demonstration, Milan reports 1,000 tokens/s, 0.09 seconds to first token, and 0.65 seconds total inference time.

These describe different parts of the interaction: generation rate, the delay before output begins, and the time needed for the response as a whole. He then enters the same question into ChatGPT for a qualitative comparison of how quickly the text appears; no matched model configuration or numerical ChatGPT result is supplied.

Next, he enables a real-time mode in which the response changes as the prompt changes. An email-drafting prompt becomes the example: instead of submitting a finished instruction and waiting, the user sees a response while still shaping the request. Low latency enables a different editing loop, useful for drafting and prompt experimentation. When an attendee asks about generation parameters, Milan checks the interface and finds that its settings appear fixed; the Python API exercise provides those controls.

21:3221:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:32 · section reference included

Set up the Python inference notebook

The first exercise loads environment variables, initializes an LLM, and makes a Python inference call. The second builds document Q&A using LangChain, data loaders, E5-large-v2 embeddings, ChromaDB, and the Llama 3 endpoint. Both begin in the SambaNova AI Starter Kit, under the historical navigation Workshops → AI Engineer 2024 → Basic Examples. The current repository has evolved and those historical workshop links are no longer available; the following walkthrough retains the recording’s SambaStudio interface and setup sequence.

The setup order matters because the notebook needs both the right Python packages and the right endpoint configuration:

  1. Clone the starter-kit repository and create .env at its root.
  2. Populate it with the SambaStudio connection information. The workshop distributes two dedicated endpoints through Discord Events.
  3. Enter the basic-examples directory and create a Conda or Python environment. Milan recommends Python 3.10 and names his Conda environment basic_x.
  4. Activate the environment, install the exercise’s requirements with pip install -r requirements.txt, and register its notebook kernel.
  5. Open the SambaStudio example notebook through Jupyter or VS Code, selecting that kernel. Milan uses VS Code so execution timing remains visible.

Inside the notebook, imports come first, then .env loading, then initialization through LangChain’s SambaStudio wrapper. The configuration includes the API key and generation settings. do_sample=False selects nonsampling generation, described in the workshop as deterministic; True enables sampling. Temperature controls sampling behavior, and the generated-token limit bounds the output. The selected expert is Meta-Llama-3-8B-Instruct. A shared CoE endpoint can address different models through expert selection, provided those models are enabled.

26:1426:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:14 · section reference included

Give the model the conversation format it expects

The first call is deliberately simple:

python

answer = llm.invoke("What is the capital of France?")
print(answer)

The model supplies the capital but also continues with material the user did not request. Milan then adds the Llama 3 instruction format, pointing to the Meta Llama 3 8B Instruct model card. The prompt must identify the beginning of the text, the user’s turn, and the position where the assistant should begin answering.

For the workshop’s raw-string invocation, the same question can be framed with the model’s special tokens:

python

question = "What is the capital of France?"
prompt = (
    "<|begin_of_text|>"
    "<|start_header_id|>user<|end_header_id|>\n\n"
    f"{question}<|eot_id|>"
    "<|start_header_id|>assistant<|end_header_id|>\n\n"
)
answer = llm.invoke(prompt)
print(answer)

<|eot_id|> closes the user turn, and the final assistant header cues the next response. In the demonstration, adding the format produces the intended answer. Prompt formatting is part of the model interface, not merely a stylistic instruction to be concise.

LangChain Expression Language, or LCEL, makes the same sequence reusable: prompt template → LLM → output parser. The country becomes an input rather than a string edited inside the prompt:

python

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate

prompt = PromptTemplate.from_template(
    "<|begin_of_text|>"
    "<|start_header_id|>user<|end_header_id|>\n\n"
    "What is the capital of {country}?<|eot_id|>"
    "<|start_header_id|>assistant<|end_header_id|>\n\n"
)
chain = prompt | llm | StrOutputParser()
print(chain.invoke({"country": "France"}))

The pipe composition preserves the model-specific formatting while allowing the caller to supply country at invocation time.

VS Code notebook showing Python cells, capital-city responses, a project explorer, and an integrated terminal.Open full source frame
The example notebook open in VS Code with executed cells.
32:5234:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

34:40 · section reference included

Separate configuration failures from output delays

During practice, an attendee gets an empty string. Milan checks the endpoint and asks to inspect .env, then suggests the alternate E4 endpoint. The concrete problem identified is stale configuration in GitHub: participants need the workshop values distributed through Discord. An attendee subsequently confirms a working result. Milan also clarifies that only Llama 3 is activated on the workshop CoE endpoint; changing an expert identifier cannot make an unavailable model callable.

Asked about other speed benchmarks, Milan estimates an 8–14× speedup over GPUs and claims 16 SambaNova chips versus 576 Groq chips. He also contrasts SambaNova’s full precision with reduced precision on Groq. The contemporary SambaNova precision article describes its representation more specifically as mixed 16-/32-bit precision. The session does not establish a matched workload for the chip counts or speedup. The unusual immediacy of the response is nevertheless noticeable to an attendee, who says it “feels like a fake.”

A later delay prompts checks of both E2 and E4, internet connectivity, notebook restart, and authentication. Milan then distinguishes another cause: the unformatted prompt generates a longer continuation, and the notebook waits for the whole generation before printing it. He expects roughly three seconds for that untagged example. A high generation rate therefore does not guarantee an immediate visible answer when the call buffers its output. The subsequent explanation returns to the prompt template used during training and the Meta model card as the source of the required format.

36:4737:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:47 · section reference included

Build a retrieval pipeline around the endpoint

The second exercise lives under the historical Workshop → AI Engineer 2024 → EKR RAG navigation and has notebook and Streamlit versions. Retrieval-augmented generation, or RAG, supplements an off-the-shelf LLM with information beyond its training data, including current documents, without retraining the model. It can reduce hallucinations in some contexts by giving the model relevant evidence to use when answering.

The pipeline separates ingestion from answering:

  1. Load and parse documents. Choose a loader for the source format, such as PDF, text, or PowerPoint, and extract editable text.
  2. Split the text. Divide documents into smaller chunks, choosing chunk size and overlap.
  3. Embed and store chunks. E5-large-v2 maps each chunk to a numerical vector. ChromaDB stores vectors alongside content and metadata; Faiss is mentioned as an alternative.
  4. Retrieve for a question. Embed the query into the same vector space and find nearby chunk vectors using a similarity metric. An optional reranker can reorder candidates and discard irrelevant chunks.
  5. Generate with context. Give the LLM the question and selected chunks so it can produce a grounded response.

The execution boundary is explicit: third-party tools handle loading, splitting, and storage. Embeddings can run locally on a CPU or through SambaNova hardware, while the generation model runs on SambaNova. Milan proposes showing both embedding options; the recorded walkthrough proceeds with the RDU endpoint.

43:5544:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

43:55 · section reference included

Prepare document extraction and the RAG environment

Reuse the cloned repository, deactivate the previous exercise’s environment, and keep .env at the AI Starter Kit root. This application needs two sets of connection details: the SambaStudio LLM endpoint and key, plus the embedding endpoint and key. The workshop supplies both through Discord.

Document extraction adds system dependencies. On macOS, the demonstrated installation uses Tesseract for OCR and Poppler for PDF tooling:

bash

brew install tesseract
brew install poppler

Milan recommends a separate Python 3.10 virtual environment for this exercise. If python3.10 is not available, install it and update the shell’s path before creating the environment. From the exercise directory, the environment setup follows this pattern:

bash

python3.10 -m venv rag-env
source rag-env/bin/activate

Then run the exercise’s installation script, install ipykernel, and register the environment as a notebook kernel. Milan estimates about five minutes for installation with good internet access. The provided setup also includes a script for NLTK or SSL-certificate problems encountered on some laptops.

GitHub documentation showing Tesseract installation options for Ubuntu, Mac, and Windows, followed by Deploy the starter kit.Open full source frame
System dependency instructions for Tesseract and starter-kit deployment.

Milan opens the RAG notebook from the exercise’s notebooks directory in VS Code. It delegates work to document_retrieval.py and vector-database modules. Two paths distinguish the scopes: the kit directory points to EKR RAG, while the repository directory points to the AI Starter Kit root. This keeps application-specific files separate from shared repository configuration.

48:2048:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

48:20 · section reference included

Turn the SN40L paper into searchable chunks

The application configuration selects the document loader, with PyPDF/PyPDF2 and Unstructured among the options. Milan begins with PyPDF and uses the SN40L paper introduced earlier: a technical document describing the hardware, software stack, and composition of experts. The repository supplies the PDF, and users can substitute their own documents under the exercise’s Data/temp location. PyPDF loads the text into a list, and LangChain’s recursive splitter divides it into chunks.

With chunk size set to 1,200 and overlap to 240, Milan reports that the 15-page PDF produces 89 chunks. These are configuration values for the demonstrated splitter; the recording does not identify their units as tokens. Overlap retains some neighboring text across boundaries, giving retrieval a chance to preserve context that a hard cut would separate.

For RDU embeddings, Milan sets the embedding type in config.yaml to SambaStudio. He describes supported batch sizes of 1 or 32, with 32 processing that many chunks together. This is a standalone embedding endpoint, so its CoE setting is false. The resulting vectors are indexed in ChromaDB. After initially expecting about ten seconds and restarting the notebook, Milan reports four seconds to embed the document’s 89 chunks on the RDU endpoint.

54:0754:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

54:07 · section reference included

Answer from passages, then test a table

Once the QA chain is initialized, a question follows the retrieval path: embed the question, retrieve the nearest chunks, and pass the question plus that context to the LLM. The demonstration sets k=3 and does not use a reranker. Milan asks, “What is a monolithic model?” The notebook displays an answer together with retrieved source text, and he describes the response as instantaneous without reporting a measured latency.

VS Code notebook displaying a question about a monolithic model, its generated answer, and a source reference with an excerpt.Open full source frame
A notebook answer about monolithic models with retrieved source text.

The final test raises the difficulty from prose to a table. Milan returns to the paper’s table relating operational intensity to fusion level and runs a prepared question to see whether the PyPDF extraction preserves enough information to answer it. He reports success. In the paper, the fully spatially fused case is 410.4 Ops/Byte—the likely value behind the spoken answer—although the recording does not expose enough of the prepared question to reconstruct its exact wording.

For more complicated PDF tables, Milan recommends changing the loader configuration from PyPDF to Unstructured. That is the final practical adjustment: when the source’s structure is lost during extraction, a fast generation endpoint cannot restore evidence it never receives. The session closes by inviting participants to try the application and then inspect individual functions with the presenters. The completed walkthrough is a local notebook demonstration; Streamlit is offered as another interface, while the proposed CPU embedding comparison and reranker walkthrough are not shown before the recording ends.

58:1158:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

58:11 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…