AI Engineer World's Fair 2024
Build enterprise generative AI apps using Llama-3 at 1,000 tokens/s on the SambaNova AI platform
Loading slides…
Read the talk
Build enterprise generative AI apps using Llama-3 at 1,000 tokens/s on the SambaNova AI platform
Build from a single SambaStudio inference call to a retrieval pipeline, following the workshop’s expert routing, memory hierarchy, prompt formatting and PDF search examples.
From a talk by Varun Badrinath Krishna, Petro Junior Milan and Rachelle Mattern
Before you start: To follow the exercises, you should be comfortable with Python, virtual environments and Jupyter notebooks, and have access to SambaStudio LLM and embedding endpoints.
What does fast inference make possible?
What can an enterprise application do with Llama 3 generating more than 1,000 tokens per second? That is the starting question for this workshop: first understand the platform delivering the inference, then connect it to Python and build a document question-answering application. Rachelle Mattern, director of solutions engineering, introduces the platform; principal AI engineer Petro Milan leads the workshop, with senior principal AI solutions engineer Varun Krishna supporting the hands-on work.
The progression is deliberately practical: platform orientation, project setup, a live build and questions along the way. Following the exercises requires a laptop with internet access, a Python environment and the ability to install packages with pip. During the workshop, Discord supplies files, endpoint information and API keys.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Integrating the path from hardware to models
SambaNova approaches enterprise AI as an integrated hardware and software problem. Founded in 2017 out of Stanford, the company brings together the chip-development and startup experience of founders introduced as Kunle, Chris and Rodrigo, including connections to Snorkel and Together AI. Mattern describes a fourth-generation chip and a stack spanning the chip, system and software layers, supporting pretraining, fine-tuning and inference. She also reports more than $1 billion in funding, naming BlackRock, Google Ventures, Intel and GIC among its backers.
For enterprise and government customers, including sovereign-AI deployments, the goal is to reduce the integration decisions between those layers. Mattern describes a platform intended to support more than a trillion parameters. Rather than compete with every component in the ecosystem, SambaNova integrates choices that otherwise have to be reconciled individually: which chip works with which compute system, which operating system supports it, and which models run on that combination. The application team gets a coordinated stack instead of assembling each dependency independently.
Samba-1 Turbo makes the inference side of that proposition visible through a public demonstration. Mattern points to Artificial Analysis comparisons and describes Llama 3 throughput around 1,000 tokens per second, ahead of competing providers in the presented results. These are historical comparisons; the presentation does not give the prompt lengths, concurrency or other workload conditions needed to reproduce the ranking. The live application later supplies a more concrete view of what the speed feels like.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combining broad capabilities with enterprise specialists
The model-management problem begins with two different ways of adopting generative AI. Broad models from OpenAI and model families such as Gemini and Claude offer general knowledge and straightforward integration. But an enterprise also has proprietary information, often accumulated in data lakes and kept separate from the public internet. Turning that information into useful model behavior introduces requirements around privacy, ownership and differentiation. Mattern adds a cost concern: with per-token billing, spending rises as more applications make more calls.
Smaller models such as Llama 3 8B offer another route: pretrain or fine-tune them for a particular enterprise task. That can improve the fit to proprietary data while giving the organization more control over the model. The operational burden then moves to managing many specialists.
| Approach | Useful property | Enterprise burden |
|---|---|---|
| Broad general model | Wide task coverage; easy integration | Proprietary-data fit, privacy and usage cost |
| Task-specific models | Adaptation and ownership | Orchestration, maintenance and repeated fine-tuning |
A legal model, an HR model and a coding model may each be useful, but they do not automatically form one manageable service.
Samba-1 uses a composition of experts, or CoE, to organize those separate models. Here, an expert means a model serving a particular set of tasks. Four controls make the collection usable:
- Single secure endpoint: Put legal, HR and coding models behind one application interface.
- Prompt routing: Inspect an incoming prompt and select the best-suited expert.
- Independent fine-tuning schedules: Retrain and swap models at their own cadence—for example, a finance model annually when policies change and a coding model quarterly as code evolves.
- Model-level RBAC: Grant an application, person or group access to only the models it should use.
The endpoint simplifies integration, while routing, update schedules and access controls preserve distinctions between the models behind it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing, evaluating and invoking experts
Organizations can choose a flexible CoE, selecting the models themselves, or begin with a precomposed Samba-1 collection. Mattern describes the precomposed catalog as 92 experts spanning languages, domains and enterprise tasks. She describes support across seven foundation-model architectures, naming Llama 2, Llama 3, Mistral, Falcon and BLOOM, and also mentions multimodal capabilities through LLaVA, CLIP and DePlot. These examples establish the breadth of the collection rather than a single uniform model architecture.
Mattern reports that SambaNova developed or co-developed 12 of the 92 experts. Examples include models for Thai, Japanese and Hungarian, a text-to-SQL model, and contributions to BLOOMChat for multilingual use. She describes BLOOMChat at the time as the second-largest open-source model. The precomposed catalog is a starting point: organizations can add experts to suit their own requirements.
Evaluation follows enterprise tasks rather than catalog size. SambaNova’s benchmark includes information extraction, text-to-SQL, coding and function calling. Mattern reports meeting or exceeding GPT-3.5 Turbo and GPT-4 on the enterprise comparisons presented. The talk does not establish the datasets, scoring or exact model versions behind those comparisons.
Applications also need more than one way to invoke the collection:
| Invocation mode | Application behavior |
|---|---|
| Routing | Submit a prompt and let the router select an expert |
| Direct selection | Call a known model for a known task |
| Model chaining | Connect model calls in an agentic workflow |
Routing handles selection when the application does not already know the right expert. Direct calls and chaining retain explicit control when it does.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping many models close to compute
The hardware supporting this collection is the SN40L reconfigurable dataflow unit, or RDU. Its memory hierarchy has three tiers: on-chip SRAM, high-bandwidth memory and a larger DDR capacity. The purpose is to keep many models available, then move the models needed for execution toward the faster tiers. Tight connections between the tiers make that movement part of the system’s execution strategy.
Mattern quotes 4 GB of SRAM, 512 GB of high-bandwidth memory and up to 6 TB of DDR. Those spoken totals should not be read as per-chip specifications: the SN40L architecture paper specifies 520 MiB of SRAM, 64 GiB of HBM and up to 1.5 TiB of DDR per socket, and describes an eight-socket node. The talk’s 6 TB DDR figure refers to a configuration whose scope is not established here.
The deployment comparison is about where a large model collection resides. In Mattern’s example, hosting 500 models on GPUs spreads them across systems, so applications must reach the system containing the required model. SambaNova instead uses its DDR tier to retain a large collection within one underlying system and move active models up the hierarchy. She claims capacity for up to five trillion parameters, but gives no precision or configuration assumptions for that total. The useful mechanism is the separation between capacity for stored experts and fast memory for active execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A workout plan, then responses while typing
The live demonstration opens Samba-1 Turbo from SambaNova’s website. Its model selector includes Llama 3 8B and Llama 3 70B, a CoE option, Mistral and SambaNova-trained models such as SambaLingo. Petro selects Llama 3 8B and asks it to create a three-day-a-week workout schedule for an intermediate fitness level. He repeats the request, then opens the performance insights.
For the demonstrated workout-plan response, Petro reports approximately 1,000 tokens/s, 0.09 seconds to the first token and 0.65 seconds total inference time. These three measurements describe different parts of the experience: generation throughput, the wait before output starts, and the duration of the response. He then submits the same prompt to ChatGPT for a visual speed comparison; the demonstration does not provide a matched model or controlled benchmark configuration.
The next mode changes the interaction itself. With real-time generation enabled, the response appears while the user is writing the prompt and changes as the prompt changes. An email-drafting example illustrates the benefit: revise the request and immediately inspect the resulting draft. The same feedback loop can support prompt engineering without repeatedly stopping to submit a finished prompt. The public demo is accessible from a laptop or phone.
An attendee asks whether generation parameters such as top-p can be adjusted. After checking the interface, the presenter qualifies the initial affirmative answer: those controls appear fixed in this UI. The API exercise is where model configuration becomes available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preparing the basic inference notebook
The hands-on work starts with loading environment variables, initializing an LLM and making one Python inference call. The second exercise builds a RAG question-answering application using LangChain, document loaders, E5-large-v2 embeddings, ChromaDB and the Llama 3 endpoint. Both exercises live in the SambaNova AI Starter Kit. The recording uses its 2024 SambaStudio workflow and Python 3.10; the repository’s current setup instructions differ.
For the first exercise, navigate to workshops/ai_engineer_2024/basic_examples. The setup sequence is:
- Clone the starter-kit repository and create
.envat its root. - Add the SambaStudio endpoint credentials supplied for the workshop.
- Enter
basic_examplesand create a Conda or Python virtual environment. - Install the exercise requirements and register the environment as a notebook kernel.
- Open the SambaStudio example notebook in Jupyter or VS Code and select that kernel.
The workshop provides two dedicated endpoints through Discord; participants can use either corresponding set of credentials. The demonstrated Conda environment is named basic_x, with Python 3.10 recommended.
With the environment active, pip install -r requirements.txt installs the basic exercise’s dependencies. A notebook can run in the browser through Jupyter or directly in VS Code. The demonstration uses VS Code, where the selected Python kernel and cell execution timing remain visible alongside the example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Configuring inference and marking the assistant turn
The notebook imports its libraries, loads .env and initializes LangChain’s SambaStudio wrapper with the endpoint credentials and model configuration. do_sample=False disables sampling, which the presenter describes as deterministic generation; setting it to True enables probabilistic sampling. Temperature and the maximum number of generated tokens are also configurable. Because this is a shared CoE endpoint, the configuration selects Meta Llama 3 8B-Instruct as the expert.
The first inference call uses llm.invoke with the question What is the capital of France? It returns the answer, but also continues with material the user did not request. The next step adds the instruction-message format documented in Meta’s Llama 3 repository: mark the beginning of the text, identify the user turn, end that turn and open the assistant turn. In this raw-prompt workflow, those boundaries tell the model where the user’s request stops and its response should begin.
The same question can then become a reusable LangChain Expression Language, or LCEL, chain. A prompt template substitutes the country, the configured LLM generates the response, and an output parser returns text. Given the initialized llm, the composition is:
python
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
def make_capital_chain(llm):
prompt = PromptTemplate.from_template(
"<|begin_of_text|>"
"<|start_header_id|>user<|end_header_id|>\n\n"
"What is the capital of {country}?<|eot_id|>"
"<|start_header_id|>assistant<|end_header_id|>\n\n"
)
return prompt | llm | StrOutputParser()
def ask_capital(llm, country: str) -> str:
chain = make_capital_chain(llm)
return chain.invoke({"country": country})
Calling ask_capital(llm, "France") preserves the original question while making the country an input rather than hard-coded prompt text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adding document retrieval before generation
The second exercise lives under workshops/ai_engineer_2024/ekr_rag, with notebook and Streamlit versions. Retrieval-augmented generation supplies the LLM with information from outside its training data. This is useful for proprietary documents or current information that should be available without retraining the model, and it can reduce hallucinations in some contexts.
The pipeline separates document preparation from answering a question:
- Load and parse documents. Use a loader suited to PDF, text or PowerPoint and extract text that downstream tools can process.
- Split the text. Create smaller chunks, with chunk size and overlap as configurable parameters.
- Embed the chunks. E5-large-v2 maps each chunk into a numerical vector.
- Store the index. Keep vectors together with their text and metadata in ChromaDB; Faiss is mentioned as another option.
- Retrieve for a query. Embed the question and find nearby chunk vectors using a similarity metric.
- Optionally rerank. Reorder retrieved chunks by relevance and remove unnecessary material.
- Generate the answer. Pass the question and selected context to the LLM.
The vector search supplies relevant source material; the final model call turns that material into an answer.
Execution crosses a clear boundary. Third-party tools handle loading, splitting and storage. The embedding model can run on the laptop’s CPU or through an endpoint on SambaNova hardware, while the LLM runs on SambaNova hardware. The recorded run uses remote embeddings; a CPU comparison is proposed but does not appear in the completed demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preparing the PDF-processing environment
Reuse the cloned repository, but deactivate the first exercise’s environment before setting up the RAG exercise. The root-level .env now needs two sets of connection details: the SambaStudio LLM endpoint and API key, plus the embedding endpoint and API key. Installation runs from the ekr_rag directory.
PDF processing adds system dependencies beyond Python packages. On macOS, the workshop installs Tesseract for OCR and Poppler outside the virtual environment. It then uses a Python 3.10 environment:
bash
brew install tesseract
brew install poppler
python3.10 -m venv .venv
source .venv/bin/activate
Run these environment commands from the exercise directory. If the shell cannot find Python 3.10, first install it and update the shell’s path, reloading .bashrc or .zshrc as appropriate. After activation, run the exercise’s installation script, install ipykernel and register the kernel for notebook use.
Some workshop laptops encountered NLTK or SSL-certificate issues, for which the presenter points to an additional repair script. Once the dependencies are ready, the notebook can open through Jupyter or VS Code. A Streamlit version is available in the exercise, but the recording proceeds with the notebook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Indexing the SN40L paper and asking a question
The RAG notebook under ekr_rag/notebooks coordinates supporting modules, including document_retrieval.py and vector-database utilities. Its initial cells establish the kit directory as ekr_rag and the repository directory as ai-starter-kit. Loader selection comes from configuration: the demonstration starts with PyPDF/PyPDF2, with Unstructured available as an alternative.
The input document is the SN40L paper, which describes SambaNova’s hardware, stack and composition of experts. The exercise supplies it through GitHub and also allows a custom PDF. PyPDF loads the document content into a list, then LangChain’s recursive splitter creates the chunks. With chunk size 1,200 and overlap 240, the demonstrated 15-page PDF produces 89 chunks. The spoken configuration does not specify the size units, so 1,200 should not be interpreted as a token count.
Next, config.yaml selects SambaStudio as the embedding type for RDU execution. The endpoint accepts batches of 1 or 32 chunks; a batch size of 32 processes that many chunks together. This embedding service is a standalone endpoint, so its CoE setting is false. The resulting vectors are indexed in ChromaDB.
The presenter initially expects vectorization to take about ten seconds, then restarts the notebook and reruns the cells. After the restart, he reports four seconds to embed the demonstrated document. That is the timing reported for this run, rather than a CPU-versus-RDU comparison.
The QA chain now embeds an incoming question, retrieves the top three chunks and supplies them with the question to the LLM. This run does not use a reranker. Asking “What is a monolithic model?” produces a response the presenter describes as instantaneous; the notebook displays the answer alongside a source link and retrieved text. The visible result connects generation back to the document used to answer the question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The harder test: retrieving from a table
The last test moves beyond a definition to information in a table. The presenter opens the paper’s table relating operation intensity to fusion level and submits a prepared question to see whether the PyPDF-based pipeline can recover the relevant information. He judges the returned answer correct.
For more complicated PDF tables, the recommended next step is to change the configured loader from PyPDF to Unstructured. This is where the workshop ends its implementation: a working notebook question-answering flow, followed by a concrete adjustment to document extraction. Fast inference makes the answer appear quickly; the loader still determines what information reaches the retrieval pipeline in a usable form.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original paper explaining SN40L's memory hierarchy, operator fusion and composition-of-experts experiments.
Application examples and setup documentation for building with SambaNova. Current instructions differ from the recorded workshop.
English retrieval embedding model with examples for query and passage encoding, normalization and input-length handling.
Further reading
Contemporary vendor account of the Llama 3 8B milestone and its proposed uses in multi-call applications.
Meta's Llama 3 inference examples and instruction-message formatting guidance.
Full transcript
Loading transcript…