AI Engineer World's Fair 2024

Decoding Mistral AI's Large Language Models

Read the talk

Decoding Mistral AI's Large Language Models

Mistral’s early model releases connect sparse computation, instruction tuning and preference feedback to a practical question: how much capability can you get for the cost of running a model?

From a talk by Devendra Singh Chaplot

Before you start: Basic familiarity with language-model prompts and Python functions is helpful; the article introduces tokens, sparse experts and the training stages as they appear.

A model portfolio built around developer access

What does it take to put capable language models in developers’ hands—and why release their weights while also selling commercial models? Mistral’s first year provides the starting point. Founded in June 2023, the company released Mistral 7B that September, then Mixtral 8x7B in December. Alongside the sparse expert model came platform APIs and commercial offerings, Mistral Medium and Mistral Embed.

February brought Mistral Large, positioned by Devendra Chaplot as the flagship for reasoning and mathematics. Mixtral 8x22B followed in April. The newest release discussed here is Codestral 22B, a model specialized for programming: Chaplot describes it as a June release, although its announcement dates to May 29, 2024. At the time of the talk, Codestral and Mistral Large were available through Mistral’s free chat interface.

Timeline with rows for open models, commercial models, products and investments, including Mistral 7B, Mixtral models and Codestral 22B.Open full source frame
Mistral AI’s model and product timeline from June 2023 to June 2024.
0:160:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Weights, deployment and customization

Developer access involves more than an endpoint. Mistral’s principles combine openness with portability: Chaplot lists Azure, AWS, GCP, virtual private clouds and on-premises deployment as options across the portfolio. Licensing weights for use on your own servers gives you control over where data travels and how it is secured. The deployment choice becomes part of the application’s privacy and security design.

The other priorities are performance relative to speed and model size, and the ability to customize a model for a particular application. The talk introduces mistral-finetune and a managed fine-tuning API for adaptation, followed by mistral-inference for running models. Both repositories were available tools in this historical workflow; both were subsequently archived in June 2026.

1:422:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:42 · section reference included

More capacity without using every parameter

Mistral 7B is a dense transformer. Chaplot describes it as the first 7B model to reach 60 on MMLU and treats that level as an observed threshold for usefulness, rather than a universal rule. The original technical report reports 60.1% on five-shot MMLU for pretrained Mistral 7B under the authors’ evaluation pipeline. Its small size also enabled community deployments on laptops and phones, with speeds Chaplot describes as reasonable; the talk supplies no device configuration or throughput measurement.

Mixtral 8x7B changes the relationship between model capacity and computation. A sparse mixture of experts stores knowledge across a larger collection of parameters but uses only a subset for each token. More weights can therefore contribute to the model’s overall capacity without every token paying the computational cost of activating them all.

ArchitectureCapacityComputation per token
Dense transformerModel weightsUses the dense network
Sparse mixture of expertsShared weights and multiple expertsUses a subset of expert parameters

The distinction is between total parameters and active parameters. Sparse activation constrains inference computation; it does not make the inactive weights disappear from the model.

Mixtral 8x22B scales up this sparse architecture. Chaplot presents it as offering better performance and a larger context window, with multilingual support that includes English, French, Italian, German and Spanish, as well as other languages.

3:193:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:19 · section reference included

Why open releases can support a business

Releasing weights raises an obvious commercial question: what remains to sell? Mistral’s answer is that community contribution and long-term business value can reinforce each other. Open science is one motivation, but the exchange also creates concrete benefits for the company.

  • Awareness: Developers try the models and recommend them to others. Chaplot says Mistral had no in-house marketing team at the time; model quality and word of mouth helped create demand.
  • Paid upgrades: Users who like an open model may move to a proprietary model when they need greater capability.
  • Deployment knowledge: Community experiments reveal ways to customize models or run them in new settings. Laptop and phone deployments are useful discoveries that Mistral does not have to produce entirely on its own.

The feedback runs in both directions: developers gain access to weights, while Mistral learns from what they build.

5:035:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:03 · section reference included

The simple objective behind pre-training

The training overview has three stages: pre-training, instruction tuning, and learning from human feedback. Pre-training starts with ordinary text. Given the preceding tokens, the model learns to predict the next one. Repeating that task across a large corpus teaches the patterns needed to continue many kinds of text.

Tokens are not necessarily whole words. Chaplot gives a rough estimate of three-quarters of a word per token, though that ratio depends on the language and tokenizer. A vocabulary may contain tens of thousands or hundreds of thousands of tokens. Each token has an integer ID and an associated embedding, so the model receives a sequence of numerical representations and predicts a continuation.

7:117:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:11 · section reference included

Why a simple objective is hard to train

The difficulty begins with the dataset. Chaplot describes pre-training corpora containing trillions to tens of trillions of tokens. Preparing that material requires preprocessing, cleaning, deduplication and curation. More data is not automatically better data: adding noise can degrade the model rather than improve it.

At the largest scales, the discussion extends to models with hundreds of billions or even trillions of parameters. Chaplot estimates training costs in the tens to hundreds of millions of dollars for the large-model setting he describes, without giving per-model accounting. That expense makes failure unusually consequential. A small company may struggle to fund another run after an unsuccessful one, so training is not an experiment it can casually repeat until the settings work.

Nor can a team assume that the best hyperparameters for a small model will remain best for a larger one. Chaplot uses the Llama 1 family to make the uncertainty concrete: why does its 65B model have 80 layers rather than 82? His answer is that such choices still depend heavily on experience and intuition. Architecture, dataset mixture and training settings are consequential decisions, but their optimal combination is not settled science.

8:148:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:14 · section reference included

Knowing Python is not the same as answering a request

Ask a pretrained model to write a Python function that determines whether a number is prime. It may continue the problem statement, provide an example or explain an approach without producing the function. That response follows its training objective: continue the text with likely next tokens. A request written in natural language does not, by itself, guarantee that code is the most likely continuation.

Now supply a Python function definition and a docstring describing the same task. That context cues a code continuation, and the demonstration obtains the implementation. For example, the definition and docstring below establish the pattern that a completion model can extend with the body:

python

def is_prime(n: int) -> bool:
    """Return whether n is a prime number."""
    if n < 2:
        return False
    divisor = 2
    while divisor * divisor <= n:
        if n % divisor == 0:
            return False
        divisor += 1
    return True

The change is in how the capability is elicited. The model can produce the answer, yet may not respond to a human request in the expected form. That gap motivates the next two training stages.

10:3710:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:37 · section reference included

Instruction tuning changes which tokens contribute to loss

Instruction tuning replaces undifferentiated text with prompt-response pairs. The prompt can now be the ordinary request to write a primality function, and the target response can be code such as is_prime above. The examples teach the model the interaction format users expect.

The prediction task remains next-token prediction, but the prompt is masked out of the training loss. It remains visible as context; the model is not being asked to learn the response without seeing the instruction.

  1. Present the prompt followed by the target response.
  2. Use the prompt and preceding response tokens to predict each response token.
  3. Compute the training loss on response tokens only.

This is response-only loss: the model learns to answer the request rather than being rewarded for predicting the request itself.

Chaplot describes instruction datasets ranging from hundreds to hundreds of thousands of examples. Compute requirements are much smaller than for pre-training: his spoken estimate is on the order of 100 GPUs for hours or days, while the slide gives a broader 1–100 GPU range. These are scale estimates, not a configuration-specific requirement.

Slide lists paired prompt-response data, masked next-word prediction, 1–100 GPUs and a few hours or days, beside an instruction-tuning model diagram.Open full source frame
Instruction fine-tuning: paired examples, masked next-word prediction and compute requirements.
11:3811:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:38 · section reference included

Learning from choices between answers

Writing an ideal answer from scratch can be expensive. Choosing the better of two candidate answers is often easier. Human feedback turns that comparison into training data: show a person a prompt and two responses, then record which response they prefer. This can scale supervision faster than requiring a fully authored answer for every prompt.

Chaplot names reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) as routes for using preference data in further tuning. The talk does not derive either algorithm.

StageTraining materialLearning signal
Pre-trainingText sequencesNext-token prediction
Instruction tuningPrompt-response pairsResponse-token prediction
Preference tuningCompared responsesHuman preference

These stages differ in both their supervision and their resource demands. They describe the training framework used for Mistral’s open models, not a claim that every released checkpoint underwent both RLHF and DPO: pretrained and instruction-tuned checkpoints remain distinct.

12:4113:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:41 · section reference included

Reading performance against active computation

The performance-cost graph returns to the purpose of sparse computation. Its horizontal axis measures active parameters, and its vertical axis measures MMLU performance. The desirable direction is toward the upper left: stronger benchmark results with fewer parameters participating in each token’s computation.

Chaplot presents active parameters as proportional to the cost of running the model. For this graph, that is a computational proxy rather than a complete deployment-cost model. It makes the earlier distinction useful: a sparse model can carry more total capacity while limiting the amount of that capacity activated per token.

13:3913:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:39 · section reference included

Codestral: completion and conversation

Codestral 22B applies the performance-and-speed goal to programming. It is a dense transformer trained on more than 80 programming languages, with two interaction modes:

  • Instruction mode: Ask a coding question or describe a bug or error, as in a chat assistant.
  • Fill-in-the-middle mode: Supply code around a gap and generate the missing portion, supporting completion inside an editor.

These modes serve different moments in development: discussing a problem and continuing the code already on screen.

Chaplot reports that Codestral outperforms Code Llama 70B, DeepSeek Coder 33B and Llama 3 70B in the coding comparisons presented, despite its smaller size. This is a benchmark-specific comparison, not a guarantee across programming tasks. He also highlights its longer context window relative to the compared open coding models. Downloadable weights do not imply identical permissions across the portfolio: Codestral’s original release used the Mistral AI Non-Production License, unlike Mistral 7B’s Apache 2.0 license.

Benchmark table comparing four models, followed by bullets on performance and speed, longer context, and instruct and fill-in-the-middle modes.Open full source frame
Codestral results alongside CodeLlama, DeepSeek Coder and Llama 3.

The following multilingual comparison extends the coding-performance discussion across languages; Chaplot says Codestral tends to perform better across the languages shown. For access, he points to free chat use and La Plateforme APIs, then tentatively mentions a free period lasting until the end of July. The launch terms distinguish an eight-week free beta on a dedicated Codestral endpoint from billed platform access, so that historical offer should not be read as free access through every API route. VS Code and JetBrains integrations bring completion into the editor.

14:2214:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:22 · section reference included

Prototype broadly, then specialize the model

For a new application, start with the adaptation methods available around the model: prompt engineering, few-shot examples, chain-of-thought prompting and retrieval-augmented generation. Open weights additionally let you perform task-specific fine-tuning with your own data and compute. Chaplot contrasts that freedom with commercial models, which he says typically do not permit fine-tuning. That is a contextual generalization, not a categorical restriction: GPT-3.5 Turbo already offered commercial fine-tuning before this talk.

The practical decision is how to balance application performance against cost. A high-end commercial model offers strong general-purpose capability and can make the first prototype easier to build. Once the task is understood and usage grows, a smaller open model fine-tuned for that task can become competitive—or outperform the commercial alternative on that specific workload.

Chaplot describes an observed progression:

  1. Prototype with the most capable model available.
  2. Establish the particular task the application needs to solve.
  3. Fine-tune an open model such as Mistral 7B or Mixtral 8x7B for that task.

Specialization can improve performance relative to cost because the deployed model no longer has to be the strongest general-purpose system. The relevant test becomes whether it does the application’s job well enough at the volume and cost the product requires.

16:0316:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:03 · section reference included

Resources

From the talk

  • Mixtral 8x22B releaseArticle4:42

    Release announcement describing the larger sparse expert model and its active versus total parameter counts.

  • Codestral 22B releaseArticle1:23

    Original coding-model announcement covering completion, benchmarks, licensing and release-era access.

  • LoRA training code and fine-tuning examples. Archived in June 2026 and no longer maintained.

  • Reference inference code with chat and fill-in-the-middle examples. Archived in June 2026.

Transcript download
Full transcript

Loading transcript…