AI Engineer World's Fair 2024

No more bad outputs with structured generation

Loading slides…

Read the talk

Making model outputs dependable with structured generation

A flight-extraction failure leads into Outlines, token masking, and the ways output constraints can improve validity, reduce generation work, and change task performance.

From a talk by Rémi Louf

Before you start: Familiarity with Python, JSON, and the idea that language models generate text one token at a time will help.

A JSON error is an interface failure

Suppose you want to extract flight details from emails. The next piece of software expects a JSON object containing an origin, a destination, and other flight information. You prompt the model carefully, insist on JSON, and try function calling—yet sometimes the result still produces a JSON decoding error. This is the opening problem Rémi Louf, co-author and co-maintainer of Outlines and co-founder and CEO of .txt, brings to structured generation.

Black slide displaying json.decoder.JSONDecodeError above an ASCII-art cow.Open full source frame
A JSON decoding error illustrates the opening failure case.

Output structure is part of a software interface. Modular systems work because one component can rely on another component’s contract. If a model sometimes returns an object and sometimes returns something the caller cannot parse, every downstream consumer inherits that uncertainty. Louf connects this failure to the difficulty of building dependable agents: the concern is not merely an untidy answer, but an unreliable boundary between components.

Structured generation moves the required format into the generation process itself. Instead of asking the model to respect an interface and checking afterward, the decoder restricts what the model can emit. That directly addresses output structure; whether it also improves the answer is a separate question, which the later experiments test.

0:000:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Turn an extraction pattern into a generation constraint

Outlines is a Python library intended to fit into an existing workflow, rather than a framework that requires the workflow to fit around it. At the time of the talk, Louf reports that vLLM and TGI use Outlines underneath their function-calling support. He also credits a contributor community of roughly 87–88 people for building the library.

The basic workflow has three steps:

  1. Choose and instantiate a model. The demonstration focuses on open models. Louf reports six provider integrations, including Transformers, llama.cpp, and the recently added MLX-LM; the OpenAI integration is used primarily for comparisons.
  2. Create a generator. In the introductory example, generate.text creates a text generator.
  3. Call it with a prompt. The prompt asks for the benefits of structured generation in one sentence, with a period used as the stopping condition.

The recording uses the historical generate.* API. Current Outlines documentation uses a different model/output-type interface, so the API names here describe the demonstrated workflow rather than a current installation recipe.

Now ask for the IP address of Google’s public DNS servers. Unconstrained generation can surround the address with a long explanation. An application might then use a regular expression to recover the address from that text. Outlines lets you move that same kind of pattern earlier: give it to generate.regex, then call the resulting generator with the question. The pattern defines the allowed answer, rather than searching an answer after generation. Louf’s demonstration uses Mistral 7B v0.1 and returns the correct address directly.

2:042:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:04 · section reference included

Specify the structure inside the fields

Regular expressions are one way to describe an output; JSON Schema and Pydantic models are another. Returning to flight extraction, a schema can require an object with origin and destination fields. But declaring both fields as arbitrary strings leaves useful information out of the contract. Louf instead uses a custom airport-code type: each value must contain three uppercase letters.

This distinction is easy to express in a Python schema definition. The following teaching example encodes the field constraint described in the talk; it is not a transcription of the custom Outlines type:

python

flight_schema = {
    "type": "object",
    "properties": {
        "origin": {
            "type": "string",
            "pattern": "^[A-Z]{3}$",
        },
        "destination": {
            "type": "string",
            "pattern": "^[A-Z]{3}$",
        },
    },
    "required": ["origin", "destination"],
    "additionalProperties": False,
}

Three uppercase letters are a stronger contract than a string, but the pattern alone does not establish that the value names a real airport. The practical principle is to express the structure you actually know, while keeping format validity distinct from factual correctness.

The same schema-first workflow extends to vision. In a recently merged capability, Louf supplies a photograph of a dish, specifies the expected JSON, instantiates a generator, and passes both the image and the prompt. The demonstration returns valid JSON. He closes the usage introduction with the installation command pip install outlines.

4:595:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:59 · section reference included

Mask invalid continuations before sampling

A language model does not directly return text. Given a prompt represented as token IDs, it produces logits: scores from which next-token probabilities are derived. Generation then processes those scores and samples a token. Temperature changes the distribution; top-k and top-p restrict which candidates remain available. The sampled token is appended to the context, and the model runs again.

Structured generation intervenes between the model’s scores and token selection. For each candidate token, the constraint processor asks whether appending it to the current output would violate the required structure. If it would, the processor masks that token so it cannot be sampled. The remaining candidates still compete according to the model and sampling configuration; the constraint determines which continuations are admissible, not which admissible answer is true.

For a concrete illustration, suppose generation is inside the origin string from the flight schema and has already emitted SF. A candidate token whose decoded text is O preserves the three-uppercase-letter pattern. A candidate o violates it, and a closing quote would end the field too early. The latter two candidates must be masked. These are illustrative token texts, not a claim about a particular tokenizer’s vocabulary.

Candidate continuations inside an airport code

Constructed example: The prefix SF and candidate texts O, o, and a closing quote are invented teaching values. They do not assert a particular tokenizer vocabulary or reproduce a recorded token trace.

Current output prefix — unchanged
{"origin":"SF

Operation: Apply the three-uppercase-letter constraint to candidate token texts before sampling.

Output prefix

Before: Before constraint filtering
{"origin":"SF
After: After constraint filtering · Unchanged
{"origin":"SF

Candidate token text: O

Before: Before constraint filtering
Candidate for sampling
After: After constraint filtering · Changed
Allowed: completes three uppercase letters

Candidate token text: o

Before: Before constraint filtering
Candidate for sampling
After: After constraint filtering · Changed
Masked: lowercase violates the pattern

Candidate token text: "

Before: Before constraint filtering
Candidate for sampling
After: After constraint filtering · Changed
Masked: would close a two-letter code
Filtering changes eligibility; no next token has been sampled yet.

The difficult engineering problem is making those decisions efficiently across a model’s vocabulary at every step. Louf presents that efficiency as .txt’s distinguishing contribution relative to other structured-generation libraries, including Guidance and LMQL. The simple rule—exclude invalid continuations—does not by itself explain how to apply it cheaply.

5:275:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:28 · section reference included

Structure reaches beyond JSON

Even text that looks like ordinary reasoning can contain a useful formal structure. Louf uses a GSM8K math example: a question marker, question text ending in a question mark, and arithmetic within the solution. Arithmetic expressions can be described with a context-free grammar. That lets the constraint extend beyond a JSON envelope into the shape of the reasoning text and its final answer—36 in the illustrated problem.

The most direct benefit remains valid output. Prompting tricks attempt to persuade a model to emit JSON; constrained sampling limits generation to the specified structure. Louf illustrates the difference with a Predibase experiment using Mistral 7B v0.1 on a version of CoNLL modified to require JSON output.

Generation methodReported valid JSON
Unconstrained Mistral 7B v0.117%
With structured generation99.9%

Louf says the constrained result used no prompt optimization. These are reported JSON-validity rates for that experiment, not extraction-accuracy scores or a universal success rate for every schema.

7:437:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:43 · section reference included

Separate constraint overhead from generation work

Checking constraints adds work, but when that work occurs matters. Louf shows a comparison with Guidance in which overhead is plotted against generated-token count. In the displayed comparison, he describes Outlines’ incremental inference overhead as approximately zero, with an upfront compilation cost. His suggestion that the difference could be imperceptible even in a Groq integration is a prospective claim, not a reported Groq deployment measurement.

Constraints can also reduce how much generation is necessary, through two different mechanisms. First, some output text is already determined by the structure. If the next part of a JSON object must be a bracket or a known field name, there may be no decision for the model to make. In Louf’s toy JSON example, only five of ten tokens require generation. Skipping model calls along deterministic paths is an additional optimization, not an automatic consequence of masking invalid tokens; it also requires care about tokenization and subsequent probabilities.

Second, a narrow output format eliminates unnecessary prose. Louf revisits the Google DNS question: in his comparison, ChatGPT produces 50 tokens, while constrained generation produces eight. This saving comes from returning just the requested value. It is distinct from skipping fixed JSON text, and neither token-count illustration establishes a universal wall-clock speedup.

9:269:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:26 · section reference included

What are prompt examples teaching?

The next experiment measures GSM8K accuracy for Mistral 7B v0.1 while varying the number of examples in the prompt. With unstructured generation, one-shot accuracy is worse than eight-shot accuracy. With structured generation, Louf reports one-shot and eight-shot accuracy in the same ballpark. The talk does not give exact accuracy values for this comparison.

That result suggests a possible role for demonstrations beyond teaching the task itself: some of their value may come from teaching the model the required output structure. If the decoder already supplies that structure, fewer examples may suffice. Louf explicitly leaves this as a direction for further investigation, rather than a general explanation of how in-context learning works.

11:0311:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:03 · section reference included

A task-specific gain for Phi-3 Medium

The function-calling experiment asks whether constraints can improve task accuracy, not just parseability. Louf evaluates Microsoft’s Phi-3 Medium on the Berkeley Function Calling Leaderboard’s simple function benchmark. He reports the following results, comparing the constrained model with the best GPT-4 version in his comparison:

Model and generation methodReported accuracy in the talk
Phi-3 Medium, unconstrained86%
Phi-3 Medium, structured96.5%
GPT-4, version unspecified93.5%

These are the figures spoken and shown in the recording. The June 17, 2024 companion report, Beating GPT-4 with Open Source, distinguishes results between runs and reports different final figures: 96.25% for Phi-3-medium-4k-instruct and 94.5% for GPT-4-0125-Preview on 400 BFCL simple examples. The recording’s chart should therefore be read as its own reported comparison, not silently replaced with the later figures.

Three outlined bars labeled Phi-3-medium at 86%, GPT-4 at 93.5%, and Phi-3-medium with a document icon at 96.5%; axes read Accuracy and BFCL simple function.Open full source frame
BFCL simple function accuracy: 86%, 93.5%, and 96.5%.

The improvement is reported without fine-tuning. Its significance is specific but useful: changing the decoding constraints can extract substantially more task performance from an existing open model. It does not establish that Phi-3 Medium is generally more capable than GPT-4. Louf’s broader optimism about open models follows from this possibility of improving their useful behavior without changing their weights.

12:1012:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:10 · section reference included

From valid syntax to usable outputs

Louf closes by moving beyond the earlier regular-expression work, which he describes as roughly a year old. The team has extended its approach to context-free grammars, with applications including code, protein structure, and the arithmetic used in the GSM8K example. His negligible-overhead claim here concerns experimental grammar generation: the contemporaneous research describes an in-house implementation, distinct from the public Outlines CFG implementation.

The next boundary is semantic constraints. Text-to-SQL makes the distinction concrete: a model can produce valid SQL syntax while referring to a table or column that does not exist. Restricting identifiers to the database’s available tables and columns addresses a failure that a syntax grammar alone cannot prevent. Louf says the team’s internal work can guarantee that a query will run, while explicitly declining to guarantee that it answers the intended question. Executability and correctness remain different contracts.

A further efficiency direction moves constraints into the model architecture. Masking logits happens after the model has already computed scores, including scores for tokens that will be discarded. Preventing unnecessary computation earlier could save additional work. Louf presents this as ongoing research, with implementation details and measured gains still to come in an anticipated blog post.

For applications outside open-ended chat, Louf predicts that structured generation will become a normal part of using language models. The flight-email example explains the practical pull: once an output must cross into another software component, its structure becomes part of what that component needs to trust. He ends by directing the audience to supporting blog posts through the closing QR code.

Large pink text on black reads: You will be using structured generation.Open full source frame
“You will be using structured generation.”
13:1913:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:19 · section reference included

Resources

From the talk

  • Python library for constrained outputs, with installation instructions and current typed-generation examples.

Updates since the talk

Transcript download
Full transcript

Loading transcript…