Contents
  1. Software proposals
    1. When generated language becomes input
      1. Four states worth naming
    2. What a schema promises
      1. Constraint and claim
  2. Continuing foundations
    1. Grammars, interfaces, and constrained generation
      1. Selected turning points
  3. Structured values
    1. Parsing is a sequence of claims
      1. Small failures at different layers
    2. How constrained decoding blocks malformed continuations
    3. Coverage, latency, and incomplete output
      1. Four independent operational questions
    4. Structure is not domain correctness
      1. Checker and supported claim
  4. Operation requests
    1. Tool definitions shape available choices
      1. Overlapping tools
    2. Choose a call—or do not call
      1. Valid dispositions
    3. A call is an envelope, not an effect
      1. Minimal correlation loop
  5. Controlled effects
    1. Authorization belongs at the protected operation
    2. Execute through a controlled dispatcher
      1. Result meanings
  6. Failure and recovery
    1. Failures need distinct meanings
      1. Failure and response
    2. Recover when the effect is unknown
  7. Boundary evidence
    1. Test each claim separately
      1. Claim-to-check map
  8. Interoperability
    1. Where MCP meets the local action boundary
  9. Check understanding
  10. Open questions
  11. Selected talks
  12. References
  13. Talk library
← All topics

Structured outputs

When a person reads a model response, the person can interpret ambiguity, notice implausible details, and decide whether to act. Software cannot rely on that informal judgment. It needs explicit representations and gates. A model emission therefore begins as an untrusted proposal. A schema can constrain its shape; parsing and validation can reject malformed or inadmissible values; authorization can decide whether a proposed action is permitted; a controlled dispatcher can execute it; and an authoritative observation can establish what actually happened. These are separate claims. Reliable systems preserve the separation instead of treating “valid JSON,” “tool call accepted,” and “task completed” as synonyms.

Part I — Software proposals

When generated language becomes input

A model response crosses a consequential boundary when software, rather than a person, consumes it. Ordinary prose can suggest, “Create an issue titled Login failure.” A machine-readable response can represent the same intent as an operation name plus arguments. That representation enables automated checks, but it does not give the model authority to execute anything. In the demonstrated function-calling loop, the model expresses an intended operation and arguments; application code performs the operation and returns its result to the model. Software contracts supply the broader idea: an interface states obligations, but code at the boundary must establish them at runtime.

Four states worth naming

  • Proposed record — Candidate data, such as an extracted invoice or classification. It has not yet been accepted as correct application data.
  • Proposed action — A request naming an operation and its arguments. It describes what the model wants the application to consider doing.
  • Accepted operation — A proposal that has passed the application’s structural, semantic, state, and authorization checks.
  • Observed effect — Evidence from the relevant external boundary that execution produced, failed to produce, or may have produced the intended state change.

These states prevent a fluent response from laundering uncertainty. A sentence saying “The message was sent” is merely generated text. A tool result saying “accepted” proves only what that component’s contract defines. If the user cares whether a message rendered in a client, a provider receipt or edge observation must establish that later fact. The evidence chain is proposal, policy decision, execution attempt, and external confirmation—not one undifferentiated success flag.

From model response to observed effect

A typed representation enables checks, but proposal, acceptance, execution, and observation remain separate states.

Model responseGenerated content that is not trusted merely because it is fluent.
ProseA person interprets meaning and decides whether to act.
Typed proposalCandidate record or action request; it has not executed.
ValidateEstablish completion, parsing, schema, and domain claims.
AuthorizeDecide whether this actor may perform this operation on this resource.
ExecuteDispatch a recognized implementation across the effect boundary.
Observed effectEvidence from authoritative state or the user-visible boundary.
One model response may remain prose for human interpretation or become a typed proposal. The software path parses and validates candidate data, independently authorizes an operation, dispatches it, and observes the external boundary. An observed effect is not the same object as the original proposal.
Read the diagram as text
  • Model response. Generated content that is not trusted merely because it is fluent.
  • Prose. A person interprets meaning and decides whether to act.
  • Typed proposal. Candidate record or action request; it has not executed.
  • Validate. Establish completion, parsing, schema, and domain claims.
  • Authorize. Decide whether this actor may perform this operation on this resource.
  • Execute. Dispatch a recognized implementation across the effect boundary.
  • Observed effect. Evidence from authoritative state or the user-visible boundary.
  • Model response → Prose: text for interpretation.
  • Model response → Typed proposal: structured representation.
  • Typed proposal → Validate: candidate data.
  • Validate → Authorize: accepted arguments.
  • Authorize → Execute: policy allows.
  • Execute → Observed effect: external observation.

What a schema promises

A schema is a machine-readable description of permitted data structure. JSON Schema can constrain object properties, required fields, primitive types, arrays, enumerated values, numeric bounds, and whether undeclared properties are allowed. The details matter. Listing a property does not make it required. additionalProperties is allowed by default unless constrained. A missing property differs from a present property whose value is null. A root $schema declaration identifies the dialect whose keyword meanings apply, although a generation engine may still implement only a subset of that dialect.

Constraint and claim

Schema elementWhat it establishesWhat it does not establish
type: "integer"The value has the validator’s integer representation.The number is factually correct or sensible for the domain.
required: ["project_id"]The named field must be present.The referenced project exists or is accessible.
enum: ["low", "high"]Only a listed value is accepted.The chosen value matches the source or user intent.
additionalProperties: falseUnknown object fields are rejected.Known fields are sufficient for the operation.
description or defaultDocumentation or a hint is attached to the schema.The description is enforced or a missing value is inserted.
items plus length rulesArray members and permitted length are constrained.The members are complete, unique in the domain, or correctly ordered.

Suppose a valid record is {"project_id":"P-17","priority":"high"}. The schema can prove that both fields are present and have permitted representations. It cannot prove that P-17 exists, that the caller may modify it, or that “high” reflects the request. Structured-output APIs themselves warn that schema-conforming values can still contain mistakes, and applications must separately handle refusals and incomplete responses. Structure is a precondition for dependable consumption, not a certificate of truth or permission.

Part II — Continuing foundations

Grammars, interfaces, and constrained generation

Modern structured output combines several older engineering lines. Formal-language work made valid symbol sequences explicit. Interface contracts separated operation descriptions from implementations. Schema languages described admissible data. Later neural systems incorporated grammatical structure into generation, and model APIs exposed function-shaped records that applications could dispatch. These lines coexist because they answer different questions: what strings belong to a language, what values an interface accepts, what operation is being requested, and who actually performs it.

Foundations of structured generation and tool calling

  1. 1963Revised ALGOL 60 reportSpecified valid program structure with recursive metalinguistic formulas.Sources & context

    Contributors: Peter Naur and the ALGOL committee

    What changed: Made a programming language’s permitted symbol sequences explicit independently of executing the programs they described.

  2. February 1984Cedar RPCGenerated client and server stubs from typed procedure interfaces.Sources & context

    Contributors: Andrew Birrell and Bruce Nelson

    What changed: Reduced bespoke communication code while preserving the reality that remote components can fail independently.

  3. December 5, 2009Early JSON Schema Internet-DraftProposed JSON-based contracts for validation, documentation, properties, arrays, bounds, and enumerated values.Sources & context

    Contributors: Kris Zyp

    What changed: Established an early standards proposal for describing admissible JSON data separately from each instance.

  4. 2017Yin and Neubig syntax-directed generationGenerated abstract syntax trees through grammar-production actions.Sources & context

    Contributors: Pengcheng Yin and Graham Neubig

    What changed: Encoded syntax in the available generation actions instead of requiring the model to learn the entire grammar from examples.

  5. November 2021PICARDRejected inadmissible SQL continuations through incremental parsing during decoding.Sources & context

    Contributors: Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau

    What changed: Showed how inference-time parsing could constrain ordinary generated text and incorporate database-schema checks.

  6. February 2023ToolformerTrained a model to decide whether, when, and how to call a bounded set of APIs.Sources & context

    Contributors: Timo Schick and colleagues

    What changed: Combined model-selected API calls with returned results across calculator, search, translation, calendar, and question-answering tools.

  7. June 13, 2023OpenAI function callingExposed JSON Schema function descriptions and model-selected argument objects.Sources & context

    Contributors: OpenAI

    What changed: Applied function-shaped output to both proposed external operations and structured data extraction.

  8. July 2023Efficient guided generationIndexed recognizer states to acceptable vocabulary tokens for regex-guided generation.Sources & context

    Contributors: Brandon Willard and Rémi Louf

    What changed: Moved repeated token-validity work outside sampling and reduced per-step lookup work in the analyzed construction.

Notice how formal syntax, typed interfaces, machine-readable schemas, grammar-guided generation, and model-selected calls contributed separate controls rather than one continuous replacement chain. Spacing is not to scale.

Selected turning points

DateDevelopmentContribution
1963Revised ALGOL 60 reportUsed recursive metalinguistic formulas to specify which symbol sequences form programs.
February 1984Birrell and Nelson’s Cedar RPC systemGenerated client and server stubs from interfaces naming procedures and argument/result types, while acknowledging independent network failures.
December 5, 2009Early JSON Schema Internet-DraftProposed JSON-based contracts for validation, documentation, properties, arrays, bounds, and enumerated values.
2017Yin and Neubig’s syntax-directed code generationGenerated abstract syntax trees through grammar-production actions instead of learning all syntax only from examples.
November 2021PICARDRejected inadmissible SQL continuations through incremental parsing during decoding.
February 2023ToolformerTrained a model to decide whether, when, and how to call a bounded set of APIs and incorporate their results.
June 13, 2023OpenAI function calling announcementLet developers provide JSON Schema function descriptions and receive model-selected argument objects for extraction or external operations.
July 2023Efficient guided generationIndexed recognizer states to acceptable vocabulary tokens for regex-guided generation.

The chronology is not a claim that one publication caused the next or that newer mechanisms replaced earlier ones. Constrained generation still needs a parser or recognizer. A schema-shaped function request still needs application dispatch. RPC-style interfaces still face remote failure. None of these mechanisms decides whether the requested action is semantically appropriate or authorized.

Part III — Structured values

Parsing is a sequence of claims

“It looks like JSON” compresses several independent questions. First, did the provider finish the response rather than stop at a length limit or refusal? Second, did transport decoding yield the intended bytes or text? Third, did the parser consume exactly one complete JSON document under the application’s chosen policy? Fourth, does the resulting value satisfy the schema? Only then can domain validation ask whether the value is usable. Validate external values develops this general boundary; model output adds the risk that a plausible-looking value was probabilistically generated.

Acceptance gates for generated data

Completion, parsing, schema conformance, and application validity fail for different reasons and justify different responses.

Provider responseContent plus refusal, completion, or truncation status.
Completion checkReject refusal, token exhaustion, or unfinished output.
Whole-document parseConsume exactly one JSON document under explicit duplicate, suffix, and number policies.
Schema validationCheck the declared structural contract.
Application validationCheck invariants, references, state, and other domain rules.
Accepted recordCandidate data admitted to the next application boundary.
Rejected resultPreserve the failed claim and safe corrective information.
A response must be complete before its decoded representation can be parsed as one unambiguous JSON document. Schema validation follows parsing, and application validation follows structural acceptance.
Read the diagram as text
  • Provider response. Content plus refusal, completion, or truncation status.
  • Completion check. Reject refusal, token exhaustion, or unfinished output.
  • Whole-document parse. Consume exactly one JSON document under explicit duplicate, suffix, and number policies.
  • Schema validation. Check the declared structural contract.
  • Application validation. Check invariants, references, state, and other domain rules.
  • Accepted record. Candidate data admitted to the next application boundary.
  • Rejected result. Preserve the failed claim and safe corrective information.
  • Provider response → Completion check: status and content.
  • Completion check → Whole-document parse: complete.
  • Completion check → Rejected result: refused or incomplete.
  • Whole-document parse → Schema validation: one parsed value.
  • Whole-document parse → Rejected result: syntax or policy failure.
  • Schema validation → Application validation: structure accepted.
  • Schema validation → Rejected result: schema rejected.
  • Application validation → Accepted record: domain accepted.
  • Application validation → Rejected result: domain rejected.

Small failures at different layers

Emission or conditionFirst failed claimWhy policy matters
Response ends after {"id":"P-CompletionA partial parser may retain earlier fields, but that does not make the original response complete.
{"n":1} trailing textWhole-document parsingSome parsers can decode the first value and report where it ended; callers must reject an unconsumed suffix when the contract requires one document.
{"role":"viewer","role":"admin"}Unambiguous object interpretationJSON implementations may reject duplicate names, retain one value, or expose both.
{"count":"7"}Schema or strict runtime validationA permissive validator may coerce a numeric string; that transformation must be an explicit interface policy.
{"project_id":"P-404"}Domain validationThe shape can be valid while the referenced entity does not exist.

Repair and coercion are transformations, not neutral cleanup. Guessing a missing quotation mark, discarding a suffix, selecting the last duplicate member, or converting strings to numbers can change meaning. A safe system preserves the raw emission for diagnosis, records the transformation policy, and rejects ambiguity when downstream consequences exceed what the repair can justify. TypeScript annotations alone do not help at this boundary because ordinary annotations are erased from emitted JavaScript; incoming data still needs runtime checks.

How constrained decoding blocks malformed continuations

Autoregressive generation chooses one token after another. Constrained decoding adds a recognizer for the permitted output language. Given the generated prefix, the recognizer determines which vocabulary tokens can still lead to a valid completion. Invalid candidates are masked before selection; the chosen token then advances the recognizer state. OpenAI has described compiling supported JSON Schema into a context-free grammar and caching an artifact used to determine valid continuations. Outlines describes the same central idea as testing candidate tokens and masking those that would violate the requested structure.

The recognizer cannot simply count braces. JSON strings can contain escaped quotation marks and brace characters that do not close an object. A tokenizer token can also span grammar boundaries or split a UTF-8 character, so implementations may need to match its bytes through several parser transitions. XGrammar describes a byte-level pushdown automaton with live stack state, cached context-independent masks, and additional checks for context-dependent tokens. The representation details belong in Tokenization; the consequence here is that token validity depends on both the prefix and the tokenizer.

Prefix state changes the allowed next piece

Example

Candidate validity depends on both the candidate text or byte piece and the recognizer state created by the prefix.

1 / 3 · Object opened

The current state is after the opening brace. Every stable candidate is classified: the id field piece is allowed and selected; the colon, closing brace, and escaped quote are masked.

New this stepToy object schemaRequires one string field named id.
New this stepCandidate piece: "id"Illustrative text or byte piece, not a claim about one tokenizer vocabulary entry.
New this stepCandidate piece: :Illustrative separator piece.
New this stepCandidate piece: }Illustrative object-closing piece.
New this stepCandidate piece: \"Illustrative bytes for an escaped quotation mark inside a JSON string.
New this stepCurrent state: after {The required property name must begin here.
New this stepNext state: after field nameReached by selecting the allowed id field piece.
This toy recognizer does not assume any particular model tokenizer. Each step shows one current state, the same illustrative candidate pieces, and an explicit allowed or masked decision for every candidate. The selected allowed piece leads to the next state; earlier states are omitted from later steps rather than remaining active.
Read the diagram as text
  • Toy object schema. Requires one string field named id.
  • Candidate piece: "id". Illustrative text or byte piece, not a claim about one tokenizer vocabulary entry.
  • Candidate piece: :. Illustrative separator piece.
  • Candidate piece: }. Illustrative object-closing piece.
  • Candidate piece: \". Illustrative bytes for an escaped quotation mark inside a JSON string.
  • Current state: after {. The required property name must begin here.
  • Next state: after field name. Reached by selecting the allowed id field piece.
  • Current state: after field name. The recognizer now expects the name/value separator.
  • Next state: before string value. Reached by selecting the allowed colon piece.
  • Current state: inside string value. Escape-aware string rules now determine candidate validity.
  • Next state: still inside string. An escaped quotation mark adds content without ending the string.
  • Toy object schema → Current state: after {: constrains.
  • Current state: after { → Candidate piece: "id": allowed and selected.
  • Current state: after { → Candidate piece: :: masked.
  • Current state: after { → Candidate piece: }: masked: required field missing.
  • Current state: after { → Candidate piece: \": masked.
  • Candidate piece: "id" → Next state: after field name: advance.
  • Current state: after field name → Candidate piece: "id": masked.
  • Current state: after field name → Candidate piece: :: allowed and selected.
  • Current state: after field name → Candidate piece: }: masked.
  • Current state: after field name → Candidate piece: \": masked.
  • Candidate piece: : → Next state: before string value: advance.
  • Current state: inside string value → Candidate piece: "id": masked: unescaped quote.
  • Current state: inside string value → Candidate piece: :: allowed string content.
  • Current state: inside string value → Candidate piece: }: allowed string content.
  • Current state: inside string value → Candidate piece: \": allowed and selected.
  • Candidate piece: \" → Next state: still inside string: advance without closing.
  1. Object opened. The current state is after the opening brace. Every stable candidate is classified: the id field piece is allowed and selected; the colon, closing brace, and escaped quote are masked. Active: Toy object schema, Candidate piece: "id", Candidate piece: :, Candidate piece: }, Candidate piece: \", Current state: after {, Next state: after field name. New: Toy object schema, Candidate piece: "id", Candidate piece: :, Candidate piece: }, Candidate piece: \", Current state: after {, Next state: after field name.
  2. Property named. The current state now expects a separator. The colon is allowed and selected; all other displayed candidates are masked. Selecting the colon advances to the value position. Active: Toy object schema, Candidate piece: "id", Candidate piece: :, Candidate piece: }, Candidate piece: \", Current state: after field name, Next state: before string value. New: Current state: after field name, Next state: before string value.
  3. Inside the string. The current state is inside a JSON string. Colon and closing-brace pieces are permitted as string content, an unescaped quoted id piece is masked, and the escaped quote is allowed and selected. Its escape keeps the recognizer inside the string. Active: Toy object schema, Candidate piece: "id", Candidate piece: :, Candidate piece: }, Candidate piece: \", Current state: inside string value, Next state: still inside string. New: Current state: inside string value, Next state: still inside string.

Masking guarantees only membership in the language implemented by the recognizer. It does not guarantee that a generated identifier exists, an extracted amount matches a document, or a requested operation is permitted. It can also alter content probabilities. Ordinary grammar-constrained decoding renormalizes among currently admissible tokens; that distribution is not generally identical to the model’s distribution conditioned on eventually producing a complete valid string, because exact conditioning would also account for each candidate’s probability of future grammatical completion.

Coverage, latency, and incomplete output

A schema can be valid JSON Schema yet unusable by a particular structured-output engine. Providers commonly support subsets; large or deeply nested schemas may be rejected, and converters can silently skip unsupported features. A production contract must therefore pin the provider, model or runtime, schema dialect, converter, and supported feature set. Inspect warnings and test the language actually enforced rather than assuming that a valid application schema was translated exactly.

Four independent operational questions

QuestionEvidence neededCommon mistake
Will the API accept the schema?A versioned feature-coverage test against the actual endpoint.Treating general JSON Schema validity as provider support.
Does the engine enforce the intended language?Positive and negative conformance cases, including unsupported keywords and nesting.Inferring exact language equivalence from a few compliant samples.
Will execution meet latency and capacity needs?Compilation, cache, per-token, batching, and memory measurements on the target stack.Repeating an implementation-specific “zero overhead” claim universally.
Is a streamed prefix safe to consume?Explicit partial-state semantics and a final completion signal.Treating a parsed partial object as a final authorized record.
Does task quality remain acceptable?Representative semantic evaluation separate from format compliance.Assuming fewer malformed outputs means better answers.

Published results illustrate why these dimensions stay separate. JSONSchemaBench evaluates schema acceptance, feature coverage, compliance, efficiency, and task quality independently; finite compliant samples cannot prove that an engine implements exactly the schema’s accepted language. A 2025 study across its selected models and tasks found that format constraints and semantic task quality could move differently, with effects varying by model and task. Treat such findings as reasons to measure your own workload, not as a universal penalty or benefit.

Structure is not domain correctness

Once a record passes structural validation, semantic validation asks whether its values are acceptable in the application’s domain and current state. Checks can include cross-field invariants, membership in a controlled set, referential integrity, units, temporal order, and preconditions. A receipt model can require numeric line items, yet still need to verify that quantity times unit price reconciles with totals. A scheduling request can contain valid timestamps while placing the end before the start. A project identifier can match its pattern while naming no current project.

Checker and supported claim

CheckerCan establishCannot establish alone
Schema validatorFields, representations, and declared local constraints.Existence, freshness, permission, or intent.
Deterministic domain validatorSpecified relationships such as start < end or totals reconciling under defined rounding rules.Facts for which it has no authoritative data or rules.
Authoritative lookupCurrent existence, version, membership, or provider state.That the user intended this entity or operation.
Authorization policyWhether this actor may perform this operation on this resource now.Whether the proposed values accurately express the user’s goal.
Intent or factual assessmentEvidence that a proposal corresponds to the request or source.A universal proof when the property itself lacks a complete specification.

Put known constraints into field descriptions before generation when that helps the model choose correctly, but retain validators afterward. In a Pydantic demonstration, a hidden date constraint caused an initial validation failure; returning the error enabled a corrected retry, while the speaker recommended stating the century requirement in the field description up front. The two controls are complementary: descriptions guide generation, and validators enforce specified acceptance rules.

No validator is stronger than its oracle. A deterministic rule is excellent for an invariant that code can fully state. It cannot generally decide whether an unrestricted answer is “safe,” whether an ambiguous request meant one customer rather than another, or whether a source supports a claim. Checks and their limits explains how to state what each oracle observes and what remains open.

Part IV — Operation requests

Tool definitions shape available choices

A tool definition is a model-visible description of an application operation. It normally includes a stable name, purpose, input schema, and—where supported—a result contract. The description should explain when to use the tool, when not to use it, prerequisites, specialized terms, resource relationships, and important effects. Parameter names and descriptions should state units and meaning; enums should expose the actual finite choices. The executable implementation and authorization policy remain separate.

Overlapping tools

Weak definitionBetter distinctionRemaining question
search: Search for information.search_web: Use for public, current webpages; do not use for private account data.Will the model obey the distinction on representative requests?
query: Run a query.query_orders: Read the current tenant’s order index using the documented filter grammar; load syntax guidance first.Can arguments still misuse the query language?
One tool per backend endpoint.One task-oriented lookup returning the contextual fields required by the workflow.Does the broader tool expose excessive data or hide useful control?

Tool granularity is an interface decision, not a mechanical reflection of backend APIs. Too many overlapping declarations can make selection harder; overly broad tools can expose unnecessary authority or return bloated results. Start with distinct task-level purposes, then use evaluations to justify routing, grouping, or progressive disclosure. Descriptions improve the model’s decision context, but they are instructions—not enforcement and not permission.

Choose a call—or do not call

Immediate tool selection is a disposition decision. The system may answer directly, ask for missing information, state that its tools cannot satisfy the request, or propose an eligible operation. Selection is separate from argument formation: choosing create_issue correctly does not mean the supplied repository, title, and labels are correct. It is also narrower than sequential planning; Choose a useful next action covers prerequisites and consequences across a longer trajectory.

Valid dispositions

ConditionDispositionReason
The answer is available without an external operation.Answer directly.A tool call would add cost and failure modes without adding needed evidence or effects.
A required argument is missing and cannot be safely inferred.Clarify.Guessing would create an under-specified action request.
Available tools cannot perform the requested capability.Decline or explain the boundary.A related tool is not an adequate substitute.
One permitted external lookup or effect is necessary and arguments are available.Propose the named call.The request can now be validated and authorized.
Several independent lookups are needed and the interface supports multiple calls.Propose separately identified calls.Independence must be established by the application, not inferred from co-occurrence.

Some APIs expose automatic, required, prohibited, or allowlisted tool modes. These modes alter the model’s available output space; they do not prove that a call is useful or permitted. Requiring a call can be appropriate inside a tightly bounded extraction interface, but it removes direct-answer and no-call options. Evaluate the exact mode together with missing-argument and unavailable-capability cases.

A call is an envelope, not an effect

A tool call needs an identity that survives dispatch and result return. This matters when one response requests several operations, including repeated calls to the same tool with different arguments. In the demonstrated parallel-call loop, each returned result retains the corresponding tool-call ID, allowing the model to distinguish the observations without relying on function name or completion order.

Minimal correlation loop

Illustrative pseudocode

Python-like pseudocode
calls = model_response.tool_calls

for call in calls:
    require call.name in allowed_tools
    args = validate(call.arguments)
    require authorize(actor, call.name, args)

    result = allowed_tools[call.name](args)
    observations.append({ call_id: call.id, result })

next_response = model(history + calls + observations)

Several calls in one model turn are not necessarily independent. The weather demonstration grouped two retrievals into one step, then waited for both results before requesting dependent arithmetic in the next step. Application code must preserve that dependency order; state-changing calls may need still stricter serialization or transactional controls.

Part V — Controlled effects

Authorization belongs at the protected operation

Authentication establishes an actor’s identity. Authorization decides whether that actor may perform a particular operation on a particular resource under current policy. A schema-valid request supplies only some decision inputs. Policy may also consider the authenticated subject, delegated task scope, resource ownership, environment, current state, and consequence. Attribute-based access control formalizes this as evaluation of subject, object, operation, and sometimes environmental attributes against policy.

Enforce authorization where the protected operation occurs, on every request. An IDOR—Insecure Direct Object Reference—illustrates why a valid token is insufficient: an application can authenticate the caller yet expose another user’s record when lookup-by-ID omits object-level authorization. In the reported agent attack, a visible user ID enabled traversal into personal records because the downstream tool checked admission but not ownership. Enforce current authority develops complete mediation and fail-safe defaults.

Permission is independent of valid arguments

A schema-valid proposal reaches the protected service only after policy evaluates actor, operation, resource, delegated scope, and current conditions.

Schema-valid callThe arguments have acceptable structure but no implied permission.
Authenticated actorThe principal and relevant delegated identity are established.
Policy decisionEvaluates actor, exact operation, resource, delegated scope, state, environment, and approval binding.
DeniedNo protected operation executes; a stable denial is returned.
Protected serviceThe enforcement point admits only policy-approved operations.
Service observationResult or authoritative state produced by the protected boundary.
The model supplies a schema-valid action request. Authentication identifies the actor, while policy evaluates the exact operation, target resource, delegated scope, current state, and any required transaction approval. Denial ends at the policy boundary; approval reaches the protected service. The model has no path around enforcement.
Read the diagram as text
  • Schema-valid call. The arguments have acceptable structure but no implied permission.
  • Authenticated actor. The principal and relevant delegated identity are established.
  • Policy decision. Evaluates actor, exact operation, resource, delegated scope, state, environment, and approval binding.
  • Denied. No protected operation executes; a stable denial is returned.
  • Protected service. The enforcement point admits only policy-approved operations.
  • Service observation. Result or authoritative state produced by the protected boundary.
  • Schema-valid call → Policy decision: operation and resource.
  • Authenticated actor → Policy decision: actor and delegation.
  • Policy decision → Denied: policy denies.
  • Policy decision → Protected service: policy allows.
  • Protected service → Service observation: execute and observe.

Human approval is also scoped data, not a vague remembered “yes.” Bind approval to the actor, session, operation, exact consequential arguments, resource scope, and lifetime. If arguments change, approval expires, or the request is regenerated under another identity, run authorization again and obtain any required new approval. OWASP’s transaction-authorization guidance similarly binds acknowledgement to significant operation details and warns against approving one transaction while executing altered details.

Credentials are capabilities, not proof of user intent. Delegated architectures can preserve both user and runtime identity—for example, by exchanging a subject token and authenticated runtime identity for narrowly requested access—but the authorization service must still decide whether to issue that authority. Whole-task accumulation and intervention policy belong in Constrain accumulated effects; broader attack paths belong in AI Security.

Execute through a controlled dispatcher

After acceptance, a controlled dispatcher resolves the operation through an allowlisted registry. It does not evaluate model-generated code, trust an arbitrary URL, or dynamically import a name chosen by the model. At the execution boundary it revalidates arguments, current resource state, authorization, deadlines, and resource limits. The handler then invokes application code or a hosted capability according to the integration’s actual execution contract. Tool-definition ownership and execution ownership are separate: some provider-defined tools still execute in the client, while server-hosted tools execute on provider infrastructure.

Result meanings

ResultMeaningNext obligation
Completed dataThe operation returned a final structured observation.Validate it and check any authoritative postcondition the task requires.
Domain failureThe operation ran but the requested domain outcome was rejected or unavailable.Return a stable error code and safe corrective details.
Infrastructure failureThe execution path failed before a final domain result was established.Apply retry policy only after considering effect knowledge and idempotency.
Pending operation handleWork was accepted but is incomplete.Track the handle; do not report final success.
Cancellation acknowledgedA cancellation request was received.Inspect eventual operation state because cancellation can be best effort.

A tool result is an observation from one boundary, not proof that the whole user task is complete. A provider can accept a mutation while dropping requested attributes; an internal message service can accept a request while the user-visible client renders nothing. Define the relevant postcondition, query authoritative state where possible, and preserve the provider receipt or operation handle. Establish the achieved outcome treats completion as a task-level question.

Execution loops also need termination. Bound tool waits with deadlines, represent timeout and cancellation explicitly, and set a maximum iteration or attempt count. A missing tool result must not leave subsequent work queued forever. Recovery commands should remain reachable outside the blocked queue, and receipts should record the terminal state that later steps can trust.

Part VI — Failure and recovery

Failures need distinct meanings

A dependable interface preserves why an attempt did not produce a completed effect. Malformed generation, schema rejection, semantic invalidity, authorization denial, dependency unavailability, domain failure, timeout, cancellation, and executor defects make different claims. They differ especially in whether an external effect is known to have occurred and who may safely decide the next step.

Failure and response

FailureKnown effect?Safe response
Malformed or incomplete generationNo action should have been dispatched.Reject; optionally regenerate within a bounded budget.
Schema or semantic rejectionNo action should have been dispatched.Return precise field or invariant feedback; let the model revise only permitted arguments.
Authorization denialNo authorized effect.Terminate or request a genuinely new authorization decision; do not retry around policy.
Dependency unavailable before dispatchUsually no effect, if the boundary can prove execution never began.Retry with bounded backoff when the operation is safe and policy allows.
Tool-declared domain failureDefined by the tool result.Preserve the domain code; revise only if the request can validly change.
Timeout after dispatchPossibly unknown.Reconcile or repeat with the same enforced operation identity; never assume failure.
Cancellation requestedPossibly unknown or partial.Inspect terminal state; cancellation acknowledgement is not rollback.
Executor defectDepends on where the defect occurred.Stop, preserve evidence, and investigate before blind replay.

Error envelopes should expose stable machine fields—such as type, code, affected argument locations, retry guidance, and an occurrence identifier—while keeping stack traces, secrets, internal capabilities, and sensitive policy details out of model-visible text. RFC 9457 is one reference for machine-readable HTTP problems and explicitly separates human-readable detail from fields clients should use for decisions. It does not define universal retryability or effect status, so those remain application-specific.

Validation feedback can support bounded correction. Pydantic demonstrations return an explanatory validator error to the model, which then proposes a corrected result. Instructor documents choosing retryable exceptions, stopping by attempts or elapsed time, and exposing attempt history after exhaustion. This is appropriate for correctable representation or value errors; it does not authorize a previously denied action or make a side effect safe to repeat.

Recover when the effect is unknown

A timeout after dispatch creates three possible realities: confirmed success, confirmed failure without an effect, or an unknown outcome. The client’s missing response cannot distinguish them. Retrying with a fresh identity can create a second effect if the first request committed. Preserve one logical operation identity and record the attempt as pending or unknown until provider evidence resolves it.

Idempotency means repeating the same logical operation adds no further externally visible effect. It requires enforcement at the receiving boundary, not merely a key written to local logs. Stripe’s documented contract illustrates bounded semantics: the caller reuses one key with identical parameters; executed responses, including some failures, are stored; mismatched parameters are rejected; validation failures or concurrent-execution conflicts may not create a stored result; and old keys can be pruned. Those details are provider-specific, so clients must implement the actual service contract rather than a generic “idempotent” checkbox.

Recovery after a lost mutation response

Safe recovery preserves logical operation identity and seeks receiving-boundary enforcement or authoritative evidence before another effect.

Logical operation op-17One immutable intent and parameter set.
Remote mutationThe service may commit before the client receives a response.
Outcome unknownThe response was lost; success and no-effect failure remain possible.
Retry with new identityThe service cannot associate the attempt with op-17.
Repeat original keyUse the provider’s idempotency contract with identical parameters.
Query authoritative stateSearch by operation metadata, provider handle, or resulting object.
Possible duplicate effectA second mutation can occur if the first committed.
Deduplicated resultThe provider returns the stored result for the same logical operation.
Outcome confirmedAuthoritative state establishes success or no effect.
Unresolved escalationNo safe automated conclusion or retry is available.
A remote mutation may commit before its response is lost. Retrying under a new identity can create a duplicate. Reusing the original provider-enforced key can return a stored result or preserve an unknown state according to the provider contract. Querying authoritative state can confirm success, confirm no effect, or leave the case unresolved for escalation.
Read the diagram as text
  • Logical operation op-17. One immutable intent and parameter set.
  • Remote mutation. The service may commit before the client receives a response.
  • Outcome unknown. The response was lost; success and no-effect failure remain possible.
  • Retry with new identity. The service cannot associate the attempt with op-17.
  • Repeat original key. Use the provider’s idempotency contract with identical parameters.
  • Query authoritative state. Search by operation metadata, provider handle, or resulting object.
  • Possible duplicate effect. A second mutation can occur if the first committed.
  • Deduplicated result. The provider returns the stored result for the same logical operation.
  • Outcome confirmed. Authoritative state establishes success or no effect.
  • Unresolved escalation. No safe automated conclusion or retry is available.
  • Logical operation op-17 → Remote mutation: submit with stable identity.
  • Remote mutation → Outcome unknown: response lost.
  • Outcome unknown → Retry with new identity: unsafe fresh identity.
  • Retry with new identity → Possible duplicate effect: first may have committed.
  • Outcome unknown → Repeat original key: provider enforces key.
  • Repeat original key → Deduplicated result: stored result exists.
  • Repeat original key → Unresolved escalation: contract cannot resolve.
  • Outcome unknown → Query authoritative state: authoritative reconciliation.
  • Query authoritative state → Outcome confirmed: state identifies outcome.
  • Query authoritative state → Unresolved escalation: evidence remains incomplete.

When provider deduplication or reliable result replay is unavailable, query authoritative state using a local operation identifier or provider job handle. Keep the local operation pending while evidence is incomplete. If a known effect must be countered, compensation performs a new business operation; it is not time travel or database rollback, and it can fail. Workflow replay likewise does not create exactly-once effects: a task may complete an external mutation and crash before its result is persisted, causing a resumed attempt to execute again.

This recovery logic specializes Represent uncertain external outcomes. Strategy-level repair after the effect is known belongs in Repair the strategy from known effects. The immediate boundary rule is simpler: do not transform missing evidence into a fabricated success or confirmed failure.

Part VII — Boundary evidence

Test each claim separately

A complete system needs complementary tests because each boundary supports a different claim. Valid JSON does not establish schema conformance; schema conformance does not establish correct disposition or arguments; correct arguments do not establish authorization; successful dispatch does not establish the external postcondition. Specify the case and its outcome explains how to preserve initial state, permitted actions, stopping conditions, and assessment criteria.

Claim-to-check map

ClaimCheckRemaining gap
The engine enforces the schema.Positive, negative, recursive, unsupported-keyword, and completion cases against the pinned runtime.Semantic correctness and provider behavior outside tested cases.
The parser and validators behave correctly.Replay recorded malformed, duplicate-field, truncated, coercible, and domain-invalid responses.Whether the live model emits them at deployment rates.
The system chooses appropriately.Evaluate answer, clarify, abstain, unavailable capability, right tool, and argument correctness separately.Authorization and external outcome.
The dispatcher is correct.Use controlled model substitutes to emit exact calls; verify allowlist, correlation, ordering, timeout, and error mapping.Real-model selection quality.
Authorization is enforced.Attempt forbidden actor-operation-resource combinations, changed arguments, expired approvals, and revoked scope.Completeness of the organization’s policy.
Retries avoid duplicate effects.Inject lost responses, crashes after external commit, concurrent attempts, expired keys, and reconciliation failures.Provider behavior beyond the tested contract.
The intended effect occurred.Inspect authoritative external state or a user-visible boundary.Whether the whole workflow was useful.

Tool evaluations should preserve intermediate expectations. Golden cases can specify whether a call is needed, the intended operation, required arguments, expected tool observation, and acceptable final response. BFCL V3 separately checks resulting backend state and required execution-path calls because a correct end state can omit necessary information gathering, while a correct path can still leave the wrong state. Several trajectories may be valid, so assert required properties rather than one arbitrary call order unless order is itself contractual.

Report stage-specific denominators: parse failures, schema rejections, semantic rejections, incorrect selections, incorrect arguments, authorization violations, executor failures, unknown outcomes, duplicate effects, verified postconditions, and completed tasks. A single “validity rate” hides where risk remains. Representative task sampling and oracle design belong in Choose checks that match the requirement.

Part VIII — Interoperability

Where MCP meets the local action boundary

Tool calling and the Model Context Protocol solve orthogonal problems. Tool calling is the model–application interface for proposing an invocation and returning observations. MCP standardizes capability exchange among a host, its clients, and servers. Servers can expose tools, resources, and prompts; a host can aggregate them and decide what to present to a model. In the 2026-07-28 architecture, each client communicates with one server and requests carry version and capability metadata.

MCP tool definitions contain a name, description, input schema, and optional output schema. Discovery and invocation standardization reduce bespoke integration work, but they do not decide whether a tool is appropriate, whether its description is trustworthy, or whether the current actor may execute it. Names are scoped to a server, catalogs can change with authorization, execution failures differ from protocol errors, and annotations are untrusted unless their server is trusted.

The host therefore retains the local control boundary: select which server capabilities enter the model’s context; disambiguate names; validate arguments and returned data; obtain confirmation where required; enforce authorization; impose timeouts; log use; dispatch according to policy; and translate results back into the model conversation. The server separately enforces access control and validates its inputs. Connecting a server advertises capability—it does not grant authority.

Keep protocol revision boundaries explicit. Older talks and implementations may describe connection-scoped initialization, historical transports, or earlier OAuth behavior. The current protocol chapter should own those version-specific details. The invariant for this chapter is stable: interoperability can supply descriptions and messages, while the application remains responsible for turning a model proposal into a permitted, controlled, and observed effect.

Open questions

  1. How can constrained decoders implement a provider’s advertised schema subset exactly while preserving acceptable latency across tokenizers, recursive structures, batching modes, and streaming? Progress would include versioned conformance suites that separate schema acceptance, recognized-language equivalence, completion behavior, and semantic task quality.

  2. How should systems measure whether tool names, descriptions, parameter semantics, prerequisites, and catalog size improve selection without consuming excessive context or encouraging unnecessary calls? Progress would require controlled, task-representative studies that separate no-call decisions, tool identity, and argument accuracy.

  3. What portable failure envelope can distinguish malformed proposals, policy denials, domain failures, pending work, cancellation, and unknown effects without leaking sensitive diagnostics or pretending that retry semantics are universal? Progress would look like precise, machine-readable effect knowledge and recovery guidance that tool implementations can map to their real contracts.

  4. How can authorization remain bound to exact user intent as arguments, resource state, delegated scope, and long-running execution change? Progress would combine transaction-specific approval, per-request service enforcement, revocation, and auditable identity without treating model reasoning or credentials as authority.

  5. How should a system verify semantic correctness when deterministic validators cover only known invariants and model judges share the generator’s blind spots? Progress would require authoritative domain evidence, calibrated deferral policies, and evaluation designs that keep oracle limitations visible instead of collapsing them into one confidence field.

Follow the curated reading path through the speakers and demonstrations behind this entry.

16 min

AI Engineer World's Fair 2024 · 2024

No more bad outputs with structured generation

Rémi Louf

Cited in this entry

Rémi Louf gives the clearest mechanism-level account of masking token continuations during structured generation and reports implementation-specific efficiency and benchmark results that are useful when read with their stated limits.

Watch talk
18 min

AI Engineer Summit 2023 · 2023

Pydantic is all you need

Jason Liu

Cited in this entry

Jason Liu shows how typed models, JSON Schema, field descriptions, nested structures, validation, and bounded re-asking turn model output into a maintainable application interface.

Watch talk
103 min

AI Engineer Summit 2025 · 2025

Function Calling is All You Need

Ilan Bigio

Cited in this entry

Ilan Bigio demonstrates the application-managed tool loop and makes the crucial ownership boundary concrete: the model expresses an intended function call, while application code executes it and returns the result.

Watch talk
18 min

AI Engineer World's Fair 2026 · 2026

Your Agent Didn’t Fail. Your Harness Did.

Vinoth Govindarajan

Cited in this entry

Vinoth Govindarajan extends the interface into production operations by separating proposals, scoped approvals, execution attempts, terminal outcomes, and user-visible evidence.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

198 matching talks

TalkSpeakerEventYear
Jason LiuAI Engineer World's Fair 20242024
Samuel ColvinAI Engineer World's Fair 20252025
Charlie GuoAI Engineer World's Fair 20252025
Ishan AnandAI Engineer World's Fair 20242024
AI Engineering 101

Transcript reviewed

Noah HeinAI Engineer Summit 20232023
Jamie Neuwirth, Zack WittenAI Engineer World's Fair 20242024
Daniel ChalefAI Engineer World's Fair 20262026
Vinoo GaneshAI Engineer World's Fair 20262026
Erik MeijerAI Engineer World's Fair 20262026
Sumaiya ShrabonyAI Engineer World's Fair 20262026
Fuzzing in the GenAI Era

Transcript reviewed

Leonard TangAI Engineer World's Fair 20252025
Frank CoyleAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Roy DerksAI Engineer Summit 20252025
Bobby Tiernay, Kam SweenAI Engineer World's Fair 20252025
Skills are the New SDKs

Transcript reviewed

Elvin AghammadzadaAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20252025
Kent C. DoddsAI Engineer World's Fair 20252025
John WelshAI Engineer World's Fair 20252025
Nimrod HauserAI Engineer Europe 20262026
Identity for AI Agents

Transcript reviewed

AI Engineer Code 20252025
Kim MaidaAI Engineer World's Fair 20262026
Liam McGarrigleAI Engineer Europe 20262026
Cornelia DavisAI Engineer Code 20252025
Erik HanchettAI Engineer World's Fair 20262026
Shaan DesaiAI Engineer Summit 20252025
Aditya BhargavaAI Engineer World's Fair 20262026
Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Steven WillmottAI Engineer Europe 20262026
Mahesh MuragAI Engineer Summit 20252025
Rene BrandelAI Engineer World's Fair 20252025
Pietro ZulloAI Engineer World's Fair 20262026
Aparna DhinakaranAI Engineer World's Fair 20252025
Nico AlbaneseAI Engineer Summit 20252025
Dippu Kumar SinghAI Engineer Europe 20262026
Leonie MonigattiAI Engineer Europe 20262026
Michael HablichAI Engineer Europe 20262026
Kam LasaterAI Engineer Summit 20252025
Sarthak AggarwalAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Jon PeckAI Engineer World's Fair 20252025
David CramerAI Engineer World's Fair 20252025
Zach BlumenfeldAI Engineer World's Fair 20252025
Cedric ClyburnAI Engineer World's Fair 20262026
Bilge YücelAI Engineer Europe 20262026
Dex HorthyAI Engineer World's Fair 20252025
A Genius With Amnesia

Metadata candidate

Victor SavkinAI Engineer World's Fair 20262026
Shelby HeineckeAI Engineer World's Fair 20242024
Tim AingeAI Engineer World's Fair 20262026
A Song of Types and Agents

Metadata candidate

Roberto StagiAI Engineer World's Fair 20262026
Ari HeljakkaAI Engineer Summit 20252025
Bala RamdossAI Engineer World's Fair 20262026
Cedric VidalAI Engineer World's Fair 20252025
Sam BhagwatAI Engineer World's Fair 20252025
Christopher ChedeauAI Engineer World's Fair 20252025
Charles FryeAI Engineer Summit 20232023
swyxAI Engineer World's Fair 20242024
Nick Nisi, Zack ProserAI Engineer World's Fair 20252025
AI’s Jurassic Park Period

Metadata candidate

Aaron StanleyAI Engineer World's Fair 20262026
Apoorva JoshiAI Engineer World's Fair 20262026
Frank CoyleAI Engineer World's Fair 20262026
Lance MartinAI Engineer World's Fair 20242024
Sina ShahandehAI Engineer World's Fair 20262026
Arjun Chintapalli, Bhavani KalisettyAI Engineer Summit 20252025
Filip KozeraAI Engineer World's Fair 20252025
Grace IsfordAI Engineer Summit 20252025
Rajiv ChandegraAI Engineer World's Fair 20262026
Łukasz GandeckiAI Engineer World's Fair 20252025
Paul HenryAI Engineer World's Fair 20242024
Angus J. McLeanAI Engineer Europe 20262026
Greg BensonAI Engineer World's Fair 20252025
Paul Klein IVAI Engineer World's Fair 20262026
Build Systems, Not Code

Metadata candidate

Angie JonesAI Engineer World's Fair 20262026
Raj NavakotiAI Engineer Europe 20262026
Louis-François Bouchard, Paul Iusztin, Samridhi VaidAI Engineer Europe 20262026
Will BrykAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20252025
Bennet FennerAI Engineer Europe 20262026
Ben KusAI Engineer World's Fair 20252025
Eugene YanAI Engineer Summit 20232023
Anoop Kotha, Toki SherbakovAI Engineer World's Fair 20252025
Simrat HanspalAI Engineer Summit 20232023
Michael FesterAI Engineer World's Fair 20252025
Anju KambadurAI Engineer Summit 20252025
Sunil PaiAI Engineer Europe 20262026
Dhruv BatraAI Engineer World's Fair 20262026
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Zeke SikelianosAI Engineer World's Fair 20252025
Ornella Bahidika, Joel AllouAI Engineer World's Fair 20262026
Laurie VossAI Engineer World's Fair 20252025
Kevin HouAI Engineer World's Fair 20242024
Maxime LabonneAI Engineer Europe 20262026
Theo BrowneAI Engineer World's Fair 20262026
Ankur GoyalAI Engineer World's Fair 20252025
Rafael LeviAI Engineer Europe 20262026
Samuel ColvinAI Engineer Code 20252025
Thor SchaeffAI Engineer Europe 20262026
Jerry LiuAI Engineer World's Fair 20242024
Kenton VardaAI Engineer World's Fair 20262026
Gateways are All You Need

Metadata candidate

Karan SampathAI Engineer Europe 20262026
Emil EifremAI Engineer World's Fair 20242024
Jonathan LarsonAI Engineer World's Fair 20252025
Iman MakaremiAI Engineer World's Fair 20252025
Rashi AgrawalAI Engineer World's Fair 20262026
Tanmai GopalAI Engineer World's Fair 20242024
Vasant KearneyAI Engineer World's Fair 20262026
Vinoo GaneshAI Engineer World's Fair 20262026
Patrick DoughertyAI Engineer Summit 20252025
Yogendra MirajeAI Engineer World's Fair 20252025
Jared HansonAI Engineer World's Fair 20252025
Ben KunkleAI Engineer Europe 20262026
Mustafa Ali, Kyle CorbittAI Engineer Summit 20252025
Sally-Ann DeLuciaAI Engineer Europe 20262026
Amol KapoorAI Engineer World's Fair 20262026
Imagination Engineering

Metadata candidate

Eve BouffardAI Engineer World's Fair 20262026
Yu SuAI Engineer World's Fair 20262026
Intro to GraphRAG

Metadata candidate

Zach BlumenfeldAI Engineer World's Fair 20252025
Robert ChandlerAI Engineer World's Fair 20252025
Tom SmokerAI Engineer World's Fair 20252025
Danilo CamposAI Engineer Europe 20262026
Dat NgoAI Engineer Europe 20262026
Thierry Moreau, Pedro TorruellaAI Engineer World's Fair 20242024
Hubert MisztelaAI Engineer World's Fair 20242024
Shafik Quoraishee, Joanne SongAI Engineer World's Fair 20262026
Kelvin MaAI Engineer World's Fair 20252025
Lin Qiao, Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Eashan SinhaAI Engineer World's Fair 20252025
Ronan McGovernAI Engineer World's Fair 20252025
MCP is all you need

Metadata candidate

Samuel ColvinAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer World's Fair 20262026
Mentoring the Machine

Metadata candidate

Eric HouAI Engineer World's Fair 20252025
Ilan BigioAI Engineer World's Fair 20252025
Ola MabadejeAI Engineer World's Fair 20252025
Rami AlhamadAI Engineer World's Fair 20252025
Ahmed MenshawyAI Engineer World's Fair 20242024
Sharif ShameemAI Engineer World's Fair 20252025
Lech KalinowskiAI Engineer World's Fair 20262026
Jeronim MorinaAI Engineer World's Fair 20242024
Samuel ColvinAI Engineer Europe 20262026
Pragmatic AI With TypeChat

Metadata candidate

Daniel RosenwasserAI Engineer Summit 20232023
Luke AlvoeiroAI Engineer Europe 20262026
Ben FlastAI Engineer World's Fair 20242024
Yuval Belfer, Niv GranotAI Engineer World's Fair 20252025
RAG for VPs of AI

Metadata candidate

Jerry LiuAI Engineer World's Fair 20242024
Rewiring the State

Metadata candidate

Eoin MulgrewAI Engineer Europe 20262026
Max RyabininAI Engineer Europe 20262026
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Scaffold Wisely

Metadata candidate

Rahul SengottuveluAI Engineer Summit 20252025
Onur SolmazAI Engineer Europe 20262026
Scaling Compute on Context

Metadata candidate

Jack MorrisAI Engineer World's Fair 20262026
Al HarrisAI Engineer Code 20252025
Rob CheungAI Engineer World's Fair 20242024
Barr YaronAI Engineer World's Fair 20252025
Tara AgyemangAI Engineer Europe 20262026
Kevin Madura, Mo BhasinAI Engineer World's Fair 20252025
Travis FrisingerAI Engineer World's Fair 20252025
The End of Apps

Metadata candidate

KitzeAI Engineer Europe 20262026
Dylan PatelAI Engineer World's Fair 20252025
Ofer MendelevitchAI Engineer Summit 20252025
Almog BakuAI Engineer Summit 20252025
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Diego Rodriguez, Eugene, Jonas Bauer, Shijia Liao, David Vorick, Alex AtallahAI Engineer World's Fair 20252025
Ted JohnsonAI Engineer World's Fair 20262026
Omer PrimorAI Engineer World's Fair 20262026
Amir HaghighatAI Engineer World's Fair 20252025
Cormac BrickAI Engineer Europe 20262026
Trust, but Verify

Metadata candidate

Shreya RajpalAI Engineer Summit 20232023
Useful General Intelligence

Metadata candidate

Danielle PerszykAI Engineer World's Fair 20252025
Jeff SchomayAI Engineer Summit 20232023
Anna Marie BenzonAI Engineer World's Fair 20262026
Harald KirschnerAI Engineer World's Fair 20252025
Vision: Zero Bugs

Metadata candidate

Johann Schleier-SmithAI Engineer Code 20252025
Moritz JohnerAI Engineer World's Fair 20262026
Nicholas ArcolanoAI Engineer Code 20252025
DottaAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20252025
Alex AlbertAI Engineer World's Fair 20242024
Dmitry PetrovAI Engineer World's Fair 20262026
Sam JulienAI Engineer World's Fair 20252025
Cormac BrickAI Engineer World's Fair 20262026
Dr. Jasper ZhangAI Engineer World's Fair 20252025
Jesús BarrasaAI Engineer World's Fair 20252025
Zach BlumenfeldAI Engineer Europe 20262026
Ari HeljakkaAI Engineer World's Fair 20252025
Chin Keong LamAI Engineer World's Fair 20252025
Balázs HorváthAI Engineer World's Fair 20262026
Rachel Lee Nabors (RL Nabors)AI Engineer Europe 20262026
Hamza TahirAI Engineer World's Fair 20262026
Zack ProserAI Engineer Europe 20262026
Ramana Siddanth EmaniAI Engineer World's Fair 20262026
Tun Shwe, Jeremy FrenayAI Engineer Europe 20262026
AI Engineer Summit 20252025
Jay Sahnan, Arik BirdEvent unknownUnknown
Pamela FoxAI Engineer World's Fair 20262026
Ivan LeoAI Engineer World's Fair 20262026
Dhruv NathawaniAI Engineer World's Fair 20262026
Gus IwanagaEvent unknownUnknown
Venky BEvent unknownUnknown

References

Coverage and source review
Processed transcripts
46 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
156 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Function Calling is All You Need

    https://www.youtube.com/watch?v=KUEmEb71vzQ&t=678

  2. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=793

  3. JSON Schema: Object reference

    https://json-schema.org/understanding-json-schema/reference/object

  4. JSON Schema 2020-12 Validation Primitives

    https://json-schema.org/draft/2020-12/json-schema-validation

  5. JSON Schema Dialect Declaration

    https://json-schema.org/understanding-json-schema/reference/schema

  6. JSON Schema: Annotations

    https://json-schema.org/understanding-json-schema/reference/annotations

  7. JSON Schema Array Items and Tuples

    https://json-schema.org/understanding-json-schema/reference/array

  8. Structured model outputs — OpenAI API

    https://developers.openai.com/api/docs/guides/structured-outputs

  9. Structured model outputs: supported contracts and incomplete responses

    https://developers.openai.com/api/docs/guides/structured-outputs

  10. Revised Report on the Algorithmic Language ALGOL 60

    https://www.cs.tufts.edu/comp/150FP/archive/peter-naur/algol60.pdf

  11. Implementing Remote Procedure Calls — Birrell and Nelson

    https://www.cs.cmu.edu/~15712/papers/birrell84.pdf

  12. JSON Schema Media Type — draft-zyp-json-schema-01

    https://datatracker.ietf.org/doc/html/draft-zyp-json-schema-01

  13. A Syntactic Neural Model for General-Purpose Code Generation

    https://aclanthology.org/P17-1041.pdf

  14. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

    https://aclanthology.org/2021.emnlp-main.779.pdf

  15. Toolformer: Language Models Can Teach Themselves to Use Tools

    https://arxiv.org/abs/2302.04761

  16. Function calling and other API updates — OpenAI

    https://openai.com/index/function-calling-and-other-api-updates/

  17. Efficient Guided Generation for Large Language Models

    https://arxiv.org/html/2307.09702

  18. Transformers Generation Configuration and Termination

    https://huggingface.co/docs/transformers/en/main_classes/text_generation

  19. Python json: Decoder behavior and interoperability

    https://docs.python.org/3/library/json.html

  20. RFC 8259: JSON object member uniqueness

    https://www.rfc-editor.org/rfc/rfc8259.html

  21. Pydantic: JSON parsing and partial JSON parsing

    https://pydantic.dev/docs/validation/latest/concepts/json/

  22. TypeScript Handbook: The Basics

    https://www.typescriptlang.org/docs/handbook/2/basic-types.html

  23. Introducing Structured Outputs: dynamic constrained decoding

    https://openai.com/index/introducing-structured-outputs-in-the-api/

  24. No more bad outputs with structured generation

    https://www.youtube.com/watch?v=aNmfvN6S_n4&t=435

  25. JSON String Escapes and Structural Recognition

    https://www.rfc-editor.org/rfc/rfc8259.html

  26. XGrammar Token Alignment and Cached Masks

    https://arxiv.org/html/2411.15100v1

  27. Grammar-Aligned Decoding

    https://arxiv.org/html/2405.21047v1

  28. Gemini API: Structured outputs

    https://ai.google.dev/gemini-api/docs/structured-output

  29. llama.cpp: Grammars and JSON Schema conversion

    https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md

  30. Generating Structured Outputs from Language Models: Benchmark and Studies

    https://arxiv.org/html/2501.10868v2

  31. The Hidden Cost of Structure: How Constrained Decoding Affects Language Model Performance

    https://aclanthology.org/2025.ranlp-1.124/

  32. OWASP Input Validation Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html

  33. Pydantic is STILL all you need

    https://www.youtube.com/watch?v=pZ4DIH2BVqg&t=366

  34. Design by Contract Introduction

    https://www.eiffel.com/values/design-by-contract/introduction/

  35. NIST SP 800-162: Guide to Attribute Based Access Control Definition and Considerations

    https://csrc.nist.gov/pubs/sp/800/162/upd2/final

  36. Human seeded Evals — Samuel Colvin, Pydantic

    https://www.youtube.com/watch?v=o_LRtAomJCs&t=282

  37. Human seeded Evals — Samuel Colvin, Pydantic

    https://www.youtube.com/watch?v=o_LRtAomJCs&t=309

  38. "I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox

    https://www.youtube.com/watch?v=-CnA2lGfymY&t=467

  39. Tools — Model Context Protocol specification 2025-06-18

    https://modelcontextprotocol.io/specification/2025-06-18/server/tools

  40. Tool Calling Is Not Just Plumbing for AI Agents

    https://www.youtube.com/watch?v=zuMw0pkPXpU&t=364

  41. Agentic Search for Context Engineering

    https://www.youtube.com/watch?v=ynJyIKwjonM&t=660

  42. Writing effective tools for agents — with agents

    https://www.anthropic.com/engineering/writing-tools-for-agents

  43. AI Engineering with the Google Gemini 2.5 Model Family

    https://www.youtube.com/watch?v=zK9lYrLbjSg&t=5598

  44. When2Call: When (not) to Call Tools

    https://aclanthology.org/anthology-files/anthology-files/pdf/naacl/2025.naacl-long.174.pdf

  45. Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents

    https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=465

  46. Function calling with the Gemini API

    https://ai.google.dev/gemini-api/docs/function-calling

  47. Let's Build an Agent from Scratch — Kam Lasater

    https://www.youtube.com/watch?v=xzXdLRUyjUg&t=483

  48. Function Calling is All You Need

    https://www.youtube.com/watch?v=KUEmEb71vzQ&t=1225

  49. Let's Build an Agent from Scratch — Kam Lasater

    https://www.youtube.com/watch?v=xzXdLRUyjUg&t=294

  50. Vercel AI SDK Masterclass: From Fundamentals to Deep Research

    https://www.youtube.com/watch?v=kDlqpN1JyIw&t=1002

  51. OWASP Authorization Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html

  52. How we hacked YC Spring 2025 batch’s AI agents

    https://www.youtube.com/watch?v=kv-QAuKWllQ&t=281

  53. OWASP Transaction Authorization Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html

  54. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=717

  55. It's 10pm. Do You Know Where Your Agents Are?

    https://www.youtube.com/watch?v=I3znWC3MEXM&t=456

  56. Function calling

    https://developers.openai.com/api/docs/guides/function-calling

  57. Tool use with Claude

    https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview

  58. MCP tools: application security responsibilities

    https://modelcontextprotocol.io/specification/2025-06-18/server/tools

  59. AIP-151: Long-running operations

    https://google.aip.dev/151

  60. RFC 9110: Conditional requests and successful response semantics

    https://www.rfc-editor.org/rfc/rfc9110.html

  61. GitHub REST API: Create an issue

    https://docs.github.com/en/rest/issues/issues

  62. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=608

  63. Your Agent Is Wasting Tokens and You Don't Know It - Erik Hanchett, AWS

    https://www.youtube.com/watch?v=uiP88SpCi1Q&t=161

  64. MCP Tasks extension: capability, lifecycle, and result retrieval

    https://tasks.extensions.modelcontextprotocol.io/specification/draft/tasks

  65. Instructor: Retry Logic with Tenacity

    https://python.useinstructor.com/concepts/retrying/

  66. RFC 9457: Problem Details for HTTP APIs

    https://www.rfc-editor.org/rfc/rfc9457.html

  67. Resolving an ambiguous payment request

    https://docs.stripe.com/error-low-level

  68. Making Retries Safe with Idempotent APIs

    https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

  69. Stripe retry keys and retention boundaries

    https://docs.stripe.com/api/idempotent_requests

  70. Idempotent requests

    https://docs.stripe.com/api/idempotent_requests

  71. Temporal Activity Definition: Idempotency

    https://docs.temporal.io/activity-definition

  72. LangGraph Functional API: deterministic resumption and idempotency

    https://docs.langchain.com/oss/python/langgraph/functional-api

  73. BFCL V3: Multi-Turn and Multi-Step Function Calling

    https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html

  74. Building enterprise LLM agents that work

    https://www.youtube.com/watch?v=4J8-D0sgU9A&t=715

  75. Pydantic AI: Unit testing

    https://pydantic.dev/docs/ai/guides/testing/

  76. Every Solo Agent Builder Eventually Reinvents a Worse Version of CI/CD

    https://www.youtube.com/watch?v=WLXxTaPagA8&t=112

  77. Model Context Protocol: Architecture, revision 2026-07-28

    https://modelcontextprotocol.io/specification/2026-07-28/architecture

  78. Model Context Protocol: Specification and trust principles, revision 2026-07-28

    https://modelcontextprotocol.io/specification/2026-07-28

  79. Model Context Protocol: Tools, revision 2026-07-28

    https://modelcontextprotocol.io/specification/2026-07-28/server/tools

  80. Remote MCPs: What we learned from shipping — John Welsh, Anthropic

    https://www.youtube.com/watch?v=0NHCyq8bBcM&t=104

  81. Letting AI Interface with Your App with MCP

    https://www.youtube.com/watch?v=EyZiAp0pelw&t=597

  82. Model Context Protocol: Transport overview, revision 2026-07-28

    https://modelcontextprotocol.io/specification/2026-07-28/basic/transports

  83. Pydantic is all you need

    https://www.youtube.com/watch?v=yj-wSRJwrrc&t=204

  84. AI Engineering with the Google Gemini 2.5 Model Family

    https://www.youtube.com/watch?v=zK9lYrLbjSg&t=5526

  85. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=882

  86. RFC 9396: OAuth 2.0 Rich Authorization Requests

    https://www.rfc-editor.org/rfc/rfc9396.html

  87. Language Models (Mostly) Know What They Know

    https://arxiv.org/html/2207.05221v4

  88. No more bad outputs with structured generation

    https://www.youtube.com/watch?v=aNmfvN6S_n4&t=576

  89. Pydantic is all you need

    https://www.youtube.com/watch?v=yj-wSRJwrrc&t=364

  90. Pydantic is all you need

    https://www.youtube.com/watch?v=yj-wSRJwrrc&t=480