AI Engineer World's Fair 2026

The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

Read the talk

The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral

Candidate for the model-switching setup described in the workshop.Open full source frame
Candidate for the model-switching setup described in the workshop.

Pamela Fox compares models on counting, grounded answers, images and agent behavior, then builds evaluations and uses prompt optimization to make a difficult task work with a different model.

From a talk by Pamela Fox

At a glance

Ideas worth remembering

  • Test the actual scenario: a model that counts incorrectly can still handle the missing-source RAG case, and a successful text prompt does not establish image capability.

  • Evaluate tool names, arguments, completion and efficiency. Independent searches can run in parallel, while budget-dependent work must wait for their results.

  • Give judges the context and traces needed for their task, then calibrate their decisions with knowledgeable humans.

  • ASSERT can bootstrap task-specific cases from a specification; DSPy can search for prompts against explicit checks. Both depend on the quality of the requirements and evaluation.

Start with a model switch and a counting failure

A model can look strong on a benchmark and still fail at a task your application depends on. Pamela Fox’s model-swap workshop tests that problem with four deployments on Microsoft Foundry: GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3. The examples use the OpenAI Python SDK with a Foundry endpoint and environment key; changing the selected model changes which deployment receives the same request.

Mistral’s revised counting prompt and result are discussed.Open full source frame
Mistral’s revised counting prompt and result are discussed.

The first prompt asks how many times E appears in “The elderly gentleman eagerly entered the elevator.” The answer is 13. In these runs, GPT-5.5 and Kimi return 13, DeepSeek returns 6, and Mistral returns 8. These are individual demonstrations, not estimates of reliability across repeated trials.

Fox then asks Mistral to break the sentence into words, count each word, and sum the counts. That run reaches 13. The change makes an intermediate operation explicit, giving the model a different route to the answer. It also establishes the workshop’s practical question: when a swap fails, can an instruction change recover the behavior the application needs?

2:413:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:11 · section reference included

A source can support half an answer

The retrieval-augmented generation example asks for differences between honeybee and carpenter-bee nest building, but the retrieved dataset contains only carpenter-bee information. The prompt asks the model to answer from the sources and cite them. A grounded answer can describe carpenter bees and say that the provided material does not establish the honeybee side of the comparison.

Fox discusses the response labeling honeybee details as unsourced.Open full source frame
Fox discusses the response labeling honeybee details as unsourced.

After a basic search, GPT-5.5 says the dataset lacks honeybee nest-construction details; DeepSeek also identifies the missing information. Mistral supplies honeybee details absent from the sources. Fox strengthens the instruction: mark unsupported information as having no source and do not make up an answer. The revised response identifies the honeybee details as unsourced, yet still provides them. That is closer to the desired behavior, but it does not satisfy the original constraint.

The distinction matters for evaluation: recognizing that a source is missing and withholding an unsupported claim are separate behaviors. A citation instruction can help expose the gap without necessarily closing it.

8:419:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:41 · section reference included

Image understanding changes the test

The image tests ask three different questions: is an animal a unicorn, are pictured reptiles alligators or crocodiles, and which plant is cheapest in a table with small text? The first animal is an aurochs; the reptiles are crocodiles, which Fox distinguishes by their snouts. The table requires reading and comparing prices.

What do the inputs demand of the models? GPT-5.5 and Kimi identify the crocodiles and find the cheapest plant. Mistral knows the first animal is not a unicorn but does not identify it as an aurochs, calls the crocodiles alligators, and selects a $2.30 plant when the cheapest costs $0.58. DeepSeek is excluded from this image test because the deployed model accepts text only.

Fox describes the plant-price table with small text.Open full source frame
Fox describes the plant-price table with small text.

A domain-specific hint might help a model read a particular kind of image, but Fox treats visual understanding as a capability to test directly on the images an application will receive. Language matters too: she reports that changing the crocodile question to Spanish produced a wrong answer. Her explanation of confusion between animal names is a hypothesis about the failure; the practical test is whether the complete image-and-language task works.

13:1913:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:49 · section reference included

Tool definitions are part of the prompt

A tool-calling model receives a function definition and a user request, then returns a proposed function name and arguments. The application is responsible for actually executing the function. In the calendar example, the definition uses JSON Schema to describe createCalendarEvent, including parameter types and descriptions. Those names and descriptions influence how the model interprets the request.

The request is for a virtual platform-team sync tomorrow at 1:30 p.m., with Sarah, Marcus and Priya; the system message supplies today’s date. Which arguments differ from the expected event? GPT-5.5 adds the requester as an attendee, which Fox did not want. DeepSeek chooses the right tool but also includes the requester and returns a location containing “virtual” and Microsoft Teams rather than the expected virtual. The schema can resolve some ambiguity by specifying that only named attendees should be included.

DeepSeek’s calendar arguments are compared with expected values.Open full source frame
DeepSeek’s calendar arguments are compared with expected values.

A loop adds another evaluation dimension: efficiency. A calculator tool accepts an expression and returns its result for a sale-and-tax word problem. GPT-5.5 puts the calculation into one expression and makes one tool call; DeepSeek reaches the correct answer using four calls. Each extra call adds a network request. Fox has also seen Kimi get the answer and then make four more calls checking it, although it behaves well in the live run.

17:0817:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:08 · section reference included

Parallel searches, then a budget-dependent activity

The LangChain v1 travel agent has tools to search flights, search hotels, check a budget and search activities. Its instructions say to search for both flights and hotels, prefer direct flights, check the budget and use the remaining amount for activities. The user wants a three-night trip from San Francisco to New York City next month, a direct flight and an activity, with a $600 total budget.

The travel-agent trace is discussed as budget and activity calls follow the parallel searches.Open full source frame
The travel-agent trace is discussed as budget and activity calls follow the parallel searches.

Which calls can happen together, and which need earlier results? In the GPT-5.5 run, flight and hotel searches are proposed in the same turn because neither depends on the other. Once both return, the agent checks their combined cost; the remaining budget then informs the activity search. The branching flow shows why parallelizing the first two calls can save time without skipping the budget calculation.

Kimi searches flights and hotels in separate turns and makes two budget checks, taking two extra tool calls overall. Fox does not know whether the lack of parallel calls reflects support or the model’s choice in this run. DeepSeek makes four calls without parallelizing. Mistral searches flights and hotels, then stops before the budget and activity steps. A final answer alone would hide these differences in completeness and execution; the tool trace makes them inspectable.

Fox also includes Pydantic AI and Agent Framework versions of the example. The framework’s tool loop—the harness—is another variable to compare alongside the model. For an agent that stops early, additional steering or a different loop may help, but it must be evaluated against the complete task.

Travel planning dependencies

Search flightsparallel — Find flight options from San Francisco to New York City.
Search hotelsparallel — Find New York City hotel options.
Check budgetUse the returned flight and hotel costs.
Search activitiesUse the remaining budget.
Flight and hotel searches can run together; their costs are needed before the remaining activity budget is known.
Read the diagram as text
  • Search flights. parallel — Find flight options from San Francisco to New York City.
  • Search hotels. parallel — Find New York City hotel options.
  • Check budget. Use the returned flight and hotel costs.
  • Search activities. Use the remaining budget.
  • Search flights → Check budget: Flight cost.
  • Search hotels → Check budget: Hotel cost.
  • Check budget → Search activities: Remaining budget.
23:5424:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:54 · section reference included

Turn observed failures into repeatable checks

Manual runs reveal failure modes; production comparisons need a collection of repeatable cases. For a single calendar call, deterministic checks can verify the tool name and compare returned arguments with expected strings or lists—for example, whether the location is virtual.

The calendar cases exercise date arithmetic, ambiguous durations, Spanish and slang. Fox recommends giving the model a date-calculation tool when dates matter, especially around leap years. As real users introduce unexpected phrasing, add those cases so the test set continues to reflect actual use. The workshop runs each case across the models and records pass or fail; these calls are slower and more expensive than ordinary local tests.

Simple string checks need little framework machinery. More complicated questions, such as whether an agent parallelized independent work, require richer trace evaluation. Fox names evaluation features in observability products and dedicated evaluation frameworks as options.

30:3131:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:31 · section reference included

Give the judge the evidence it needs

For less exact outputs, an LLM judge can receive an evaluation instruction and return pass or fail with a reason. A groundedness judge needs the question, answer and source context: without the context, it cannot compare the answer with what the application was allowed to use. A judge assessing tool efficiency needs the tool definitions and execution trace as well.

The demonstration uses GPT-5.5 as judge and requests JSON. Each fresh case can require an application model call plus a judge call; multiple judges add more calls. Fox says her production RAG evaluations often use at least 200 cases, sometimes 800, so frequency and cost become practical decisions.

Fox pre-runs the questions and answers for this example. Saving those inputs lets her revise the evaluator without rerunning the application each time. It also helps investigate a crucial ambiguity: a failure may belong to the application or to the judge. Calibrate the judge against labels from people who know the domain. In the small demonstration, most models score two out of three and Mistral one out of three; those three cases are a starting point for inspection, not a broad ranking.

34:5435:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

34:54 · section reference included

Bootstrap travel-agent tests from a specification

ASSERT starts from a description of the application, its intended behavior and its users. It develops behavior categories and judging rubrics, then generates scenarios for testing the target. This helps with the awkward period before launch: you need an evaluation set before you have a history of real user requests.

For the travel planner, Fox describes a YAML specification that names the judge and dimensions such as budget pressure. The author can predefine ample, tight and infeasible budgets, or let ASSERT propose the divisions. The agent is traced so evaluation can inspect its tool calls. Generated categories include budget infeasibility, direct-flight preference, rule violations and required tool use.

How do the models compare on those generated scenarios? The displayed results cover DeepSeek, Kimi and GPT-5.5; lower violation counts are better. GPT-5.5 has the fewest violations in this run, surprising Fox after Kimi’s generally strong demonstrations. The result concerns this generated travel-planner evaluation, and Fox notes that she is new to ASSERT.

The useful target is application-specific behavior: whether a travel agent respects a budget or uses required tools, rather than merely producing fluent prose. Generated tests and judges still need inspection and calibration against the behavior the application actually requires.

39:4040:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

39:40 · section reference included

Optimize instructions against explicit constraints

Hand-tuning a prompt for every model swap takes time. DSPy offers a different arrangement: declare the program’s inputs and desired behavior, supply checks, and use evaluation results to search for instructions that satisfy them. Fox describes using the GEPA optimizer with a student model and a reflection model to propose and assess prompt changes.

The demonstrated task is a numbered list whose lines each have an exact word count and whose first letters spell a target word. Checks determine whether an output meets those constraints, giving the optimizer feedback on candidate prompts. This requires a different way of writing the application: the specification and checks are inputs to the process, while the prompt becomes an output.

What instructions did the search produce for the model that struggled earlier? Fox opens a saved system prompt optimized for Mistral; she reports that the optimization took about six minutes and that the resulting prompt works for this task. The demonstrated result supports testing a cheaper model with optimized instructions; it does not establish that the same prompt or savings transfer to every task.

45:1145:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

45:11 · section reference included

A model swap may also change the API

Fox closes with an integration difference she had not demonstrated earlier. Anthropic models in her setup use the Messages API, so switching to them requires code changes. For RAG, she values their structured citation support: provide source documents in the expected format, enable citations, and receive text blocks with citation information that a frontend can extract and link.

The workshop also supports local experiments through Ollama by pointing examples at a local server and changing model names. The shared workshop keys were temporary, so continuing the exercises requires an available model endpoint and appropriate credentials or a local setup.

49:1049:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:10 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…