AI Engineer World's Fair 2024

Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

Loading slides…

Read the talk

Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

A useful insurance chatbot needs domain examples, visible execution traces, and trustworthy evaluations before modular programs and DSPy optimizers can improve it.

From a talk by Jeronim Morina

Before you start: Basic familiarity with LLM prompts and Python classes is useful; no prior DSPy experience is required.

When a plausible answer becomes the acceptance test

What problem are we actually solving when we spend the day trying new models, tools, and prompts? Jeronim Morina opens with a provocation: AI engineers have become too comfortable tinkering with tools and HotpotQA instead of solving real problems. His proposed way out is to return to engineering: define the problem and program an AI system around it.

Dark slide with the opening claim in white and the fully revealed response in red; speaker inset at lower left.Open full source frame
“We’re all bad AI Engineers”—“But there is a way out.”

A neural network becomes useful through the larger system that surrounds it. Yet excitement about model capabilities often precedes a clear account of what that system must achieve. Handwritten prompts and prompt libraries then accumulate around a vaguely defined goal, with the hope that another API will supply the missing solution.

Morina makes this concrete with an engineer’s morning routine: brew specialty coffee, settle into focused work, tweak prompts, and try another tool to make the output JSON. Then inspect the answers and approve them using the joke metric “Looks good to me” at ten. That may feel like progress, but it leaves the acceptance criterion in the engineer’s head. A prompt that seems stable is not yet a measured system. Moving beyond that routine starts with a problem worth solving.

0:190:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:19 · section reference included

The workload behind the chatbot

Insurance supplies a concrete setting for that work. Morina describes it as a business built around data and intangible protection: customers pay for a promise of financial support when something happens. Climate change brings increasing claims and more work serving those customers. AXA Germany founded its Data Innovation Lab in 2017; its machine learning engineers collaborate with business-unit data scientists to help customer service agents handle that work.

Forest-canopy background behind text about insurers’ data history, AI investment, and a cross-functional group of machine learning experts.Open full source frame
AXA’s “Pioneers in Data Mastery” slide introduces its Data Innovation Lab.

The organizational pace differs from a startup’s. Morina compares Silicon Valley companies to speedboats and large insurers to tankers—then refines the latter image to icebreakers, moving slowly but opening a path. He dates AXA’s work with OpenAI and its internal, Azure-hosted Secure GPT platform to early 2023. At the time of the talk, he also describes a Mistral AI collaboration announced that Monday. Expanding the platform’s usefulness had to preserve customer data security.

2:503:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:50 · section reference included

Define the chatbot through easy and hard examples

The walkthrough begins with an illustrative insurance problem: a customer-facing chatbot that helps people navigate policy terms and conditions. Before choosing its implementation, the team needs to define what useful behavior means. Domain experts guide that process by crafting examples at both extremes: questions that should be straightforward and questions that are unusually difficult. Those examples expose the domain’s boundaries and the expectations for an eventual production system.

Morina reports starting with about a dozen examples and immediately using them to prompt internal Secure GPT. This first round is an exploration of feasibility: can the model produce useful answers to the questions the domain experts care about? The examples make that investigation concrete before the system acquires more machinery.

5:235:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:23 · section reference included

Look inside the structured-output machinery

The next challenge is getting clean, structured output without treating prompting as an incantation. Morina rejects pleading for JSON or threatening to quit if the model fails to produce it. He describes a technique attributed to Jason Liu: begin an assistant response with an opening code fence and the json language label, steering the model to continue in that format. The mechanism is continuation from a supplied prefix, rather than another instruction asking for JSON.

Chain-of-thought, zero-shot and few-shot prompting, and libraries such as LangChain, Guardrails, DSPy, and Instructor offer different ways to shape behavior. The desired properties include safer, more deterministic, structured, and resilient output, sometimes optimized against a chosen metric. But finding a good prompt remains laborious when templates are hard-coded and library behavior is opaque.

That opacity makes request inspection part of the engineering work. Morina recommends Hamel Husain’s prompt-inspection article and its use of mitmproxy to examine what frameworks actually send to the model. Inspect the generated template and outgoing request before attributing an improvement to a library. The tool’s abstraction does not remove responsibility for understanding its prompt.

6:266:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:26 · section reference included

A feasible task inside a fragile system

Minor prompt changes caused large downstream problems. The team responded with chaining libraries and additional error handling, but those repairs accumulated into an overly complex, fragile system. Morina judged it unsuitable for production. It still established something useful: the task appeared achievable, and the prototype could serve as a baseline.

Batman slapping Robin meme with “SHOW ME THE PROMPT,” beside a matching slide heading and a hamel.dev link.Open full source frame
“Show me the prompt” challenges claims of library magic.

Inspecting individual requests did not explain the whole execution. The team was not logging traces, so it lacked visibility into how related calls fit together. Adding the open-source Arize Phoenix library helped reveal which calls were bundled together and what happened behind the abstractions. Request inspection exposed prompt contents; tracing exposed relationships among calls.

9:019:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:01 · section reference included

Turn production data into an evaluation

Visibility alone did not replace informal approval. The next requirement was a basic evaluation, which needed labels as well as inputs. Morina reports spending several months collecting and preparing existing production data into clean input-output pairs. The expensive part was establishing usable examples of expected behavior, not adding another model call.

Those pairs also needed protection against data leakage. If an input exposes information that already reveals its target label, an evaluation can reward access to the answer instead of the intended task. A modern API does not repair that defect in the dataset. Morina recommends Hamel Husain’s writing on evaluations, then makes the prerequisite explicit: understand the data and gather enough examples before moving to DSPy or another optimization tool.

10:1710:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:17 · section reference included

DSPy does not skip the foundations

Skipping those foundations produces the caricature Morina warns against: prompt engineering, then DSPy, then profit. His own interest came from meeting Omar the previous November. That encounter prompted the team to try DSPy, but enthusiasm did not make the transition effortless.

The starting point was a system with huge prompts and extensive error handling. Replacing it with DSPy seemed as though it should simplify everything; instead, the team encountered a steep learning curve. Morina encourages experimentation without offering an unconditional recommendation. The framework still requires learning how to express the problem in a form it can optimize.

Slide headed “Steep learning curve” with a butterfly meme labeled “DSPY” and “WOW IS THAT LEARNING CURVE STEEP.”Open full source frame
DSPy’s steep learning curve.
12:1212:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:12 · section reference included

Give the optimizer a program it can work with

The manual workflow already contained several distinct jobs: break the problem into steps, make each step work in isolation, adjust the steps to work together, and generate examples for tuning. Those examples might eventually support fine-tuning smaller models. DSPy retains the need for decomposition. Without meaningful modules, the optimizer has too little structure and signal to guide improvement.

The team’s attempt to express everything as one huge ReAct program was unsuccessful. A more useful design makes the flow explicit: retrieve information, perform additional retrieval if the question requires multiple hops, and combine the relevant pieces into a final answer. Each operation becomes a module with a defined role.

An optimizer then uses a supplied metric to tune the program’s prompts or weights. For this chatbot, Morina describes the objective as finding the right answer; he supplies neither a formal scoring rule nor a measured improvement. Compilation can produce different instructions, few-shot examples, or fine-tuning from the same program, depending on the optimization approach. The important separation is between the program’s structure and the choices being tuned inside it.

The PyTorch analogy makes that separation tangible. In __init__, declare components such as ChainOfThought or ReAct, much as a neural-network module declares convolution or dropout layers. In forward, specify how data moves through them, including retrieval. This Python example expresses that structure for the insurance chatbot: a supplied retriever returns policy passages, and an answer module consumes those passages with the customer’s question.

python

import dspy


class PolicyAnswer(dspy.Signature):
    """Answer the question using the supplied policy passages."""

    context: list[str] = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()


class InsuranceQA(dspy.Module):
    def __init__(self, retrieve):
        super().__init__()
        self.retrieve = retrieve
        self.answer = dspy.ChainOfThought(PolicyAnswer)

    def forward(self, question: str):
        passages = self.retrieve(question)
        return self.answer(context=passages, question=question)

The constructor defines the parts; forward defines their composition. Retrieval and answer generation now have explicit places in the program instead of being hidden inside a single enormous prompt.

13:1313:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:13 · section reference included

German text and task-specific modules need care

Two customization needs emerged in the team’s DSPy experience:

  • German evaluation: Morina found answer_exact_match and answer_passage_match unsuitable out of the box for the team’s German text. They needed their own metrics and evaluators, particularly for exact and passage matching.
  • Typed behavior: The version he describes offered TypedPredictor and TypedChainOfThought, but lacked the typed ReAct component they wanted. More specific requirements called for a custom module.

These are historical implementation observations. The Python example above uses the current typed-signature style; current DSPy also documents standard modules including ReAct. English-oriented normalization remains a reason to inspect matching behavior, rather than assuming either that a built-in metric fits German or that every German comparison necessarily fails.

The larger architectural lesson survived those API details: the team’s attempt at one big model did not work. A framework cannot supply useful boundaries for a task the engineer has not yet decomposed.

15:4715:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:47 · section reference included

Make the next improvement small enough to measure

The practical workflow begins before automated optimization:

  1. Write prompts by hand to determine whether the task is achievable at all.
  2. Prepare annotated examples and a clear goal metric so an apparent improvement has an objective test.
  3. Start with basic evaluations, such as regex checks or string comparisons, before introducing LLM-as-judge.
  4. Make very small adjustments so the system does not become harder to understand with every repair.

This sequence keeps evaluation proportional to the question being tested. A straightforward check should not become another elaborate AI subsystem without a reason.

A short talk cannot provide complete DSPy training. Morina points readers toward Connor Shorten’s learning materials and the DSPy website for that next step. His own curiosity grew through meeting the people who built these tools; the intended result is the same curiosity paired with first-principles engineering. The closing challenge is to find worthwhile problems that AI now makes possible to solve—and let those problems determine what to build.

16:5617:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:56 · section reference included

Resources

From the talk

  • A practical guide to domain-specific evaluation, including scoped tests, trace review, human evaluation, and model-based evaluation.

  • AXA's July 2023 announcement describes Secure GPT's Azure OpenAI foundation and its initial employee rollout.

  • Mistral describes its collaboration with AXA and the integration of its models into Secure GPT.

Updates since the talk

Transcript download
Full transcript

Loading transcript…