AI Engineer World's Fair 2024
Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning
Loading slides…
Read the talk
Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning
A useful insurance chatbot needs domain examples, visible execution traces, and trustworthy evaluations before modular programs and DSPy optimizers can improve it.
From a talk by Jeronim Morina
Before you start: Basic familiarity with LLM prompts and Python classes is useful; no prior DSPy experience is required.
When a plausible answer becomes the acceptance test
What problem are we actually solving when we spend the day trying new models, tools, and prompts? Jeronim Morina opens with a provocation: AI engineers have become too comfortable tinkering with tools and HotpotQA instead of solving real problems. His proposed way out is to return to engineering: define the problem and program an AI system around it.
A neural network becomes useful through the larger system that surrounds it. Yet excitement about model capabilities often precedes a clear account of what that system must achieve. Handwritten prompts and prompt libraries then accumulate around a vaguely defined goal, with the hope that another API will supply the missing solution.
Morina makes this concrete with an engineer’s morning routine: brew specialty coffee, settle into focused work, tweak prompts, and try another tool to make the output JSON. Then inspect the answers and approve them using the joke metric “Looks good to me” at ten. That may feel like progress, but it leaves the acceptance criterion in the engineer’s head. A prompt that seems stable is not yet a measured system. Moving beyond that routine starts with a problem worth solving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The workload behind the chatbot
Insurance supplies a concrete setting for that work. Morina describes it as a business built around data and intangible protection: customers pay for a promise of financial support when something happens. Climate change brings increasing claims and more work serving those customers. AXA Germany founded its Data Innovation Lab in 2017; its machine learning engineers collaborate with business-unit data scientists to help customer service agents handle that work.
The organizational pace differs from a startup’s. Morina compares Silicon Valley companies to speedboats and large insurers to tankers—then refines the latter image to icebreakers, moving slowly but opening a path. He dates AXA’s work with OpenAI and its internal, Azure-hosted Secure GPT platform to early 2023. At the time of the talk, he also describes a Mistral AI collaboration announced that Monday. Expanding the platform’s usefulness had to preserve customer data security.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define the chatbot through easy and hard examples
The walkthrough begins with an illustrative insurance problem: a customer-facing chatbot that helps people navigate policy terms and conditions. Before choosing its implementation, the team needs to define what useful behavior means. Domain experts guide that process by crafting examples at both extremes: questions that should be straightforward and questions that are unusually difficult. Those examples expose the domain’s boundaries and the expectations for an eventual production system.
Morina reports starting with about a dozen examples and immediately using them to prompt internal Secure GPT. This first round is an exploration of feasibility: can the model produce useful answers to the questions the domain experts care about? The examples make that investigation concrete before the system acquires more machinery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look inside the structured-output machinery
The next challenge is getting clean, structured output without treating prompting as an incantation. Morina rejects pleading for JSON or threatening to quit if the model fails to produce it. He describes a technique attributed to Jason Liu: begin an assistant response with an opening code fence and the json language label, steering the model to continue in that format. The mechanism is continuation from a supplied prefix, rather than another instruction asking for JSON.
Chain-of-thought, zero-shot and few-shot prompting, and libraries such as LangChain, Guardrails, DSPy, and Instructor offer different ways to shape behavior. The desired properties include safer, more deterministic, structured, and resilient output, sometimes optimized against a chosen metric. But finding a good prompt remains laborious when templates are hard-coded and library behavior is opaque.
That opacity makes request inspection part of the engineering work. Morina recommends Hamel Husain’s prompt-inspection article and its use of mitmproxy to examine what frameworks actually send to the model. Inspect the generated template and outgoing request before attributing an improvement to a library. The tool’s abstraction does not remove responsibility for understanding its prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A feasible task inside a fragile system
Minor prompt changes caused large downstream problems. The team responded with chaining libraries and additional error handling, but those repairs accumulated into an overly complex, fragile system. Morina judged it unsuitable for production. It still established something useful: the task appeared achievable, and the prototype could serve as a baseline.
Inspecting individual requests did not explain the whole execution. The team was not logging traces, so it lacked visibility into how related calls fit together. Adding the open-source Arize Phoenix library helped reveal which calls were bundled together and what happened behind the abstractions. Request inspection exposed prompt contents; tracing exposed relationships among calls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn production data into an evaluation
Visibility alone did not replace informal approval. The next requirement was a basic evaluation, which needed labels as well as inputs. Morina reports spending several months collecting and preparing existing production data into clean input-output pairs. The expensive part was establishing usable examples of expected behavior, not adding another model call.
Those pairs also needed protection against data leakage. If an input exposes information that already reveals its target label, an evaluation can reward access to the answer instead of the intended task. A modern API does not repair that defect in the dataset. Morina recommends Hamel Husain’s writing on evaluations, then makes the prerequisite explicit: understand the data and gather enough examples before moving to DSPy or another optimization tool.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
DSPy does not skip the foundations
Skipping those foundations produces the caricature Morina warns against: prompt engineering, then DSPy, then profit. His own interest came from meeting Omar the previous November. That encounter prompted the team to try DSPy, but enthusiasm did not make the transition effortless.
The starting point was a system with huge prompts and extensive error handling. Replacing it with DSPy seemed as though it should simplify everything; instead, the team encountered a steep learning curve. Morina encourages experimentation without offering an unconditional recommendation. The framework still requires learning how to express the problem in a form it can optimize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the optimizer a program it can work with
The manual workflow already contained several distinct jobs: break the problem into steps, make each step work in isolation, adjust the steps to work together, and generate examples for tuning. Those examples might eventually support fine-tuning smaller models. DSPy retains the need for decomposition. Without meaningful modules, the optimizer has too little structure and signal to guide improvement.
The team’s attempt to express everything as one huge ReAct program was unsuccessful. A more useful design makes the flow explicit: retrieve information, perform additional retrieval if the question requires multiple hops, and combine the relevant pieces into a final answer. Each operation becomes a module with a defined role.
An optimizer then uses a supplied metric to tune the program’s prompts or weights. For this chatbot, Morina describes the objective as finding the right answer; he supplies neither a formal scoring rule nor a measured improvement. Compilation can produce different instructions, few-shot examples, or fine-tuning from the same program, depending on the optimization approach. The important separation is between the program’s structure and the choices being tuned inside it.
The PyTorch analogy makes that separation tangible. In __init__, declare components such as ChainOfThought or ReAct, much as a neural-network module declares convolution or dropout layers. In forward, specify how data moves through them, including retrieval. This Python example expresses that structure for the insurance chatbot: a supplied retriever returns policy passages, and an answer module consumes those passages with the customer’s question.
python
import dspy
class PolicyAnswer(dspy.Signature):
"""Answer the question using the supplied policy passages."""
context: list[str] = dspy.InputField()
question: str = dspy.InputField()
answer: str = dspy.OutputField()
class InsuranceQA(dspy.Module):
def __init__(self, retrieve):
super().__init__()
self.retrieve = retrieve
self.answer = dspy.ChainOfThought(PolicyAnswer)
def forward(self, question: str):
passages = self.retrieve(question)
return self.answer(context=passages, question=question)
The constructor defines the parts; forward defines their composition. Retrieval and answer generation now have explicit places in the program instead of being hidden inside a single enormous prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
German text and task-specific modules need care
Two customization needs emerged in the team’s DSPy experience:
- German evaluation: Morina found
answer_exact_matchandanswer_passage_matchunsuitable out of the box for the team’s German text. They needed their own metrics and evaluators, particularly for exact and passage matching. - Typed behavior: The version he describes offered
TypedPredictorandTypedChainOfThought, but lacked the typedReActcomponent they wanted. More specific requirements called for a custom module.
These are historical implementation observations. The Python example above uses the current typed-signature style; current DSPy also documents standard modules including ReAct. English-oriented normalization remains a reason to inspect matching behavior, rather than assuming either that a built-in metric fits German or that every German comparison necessarily fails.
The larger architectural lesson survived those API details: the team’s attempt at one big model did not work. A framework cannot supply useful boundaries for a task the engineer has not yet decomposed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the next improvement small enough to measure
The practical workflow begins before automated optimization:
- Write prompts by hand to determine whether the task is achievable at all.
- Prepare annotated examples and a clear goal metric so an apparent improvement has an objective test.
- Start with basic evaluations, such as regex checks or string comparisons, before introducing LLM-as-judge.
- Make very small adjustments so the system does not become harder to understand with every repair.
This sequence keeps evaluation proportional to the question being tested. A straightforward check should not become another elaborate AI subsystem without a reason.
A short talk cannot provide complete DSPy training. Morina points readers toward Connor Shorten’s learning materials and the DSPy website for that next step. His own curiosity grew through meeting the people who built these tools; the intended result is the same curiosity paired with first-principles engineering. The closing challenge is to find worthwhile problems that AI now makes possible to solve—and let those problems determine what to build.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Hamel Husain shows how to inspect outgoing LLM requests with mitmproxy and understand what prompting frameworks actually send.
The Phoenix project provides tools for observing and evaluating LLM applications.
The current DSPy learning hub covers signatures, modular programs, evaluation, and optimization.
Further reading
- Your AI Product Needs EvalsArticle
A practical guide to domain-specific evaluation, including scoped tests, trace review, human evaluation, and model-based evaluation.
AXA's July 2023 announcement describes Secure GPT's Azure OpenAI foundation and its initial employee rollout.
- AXA and Mistral AIArticle
Mistral describes its collaboration with AXA and the integration of its models into Secure GPT.
Updates since the talk
Current documentation explains custom metrics and the normalization behavior behind built-in string and passage matching.
The current API reference for constructing and using DSPy's ReAct module.
Full transcript
Loading transcript…