AI Engineer World's Fair 2024
Understanding AI Stakes to Break Production Code
Loading slides…
Read the talk
Understanding AI Stakes to Break Production Code
Production AI requires more than a convincing prototype: the consequences of error determine where reasoning belongs, how systems are evaluated, and what users must be able to inspect.
From a talk by Philip Rathle
Before you start: Familiarity with language models, vector search, and basic application pipelines will help; the Python example uses only the standard library.
Why does an impressive prototype stall?
What changes between an impressive AI demonstration and an application people can safely depend on? The answer starts with application stakes: what happens when the system is wrong. Philip Rathle opens this production roundtable by asking participants to compare the hurdles they have solved with those they have not. The discussion operates under the Chatham House Rule, with audience contributions kept anonymous. Its organizing premise is that higher stakes bring higher acceptance bars—and different engineering obstacles.
Generative AI initially looks like a hockey stick: capability rises quickly, and compelling prototypes are easy to build. Actual projects encounter a production barrier. Adding retrieval-augmented generation, or RAG, with vector search can move some applications across it. Others still fall short.
Rathle presents knowledge graphs and GraphRAG as a further step in Neo4j’s approach. But even this progression is better understood as successive S curves than as unlimited acceleration: a technique unlocks a class of applications, then encounters another limit. The practical question is which barrier matters for the application being built.
| Application position | Example or consequence | What changes the stakes |
|---|---|---|
| Lower stakes | Bedtime stories; some text summaries | Several answers may be acceptable |
| Intermediate | A business copilot | Human oversight mitigates consequential errors |
| Higher stakes | An autonomous pilot | Errors can affect money, reputation, health, or safety |
These are positions on a spectrum, not fixed labels for entire product categories. Regulation, fines, and bias also raise the consequences of failure. A human in the loop can move a consequential application toward the middle: the system assists a decision instead of owning it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An apple, a vector, and a knowledge graph
Higher-stakes applications often need enterprise-specific facts and up-to-the-moment information that a model cannot reliably supply from its parameters. Retrieval brings that information into the answer-generation process. It can also reduce hallucinations, but Rathle’s metaphor of repeatedly loading the dice captures the remaining problem: improving the odds does not establish certainty.
The acceptable residual risk depends on the task. Rathle asks how tightly to fasten bolts on a 737 MAX 9. In Rathle’s hypothetical aircraft-maintenance example, even a 0.001% chance of error is unacceptable. That figure expresses his safety concern, rather than a measured model error rate or an established aviation requirement.
Now consider an apple. A person’s concept of it combines appearance, taste, texture, and other associations. A vector represents it as coordinates. Cosine similarity or spatial distance can use those coordinates to retrieve related fruit; that is a useful operation, but it is not the whole concept of an apple. Rathle characterizes LLMs and vector RAG as statistical ways of approximating the world through patterns such as word proximity and frequency.
His right-brain analogy describes the resulting behavior: creative, often right, occasionally very wrong, and difficult to explain. Knowledge graphs provide the complementary representation. Explicit structure makes information understandable to people and available for machine reasoning—the left-brain side of his analogy. These are metaphors for system behavior, not a neurological account of language models.
Structured knowledge and language generation have different jobs. A graph cannot do everything an LLM can. Combined with an LLM, however, it can supply facts, relationships that support reasoning and discernment, and persistent information that serves as long-term memory. The proposal is to combine those capabilities rather than expect either representation to solve every problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put exact reasoning in the right component
The discussion turns to a practical question: which lesson would have saved time if it had been learned sooner? Rathle’s first answer is to understand the stakes early without letting planning stop iteration. Start building, but let the consequences of failure influence tool selection.
His term for the next decision is locus of reasoning. If a question has an exact answer, a database or another reasoning engine may be the right place to compute it. The LLM can handle the language on either side:
- Translate the user’s question into a query.
- Have the database or reasoning engine compute the result.
- Express that result in language appropriate to the audience.
Deterministic execution does not make the whole pipeline correct: the generated query must still express the intended question, and the final response must preserve the result. But it moves the exact computation into a component designed for that work.
Audience adaptation then becomes a separate task from determining the answer. A response to a regulator should be serious and joke-free. An outbound marketing message might use different language for a high-school student and a retiree. The facts need not change when the presentation does.
An audience member offers a concrete version of this separation: instead of pasting Excel data into an LLM and asking it to identify an outlier, ask it to write Python that finds the outlier. The resulting computation can run without keeping the LLM in the execution loop. The model’s ability to write a procedure can be more useful than its ability to answer the numerical question directly.
For a small illustrative dataset, an explicit interquartile-range rule makes that division of labor visible. The values and rule below are a worked example of the suggested approach:
python
from statistics import quantiles
values = [10, 11, 12, 12, 13, 14, 50]
q1, _, q3 = quantiles(values, n=4, method="inclusive")
iqr = q3 - q1
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr
outliers = [
value
for value in values
if value < lower_bound or value > upper_bound
]
print(outliers) # [50]
Here the program applies a specified definition of an outlier. Choosing whether that definition fits the data remains a separate decision; the benefit is that the calculation itself is inspectable and repeatable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantic search still meets keyword habits
A team can build good retrieval and still misunderstand how people will use it. One participant describes investing in RAG, indexing, and semantic search, then discovering in production that customers continued entering keywords. The team had expected natural-language questions; users brought the habits of a familiar search box.
That mismatch required work to interpret or rewrite the input: was it a keyword query, or a question? The team had spent heavily on making search work before asking whether this was how customers wanted to search. Retrieval quality alone did not resolve the interaction problem.
Rathle asks whether a curriculum could help users discover that they can ask more than keyword queries. The participant acknowledges that onboarding had initially received little attention: the assumption was that everyone would simply start asking questions. Easing users into the new behavior became part of the product work, alongside making the underlying search function well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose projects that can meet the acceptance bar
Another production lesson is to kill more projects during experimentation. Running many experiments is useful only if the team eventually concentrates effort on the few that can meet users’ expectations. Some projects will not reach the required accuracy even with substantial additional work. The participant would rather focus on one or two that can make it all the way to production.
Rathle raises both achievability and business value as selection criteria. The response brings the discussion back to the user’s acceptance bar. A participant uses 90–95% accuracy as an illustrative level that may be insufficient for one use case, while 90% may be acceptable for another. No task, dataset, or accuracy definition accompanies those figures; they illustrate why expensive commercialization should follow a match between achievable performance and user needs.
A brief contribution sharpens the same point: expect less and use more common sense. That does not mean abandoning ambition. It means avoiding a product plan that depends on capabilities the system has not demonstrated.
The economic version of the problem begins with a customer asking for the best model at the fastest possible speed. Those are not sufficient requirements. Cost–quality and cost–speed trade-offs depend on what the application actually needs to do and how quickly it needs to respond. Without that understanding, a deployment can spend far more than necessary.
A small laboratory question set naturally encourages maximizing accuracy; production exposes the cost of doing so repeatedly. Rathle imagines a deployment consuming a year’s budget in a day to illustrate the danger of ignoring scale. His suggested response includes smaller models, with the choice driven by required quality and latency rather than by an unconditional demand for the strongest model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ship now without freezing the model stack
Rapid model improvement complicates the investment decision. The moderator tentatively recalls Bloomberg training a financial-data model, perhaps based on Llama 2, that performed well in evaluations. The moderator recalls Llama 3 outperforming that financial fine-tune, but the model identity and comparison are unverified. The anecdote raises a useful question without settling it: how much work will a subsequent base-model release make unnecessary?
Waiting is not a complete product strategy either. Enterprises want something working today, and teams want the competitive advantage of shipping. Rathle initially frames the wait in years; participants push back that meaningful advances are arriving on the scale of weeks. The engineering problem is to deliver value while retaining the ability to benefit from those advances.
One participant’s approach is to establish the full path through the system first, then improve its stages in parallel. Deliver an end-to-end result for the customer. If the first data-engineering step will not scale, assign work to improve it. If acceptable model performance currently requires an expensive option such as Claude 3.5 Sonnet, keep that stage replaceable as model capabilities and economics change. These model names describe the choices discussed at the time of the recording.
A working pipeline makes component improvements usable. Once the steps and their boundaries are understood, better components can be substituted into the path rather than requiring the product to be reinvented. Rathle connects this to the rising capability curves from the opening: improvements become valuable when the system has somewhere to put them.
That approach has a limit the participant has not solved. More agentic behavior makes orchestration itself a moving target. The team is looking for better tools, and it has already had to slow down because too many layers were changing at once. Interfaces and contracts between layers are necessary so that improving one stage does not unexpectedly break a downstream stage. The participant presents this as ongoing work, not a finished orchestration solution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate capability before optimizing it away
The next question is how long an optimization should take. If a latency improvement requires months of engineering, a new model may arrive before the work finishes. A participant proposes putting a ceiling on implementation time, without specifying a universal number of months. Rathle’s two-year-project example pushes the concern to an extreme: a long project aimed at a temporary model limitation may become obsolete before it delivers value.
A different participant warns about optimizing too early. Teams that begin with a small fine-tuned model can incorrectly conclude that a task is impossible. Starting with a more capable model helps distinguish a genuine product or capability problem from a limitation introduced by the initial cost constraint. Production latency still matters, but it should not prevent the team from first learning whether the desired behavior is achievable.
The resulting sequence is:
- Validate the product need and capability. Establish whether a strong model can perform the task well enough.
- Account for proprietary information. Determine whether missing domain data calls for retrieval or fine-tuning.
- Optimize the working behavior. Move toward faster, cheaper models, potentially using larger-model outputs as fine-tuning data, while checking that accuracy holds.
Rathle names GPT-4o and Claude 3.5 as starting points for that feasibility check. His heuristic is that if the stronger model cannot do the task, a smaller model probably will not—while acknowledging that highly proprietary data changes what the model must be given.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation starts with the data you will need
Consider a pipeline in which an LLM converts unstructured material into structured data for a traditional machine-learning model. The application needs a way to determine whether that conversion is good enough and whether subsequent changes improve it. A participant’s lesson is to build the evaluation pipeline from the start—and first ask whether the necessary evaluation data exists.
If suitable data is missing, begin generating it on day zero. Otherwise, a team may reach the point of having an LLM embedded in its application without being able to tell whether a prompt change made the system better or worse. Designing evaluation only then creates a long delay: the application exists, but the evidence needed to guide its development does not.
Rathle places this alongside the usual operational framework. DevOps, evaluation, and data quality are connected responsibilities. The ability to deploy a change is not the ability to judge it; the evaluation framework, in turn, depends on having the right data at sufficient quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make generated answers recognizable and inspectable
One team deliberately avoids hiding AI behind an apparently ordinary interface. A participant describes gradually displaying generated responses as a cue for users to be vigilant, along with distinct colors and icons that identify AI interactions. The intent is to make the nature of the output visible before users treat it as authoritative.
The interface also gives users tools to inspect an answer: what sources were used, what inputs went into it, and what might have gone wrong. The team does not simply abstract generation away and hope for the best. Rathle connects visible generation to moderated expectations: seeing an answer emerge can remind a user that it deserves consideration rather than automatic acceptance. These are the team’s reported interface choices; the discussion does not establish how much they change user vigilance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human review needs more than a citation list
Explainability becomes especially concrete when the output could influence medical treatment or equipment maintenance. In response to Rathle’s question about regulation, a behavioral-health practitioner describes the team’s current operating principle: always retain a human in the loop. They are building a copilot. The participant’s remark about never becoming a pilot is offered as reassurance to others, rather than as a firm technical prediction.
For that human review to mean something, the reviewer needs access to the inputs that produced the output. Recommendations and selected case information must be traceable to root data. Rathle asks whether showing inputs is enough to earn trust, then points out the practical burden of references: users may not read three 50-page papers just to check an answer. Evidence must be available in a form that fits the review task.
The practitioner describes two review arrangements:
- Knowledge lookup: show the retrieved source documentation beside the answer, with citations.
- Therapy-session summary: show the collected session data beside the generated summary, paragraph by paragraph.
For summaries, the user can change and edit the resulting paragraphs. The system supports a reviewable draft; the final summary becomes the user’s work rather than an untouchable model artifact.
Rathle then asks whether human judgments are used to improve the producer. The practitioner describes several checks and feedback channels:
- A separate, quick evaluation stage checks whether the output matches the inputs, including a safety-oriented check.
- User edits and thumbs-up or thumbs-down responses provide feedback.
- The team performs its own clinical evaluation of outputs.
This describes layered evaluation and feedback collection, not a specified training procedure. The team inserts checks wherever it needs additional confidence in what the system is producing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a human reviewer cannot be included
The final example tests the limit of human oversight. A participant working on regulated personal-loan recommendations cannot include a human reviewer in the workflow. Instead of declaring a single loan best, the team gives the end user three options with explanations. The participant describes this as less deterministic in the sense that the product presents choices rather than one definitive recommendation, and reports that it helps address regulatory concerns; the discussion does not establish compliance with any particular rule.
Rathle checks who receives the options: an intermediary reviewer or the end user? They go directly to the end user because intermediary review is unavailable. The participant adds that the selection uses logic beyond raw LLM output, although the logic itself is not specified. The final design therefore changes the decision the AI is allowed to make: it supplies explained alternatives for the user to consider, rather than presenting one generated answer as the decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official explanation of the rule permitting information sharing while protecting participant identities and affiliations.
Further reading
- The GraphRAG manifestoArticle
Philip Rathle explains graph-enhanced retrieval, contrasts vector and graph representations, and develops his argument for enterprise GraphRAG.
Foundational research combining pretrained language generation with retrieved Wikipedia passages.
Bloomberg's original account of its financial language model, training data, and evaluation methodology.
Updates since the talk
Current examples for vector search, hybrid retrieval, graph traversal, and natural-language-to-Cypher pipelines.
Full transcript
Loading transcript…