AI Engineer World's Fair 2024

Navigating Challenges and Technical Debt in LLMs Deployment

Ahmed Menshawy· Vice President of AI Engineering, Mastercard16:15

Loading slides…

Read the talk

Navigating Challenges and Technical Debt in LLMs Deployment

Enterprise data can give language models useful organizational memory, but deploying that memory requires retrieval, authorization, infrastructure and governance beyond the model itself.

From a talk by Ahmed Menshawy

Before you start: Familiarity with language-model prompting and the distinction between model training and inference will help; retrieval-augmented generation is explained as it appears.

From structured predictions to organizational memory

What happens when the data an organization wants to use is mostly outside the structured datasets its machine-learning systems handle well? Ahmed Menshawy opens with that mismatch. Supervised learning and deep learning made labeling and prediction valuable enterprise capabilities, but much of an organization’s knowledge remains unstructured.

Menshawy cites estimates that more than 80% of organizational data is unstructured and that 71% of organizations struggle to manage and secure it. The talk does not establish the survey populations or measurement methods behind those figures. The practical problem is nevertheless clear: possessing information does not mean an organization can safely turn it into answers or recommendations.

Language models offer a way to use that information as external memory. Supply relevant domain data as context, and a model can formulate an answer around information specific to the organization. That changes the deployment problem: the system must make organizational knowledge available to the model while retaining the constraints that governed that knowledge before it entered an AI application.

Slide titled “The unstructured data problem” with blue and orange concentric rings, an over-80% statistic on the left and a 71% statistic on the right.Open full source frame
The unstructured data problem: over 80% of enterprise data is unstructured, and 71% of enterprises struggle to manage and protect it.
0:440:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:44 · section reference included

Augmentation, current risks and autoregressive errors

At Mastercard, Menshawy frames generative AI as augmenting human productivity. He contrasts that goal with job-replacement predictions and doomsday rhetoric, invoking the Nature editorial Stop talking about tomorrow’s AI doomsday when AI poses risks today. His criticism is pointed: some prominent advocates of catastrophic-risk narratives also release systems that expose users to immediate risks. Identifying those existing harms gives regulators concrete targets for policy, while leaving room to adapt when new algorithmic approaches emerge.

He then separates the usefulness of LLMs from his expectations about artificial general intelligence. Referring to Yann LeCun’s work on objective-driven learning, Menshawy argues that current LLM foundations do not provide the route to AGI. The technical concern he emphasizes is autoregressive dependence: each generated token becomes part of the context for subsequent tokens. An early mistake can therefore influence later generation and grow into a longer, internally consistent but incorrect answer.

Menshawy connects this concern to Ada Lovelace’s 1843 discussion of the Analytical Engine: a machine performs operations it is directed to perform rather than originating something independently. He treats that observation as still relevant roughly 180 years later. Extending an argument about the Analytical Engine to modern machine learning is his interpretation; the concrete deployment concern remains the dependence of later output on earlier generated content.

1:522:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:52 · section reference included

The interface breakthrough and a fraud-detection application

Next-token prediction did not begin with OpenAI. The idea of predicting a token from its context predates today’s assistants by decades. In Menshawy’s account, ChatGPT made the capability accessible through a conversational interface: users could ask for something naturally and receive a response without learning how to interact with a raw language model.

That interface also depended on training data. Menshawy describes a base model trained on internet-scale material, followed by additional phases using manually produced question-and-response pairs. Conversational examples supplied behavior that was comparatively scarce in the underlying corpus. The distinction is between learning broad language patterns and making those capabilities usable as an assistant.

A critique of the foundations does not preclude useful adoption. Menshawy describes Mastercard’s approach as responsibly reducing the risks of the technology and points to its February 2024 fraud-detection announcement. Mastercard’s initial modelling for Decision Intelligence Pro reported fraud-detection-rate improvements of 20% on average and up to 300% in some instances. Rollout was then forecast for later in 2024; the announcement did not disclose an evaluation dataset or a precise baseline, and it did not establish a conversational LLM architecture.

4:414:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:41 · section reference included

Four requirements for a deployable application

A successful generative AI application needs more than a foundation-model endpoint. Menshawy identifies four essentials: access to a variety of models, an environment for contextual customization, tools for building and deploying applications, and scalable ML infrastructure. Scaling includes the time required to bring replicas online: capacity that arrives too late does not solve the user’s immediate demand.

RequirementMain deployment challenge
Foundation-model accessBalance model size against cost
Customization environmentAccommodate models larger than existing platforms expect
Build and deployment toolsWork with a young, changing ecosystem
Scalable ML infrastructureAdd usable capacity quickly enough

Menshawy rates model access as comparatively straightforward because models are available, even though selection still involves trade-offs. Existing enterprise AI environments are less certain to fit these models. He considers application tooling the hardest part: much of the earlier tooling does not fit the new workflows, while the replacement ecosystem is itself new. His slide uses color to distinguish these levels of difficulty.

Four connected boxes show access to foundation models, an environment to customize contextual LLMs, tools to build and deploy applications, and scalable ML infrastructure. Blue shades indicate differing levels of challenges and technical debt.Open full source frame
Four essentials for a generative AI application, color-coded by challenges and technical debt.
6:326:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:32 · section reference included

The model is a small part of the system

Infrastructure demand also continues after training. Menshawy describes an OpenAI chart as showing inference GPU compute and RAM growing beyond the resources used for training. Without the chart’s workload and accounting conditions, that comparison cannot serve as a capacity-planning ratio; its relevance here is that serving a model can become a substantial, continuing infrastructure obligation.

The 2015 paper Hidden Technical Debt in Machine Learning Systems provides the broader architectural picture: ML code occupies a small box surrounded by the rest of the production system. Menshawy describes model code as less than 5% of the pipeline and surrounding engineering as more than 95% of the work. The paper’s numerical example is narrower: under a glue-code anti-pattern, a mature system might contain at most 5% ML code and at least 95% glue code. That is a warning about code composition, not a measured allocation of engineering labor.

AI engineering includes responsibility for the end-to-end system. Connecting APIs is part of that work, but it does not account for everything surrounding the model. The small central box is useful precisely because it redirects attention to the components that make a model usable, maintainable and operable.

8:078:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:07 · section reference included

Where closed-book models meet enterprise constraints

The first architecture Menshawy considers is closed-book generation. An application uses a foundation model directly, supplies a few examples, or fine-tunes it on domain data. In each case, the application relies on knowledge held in the model rather than consulting an external source at answer time. Enterprise accuracy requirements expose several limitations of that arrangement.

  • Hallucination: The model can produce incorrect information confidently.
  • Attribution: An answer does not necessarily reveal why the model produced it or which source supports it.
  • Staleness: Knowledge embedded in parameters becomes outdated.
  • Revision: Changing or removing particular information requires difficult model editing.
  • Customization: Domain fine-tuning does not by itself ensure that generated claims come only from the organization’s information.

For revision, Menshawy invokes GDPR and an unspecified California law to motivate honoring opt-outs and limiting the future influence of personal information. Those references motivate an engineering requirement here, rather than establish a blanket legal obligation to remove every training influence. Editing information in either a foundation model or a fine-tuned model remains the technical difficulty.

9:229:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:22 · section reference included

Give the generator an external memory

Retrieval-augmented generation, or RAG, retains the foundation model and adds context retrieved from domain-specific data. The application can consult external memory when answering instead of requiring every relevant fact to be encoded in the model’s parameters.

Retrieved evidence can improve factual recall and reduce hallucination. Menshawy points to Retrieval Augmentation Reduces Hallucination in Conversation, which evaluates that benefit in knowledge-grounded dialogue. Reduction is the relevant claim: supplying context does not guarantee that a generator will use it correctly or restrict itself to approved facts.

External memory also changes how knowledge is maintained. Updating or swapping a vector index can refresh the information available to retrieval without editing the generator’s parameters. Removing material from that memory supports revision of what the application retrieves, although it is a different operation from erasing information learned during training. Returned sources provide an attribution path: an application can inspect the evidence supplied for a generated answer or decision.

Diagram titled “Contextualization” shows document and query encoders feeding a retriever, context flowing to a generator alongside input and prompt, and an output box. Gray callouts ask about chunking, retrieval, prompting, verification, memory, and optimization.Open full source frame
A contextualization pipeline connects document and query encoders, retrieval, context, and generation, surrounded by implementation questions.
10:4610:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:46 · section reference included

Retrieval and generation must work together

The contextualization diagram surrounds its retrieval-and-generation path with questions about chunking, retrieval, prompting, verification, memory and optimization. Menshawy emphasizes that even these are only part of the production problem. A central question is whether the retriever and generator are optimized to work together.

A common RAG implementation connects two independently developed components: the retriever returns passages, and the generator receives them as context. Menshawy calls them “two separate brains.” The original FAIR paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, instead describes coordinated optimization of retrieval and generation. His phrase about training in parallel refers to that shared optimization, not merely running two unrelated training jobs simultaneously.

Access to model parameters makes this coordination possible. With open model weights, teams can fine-tune the generator to produce factual answers from the evidence the retriever actually supplies. The objective goes beyond attaching memory: the generator must learn how to use the retrieved information. That is a different level of control from treating the model as an opaque endpoint and modifying only its prompt.

11:5412:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:54 · section reference included

Organizational memory must preserve permissions

Menshawy’s Navigating Challenges and Technical Debt in Large Language Models Deployment applies the surrounding-system perspective to LLM applications. Menshawy estimates that components surrounding the LLM account for more than 90% of building the application. He uses that estimate to emphasize the scale of the surrounding work, rather than present a measured workload breakdown.

Domain-specific data collection illustrates why those surrounding components matter. Collecting the right documents is only part of the job. An employee may have access to one source system but not another; combining those systems into a shared knowledge store must not erase that distinction. A global LLM with unrestricted access could make information available through an answer that the same user could not obtain from the original system.

The permission boundary therefore needs to survive the move into the LLM application. Consider a small TypeScript example: an employee in the support group asks about an escalation process. The candidate records include an approved support procedure and a restricted finance forecast. Filtering by the caller’s groups before constructing context preserves the distinction:

typescript

type KnowledgeRecord = {
  id: string;
  text: string;
  allowedGroups: readonly string[];
};

function buildContext(
  records: readonly KnowledgeRecord[],
  callerGroups: ReadonlySet<string>,
): string {
  return records
    .filter(record =>
      record.allowedGroups.some(group => callerGroups.has(group)),
    )
    .map(record => `[${record.id}] ${record.text}`)
    .join("\n\n");
}

const records: KnowledgeRecord[] = [
  {
    id: "support-escalation",
    text: "Escalate unresolved cases to the support lead.",
    allowedGroups: ["support"],
  },
  {
    id: "finance-forecast",
    text: "The internal forecast is awaiting finance review.",
    allowedGroups: ["finance"],
  },
];

const context = buildContext(records, new Set(["support"]));

Here, context includes support-escalation and excludes finance-forecast. In an application, caller groups must come from authenticated identity, and authorization must precede exposure of protected content to the generator. Menshawy pairs preserving access controls with using specialized models for particular tasks and building safeguards around present-day risks.

13:0313:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:03 · section reference included

Governance and the decision to use an LLM

Mastercard’s adoption framework, as Menshawy describes it, includes seven responsible-AI principles. He names privacy, security and reliability, then adds the organizational mechanism that makes principles enforceable: a governing body and a clear strategy for applying them during development. Adopting an LLM for a service remains conditional on safeguards, including access controls and protection against biased output.

That work can change the decision about whether to use an LLM at all. Menshawy recalls that a reviewer who accepted the deployment paper nevertheless wondered whether LLMs were the right tools for some applications, given the challenges and technical debt it described. His response is that transformative technology brings implementation difficulty with it. The tension is productive: potential value does not remove the obligation to assess the engineering burden, and that burden does not automatically negate the value.

He closes by directing readers to Mastercard’s AI engineering blogs for work on putting AI into production—the same surrounding components represented by the boxes outside the model. The application has to deliver useful answers while preserving the organization’s operational and governance requirements. Those obligations remain even when the model itself is only an API call away.

14:1914:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:19 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…