AI Engineer World's Fair 2024
Navigating Challenges and Technical Debt in LLMs Deployment
Loading slides…
Read the talk
Navigating Challenges and Technical Debt in LLMs Deployment
Enterprise data can give language models useful organizational memory, but deploying that memory requires retrieval, authorization, infrastructure and governance beyond the model itself.
From a talk by Ahmed Menshawy
Before you start: Familiarity with language-model prompting and the distinction between model training and inference will help; retrieval-augmented generation is explained as it appears.
From structured predictions to organizational memory
What happens when the data an organization wants to use is mostly outside the structured datasets its machine-learning systems handle well? Ahmed Menshawy opens with that mismatch. Supervised learning and deep learning made labeling and prediction valuable enterprise capabilities, but much of an organization’s knowledge remains unstructured.
Menshawy cites estimates that more than 80% of organizational data is unstructured and that 71% of organizations struggle to manage and secure it. The talk does not establish the survey populations or measurement methods behind those figures. The practical problem is nevertheless clear: possessing information does not mean an organization can safely turn it into answers or recommendations.
Language models offer a way to use that information as external memory. Supply relevant domain data as context, and a model can formulate an answer around information specific to the organization. That changes the deployment problem: the system must make organizational knowledge available to the model while retaining the constraints that governed that knowledge before it entered an AI application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Augmentation, current risks and autoregressive errors
At Mastercard, Menshawy frames generative AI as augmenting human productivity. He contrasts that goal with job-replacement predictions and doomsday rhetoric, invoking the Nature editorial Stop talking about tomorrow’s AI doomsday when AI poses risks today. His criticism is pointed: some prominent advocates of catastrophic-risk narratives also release systems that expose users to immediate risks. Identifying those existing harms gives regulators concrete targets for policy, while leaving room to adapt when new algorithmic approaches emerge.
He then separates the usefulness of LLMs from his expectations about artificial general intelligence. Referring to Yann LeCun’s work on objective-driven learning, Menshawy argues that current LLM foundations do not provide the route to AGI. The technical concern he emphasizes is autoregressive dependence: each generated token becomes part of the context for subsequent tokens. An early mistake can therefore influence later generation and grow into a longer, internally consistent but incorrect answer.
Menshawy connects this concern to Ada Lovelace’s 1843 discussion of the Analytical Engine: a machine performs operations it is directed to perform rather than originating something independently. He treats that observation as still relevant roughly 180 years later. Extending an argument about the Analytical Engine to modern machine learning is his interpretation; the concrete deployment concern remains the dependence of later output on earlier generated content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The interface breakthrough and a fraud-detection application
Next-token prediction did not begin with OpenAI. The idea of predicting a token from its context predates today’s assistants by decades. In Menshawy’s account, ChatGPT made the capability accessible through a conversational interface: users could ask for something naturally and receive a response without learning how to interact with a raw language model.
That interface also depended on training data. Menshawy describes a base model trained on internet-scale material, followed by additional phases using manually produced question-and-response pairs. Conversational examples supplied behavior that was comparatively scarce in the underlying corpus. The distinction is between learning broad language patterns and making those capabilities usable as an assistant.
A critique of the foundations does not preclude useful adoption. Menshawy describes Mastercard’s approach as responsibly reducing the risks of the technology and points to its February 2024 fraud-detection announcement. Mastercard’s initial modelling for Decision Intelligence Pro reported fraud-detection-rate improvements of 20% on average and up to 300% in some instances. Rollout was then forecast for later in 2024; the announcement did not disclose an evaluation dataset or a precise baseline, and it did not establish a conversational LLM architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four requirements for a deployable application
A successful generative AI application needs more than a foundation-model endpoint. Menshawy identifies four essentials: access to a variety of models, an environment for contextual customization, tools for building and deploying applications, and scalable ML infrastructure. Scaling includes the time required to bring replicas online: capacity that arrives too late does not solve the user’s immediate demand.
| Requirement | Main deployment challenge |
|---|---|
| Foundation-model access | Balance model size against cost |
| Customization environment | Accommodate models larger than existing platforms expect |
| Build and deployment tools | Work with a young, changing ecosystem |
| Scalable ML infrastructure | Add usable capacity quickly enough |
Menshawy rates model access as comparatively straightforward because models are available, even though selection still involves trade-offs. Existing enterprise AI environments are less certain to fit these models. He considers application tooling the hardest part: much of the earlier tooling does not fit the new workflows, while the replacement ecosystem is itself new. His slide uses color to distinguish these levels of difficulty.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model is a small part of the system
Infrastructure demand also continues after training. Menshawy describes an OpenAI chart as showing inference GPU compute and RAM growing beyond the resources used for training. Without the chart’s workload and accounting conditions, that comparison cannot serve as a capacity-planning ratio; its relevance here is that serving a model can become a substantial, continuing infrastructure obligation.
The 2015 paper Hidden Technical Debt in Machine Learning Systems provides the broader architectural picture: ML code occupies a small box surrounded by the rest of the production system. Menshawy describes model code as less than 5% of the pipeline and surrounding engineering as more than 95% of the work. The paper’s numerical example is narrower: under a glue-code anti-pattern, a mature system might contain at most 5% ML code and at least 95% glue code. That is a warning about code composition, not a measured allocation of engineering labor.
AI engineering includes responsibility for the end-to-end system. Connecting APIs is part of that work, but it does not account for everything surrounding the model. The small central box is useful precisely because it redirects attention to the components that make a model usable, maintainable and operable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where closed-book models meet enterprise constraints
The first architecture Menshawy considers is closed-book generation. An application uses a foundation model directly, supplies a few examples, or fine-tunes it on domain data. In each case, the application relies on knowledge held in the model rather than consulting an external source at answer time. Enterprise accuracy requirements expose several limitations of that arrangement.
- Hallucination: The model can produce incorrect information confidently.
- Attribution: An answer does not necessarily reveal why the model produced it or which source supports it.
- Staleness: Knowledge embedded in parameters becomes outdated.
- Revision: Changing or removing particular information requires difficult model editing.
- Customization: Domain fine-tuning does not by itself ensure that generated claims come only from the organization’s information.
For revision, Menshawy invokes GDPR and an unspecified California law to motivate honoring opt-outs and limiting the future influence of personal information. Those references motivate an engineering requirement here, rather than establish a blanket legal obligation to remove every training influence. Editing information in either a foundation model or a fine-tuned model remains the technical difficulty.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the generator an external memory
Retrieval-augmented generation, or RAG, retains the foundation model and adds context retrieved from domain-specific data. The application can consult external memory when answering instead of requiring every relevant fact to be encoded in the model’s parameters.
Retrieved evidence can improve factual recall and reduce hallucination. Menshawy points to Retrieval Augmentation Reduces Hallucination in Conversation, which evaluates that benefit in knowledge-grounded dialogue. Reduction is the relevant claim: supplying context does not guarantee that a generator will use it correctly or restrict itself to approved facts.
External memory also changes how knowledge is maintained. Updating or swapping a vector index can refresh the information available to retrieval without editing the generator’s parameters. Removing material from that memory supports revision of what the application retrieves, although it is a different operation from erasing information learned during training. Returned sources provide an attribution path: an application can inspect the evidence supplied for a generated answer or decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval and generation must work together
The contextualization diagram surrounds its retrieval-and-generation path with questions about chunking, retrieval, prompting, verification, memory and optimization. Menshawy emphasizes that even these are only part of the production problem. A central question is whether the retriever and generator are optimized to work together.
A common RAG implementation connects two independently developed components: the retriever returns passages, and the generator receives them as context. Menshawy calls them “two separate brains.” The original FAIR paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, instead describes coordinated optimization of retrieval and generation. His phrase about training in parallel refers to that shared optimization, not merely running two unrelated training jobs simultaneously.
Access to model parameters makes this coordination possible. With open model weights, teams can fine-tune the generator to produce factual answers from the evidence the retriever actually supplies. The objective goes beyond attaching memory: the generator must learn how to use the retrieved information. That is a different level of control from treating the model as an opaque endpoint and modifying only its prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Governance and the decision to use an LLM
Mastercard’s adoption framework, as Menshawy describes it, includes seven responsible-AI principles. He names privacy, security and reliability, then adds the organizational mechanism that makes principles enforceable: a governing body and a clear strategy for applying them during development. Adopting an LLM for a service remains conditional on safeguards, including access controls and protection against biased output.
That work can change the decision about whether to use an LLM at all. Menshawy recalls that a reviewer who accepted the deployment paper nevertheless wondered whether LLMs were the right tools for some applications, given the challenges and technical debt it described. His response is that transformative technology brings implementation difficulty with it. The tension is productive: potential value does not remove the obligation to assess the engineering burden, and that burden does not automatically negate the value.
He closes by directing readers to Mastercard’s AI engineering blogs for work on putting AI into production—the same surrounding components represented by the boxes outside the model. The application has to deliver useful answers while preserving the organization’s operational and governance requirements. Those obligations remain even when the model itself is only an API call away.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The 2015 paper examining maintenance costs, data dependencies and system design anti-patterns around machine learning.
Research evaluating retrieval-based dialogue models and their reduction of factual hallucinations.
The original RAG paper combining a neural retriever, external document memory and a generative model.
Further reading
A compact presentation of deployment requirements, retrieval architecture and the engineering systems surrounding an LLM.
An author-contributed overview of the April 2024 ACM paper on production deployment challenges.
Full transcript
Loading transcript…