AI Engineer World's Fair 2024
AI Platform Engineering
Loading slides…
Read the talk
AI Platform Engineering: From Pilot Teams to Production
Scaling GenAI across an organization requires shared infrastructure, practical enablement, and governance that helps application teams build and operate useful products.
From a talk by Patrick Debois
Before you start: Familiarity with application development, APIs, and basic deployment practices will help; no model-training background is required.
What happens after the first AI pilot?
You have a team experimenting with GenAI. What will it take for the rest of the organization to use it? Patrick Debois approaches that question through an earlier transformation: he organized the first DevOpsDays in 2009, with about 60 attendees, and subsequently watched DevOps spread through enterprises. That experience supplies a useful lens, with a qualification he makes explicit: applying an old paradigm to a new technology can reveal a pattern—or get it completely wrong.
His interest in GenAI starts with automation intelligence. Much of his DevOps work involved automating pipelines, making them robust, and ensuring that they delivered value. AI extends that work, but it also raises the familiar question of how a promising capability becomes something an organization can reliably use.
The adoption pattern begins with one pilot team. Two or three more teams try the technology; their experience exposes repeatable patterns; those patterns then support wider adoption. Cloud infrastructure followed this path. Shared abstractions eventually let application teams move faster within their own domains without understanding every part of the underlying infrastructure.
Even the unsettled name of the practice has a function. Debois recalls the arguments over DevOps, alongside earlier movements around agile and ITIL. A label such as AI engineer gives people a way to find emerging stories, job postings, and one another. Defining it too rigidly, too early, can restrict what the practice becomes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production work changes the team
In Debois's experience building GenAI applications over two years, the first organizational friction appeared when the work landed with data scientists who were not accustomed to running production services. Engineers began moving into the data science team. As integration became a larger part of the work, engineering's relative share increased. Capabilities then spread to other teams and eventually moved into the platform.
That movement also reconnected data that had been isolated in a data lake with the applications that needed it. Debois calls this a shift right, contrasting it with security's shift left: responsibilities move as teams discover where the friction actually lies.
Bringing development and operations into the same room was a start, but permanent, intensive collaboration did not scale across an organization. Kubernetes pipelines, infrastructure, and security tooling became shared capabilities. Team Topologies, which grew out of DevOps Topologies, provides a vocabulary for this arrangement: feature teams build and run their applications on top of a platform.
The platform team can interact with those teams in three ways:
| Interaction | What the platform team does |
|---|---|
| Service consumption | Provides an API or service the team can use |
| Collaboration | Works with the team to shape the capability it needs |
| Facilitation | Helps the team overcome problems and learn |
The platform is a product, and application teams are its customers. Its purpose is to help traditional application developers, supported by data science expertise, deliver AI features within their own teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control access and observe quality
Larger enterprises often want independence from a particular model provider. A shared proxy can mediate model access while enforcing who may use which services. Debois tentatively compares the OpenAI API's possible role to S3's role in storage: a common interface around which providers might converge. This is a historical forecast in the talk, not a claim of a formal standard or verified compatibility.
Once applications run in production, ordinary request monitoring is insufficient. The platform needs to capture prompts and trace their relationships. Debois describes an initial prompt leading to five or six iterations or questions, so the useful unit of observation is often a sequence rather than a single API call. Providing that tracing centrally saves each team from assembling it independently.
An available API can still produce poor results. Production monitoring therefore needs checks on data and output quality, extending practices familiar from MLOps. Continuously running evaluations act as another kind of health check: they can help reveal model changes or unusual user behavior that an uptime check would miss.
Caching and feedback collection are further shared services. Feedback should extend beyond thumbs up, thumbs down, and retry: inline edits reveal how a user would correct the result. Building and managing these capabilities well is expensive enough that duplicating them across teams is wasteful. The growing ecosystem resembles Kubernetes in its breadth; an AI platform is already more than cloud access and a few API keys.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Help teams experiment—and find a use case
Providing infrastructure does not automatically make it usable. Like any product team, the platform team must guide its customers and learn from their experience. Enablement is both instruction and a feedback channel: it reveals which capabilities help and which get in the way.
Several forms of support make experimentation easier:
- Prototyping tools: Include product owners, who need to explore potential use cases as much as engineers do. Connect those tools securely to company data.
- Frameworks: Give developers a way to discover what is possible, supported by education and documentation suited to their existing stacks.
- Local development: Let developers experiment in a familiar environment, including while traveling. Improving laptop models can shorten iteration loops, although local stacks have tradeoffs.
These are ways to help teams learn, not merely additional items in a service catalog.
The next obstacle is choosing work worth doing. Mandating GenAI in every part of a product can put implementation ahead of a real use case; Debois compares some of this activity to spending a marketing budget. He also warns that an excessive focus on model training and fine-tuning can postpone getting anything into production.
Premature cost optimization creates a similar distraction. Insisting on a local or cheaper model before establishing the business case can consume engineering effort without producing value. Debois's ordering is to establish the return first, then reduce the cost of delivering it.
Once users provide feedback, someone must interpret it. Engineers accustomed to error stacks may not know what a qualitative correction means for the product. The product owner often holds the domain knowledge needed to distinguish an acceptable answer from a misleading one. Collecting feedback without involving that expertise leaves the improvement loop unfinished.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model changes make evaluation essential
Developer experience includes some unusually difficult choices: which model to start with, how to obtain representative test data, and how much to depend on a fast-changing framework. Debois describes developers first learning through frameworks, then abandoning them after disruptive changes and rebuilding against lower-level APIs. A prompt call may be simple, but tracing, monitoring, and the surrounding integrations still have to work together. His hope is for a dependable tool ecosystem rather than every team rebuilding that machinery.
Debois recalls using approximately eight LLM models over two years while pursuing better quality. Each change required rewriting and reworking application prompts, and the work was manual. From an engineering-management perspective, that combined substantial cost with little assurance that the replacement would behave correctly. Establish evaluations before refactoring prompts for a new model.
Evaluation becomes easier to discuss when it starts with checks engineers already understand. A character-count requirement or regular expression has a definite result. For example, a TypeScript check for the talk's twenty-character constraint can be expressed directly:
typescript
function hasTwentyCharacters(output: string): boolean {
return Array.from(output).length === 20;
}
const outputs = ["Your order is ready.", "Ready."];
const results = outputs.map(output => ({
output,
passes: hasTwentyCharacters(output),
}));
console.log(results);
This checks a format requirement; it does not establish whether the answer is useful or correct.
The evaluation ladder then moves beyond deterministic rules:
| Method | What it checks |
|---|---|
| Sentiment helper model | The answer's sentiment |
| Semantic comparison | Relatedness, such as question–answer relevance |
| LLM judge | Qualities assessed by another language model |
| Human feedback | Human assessment of the result |
Using an LLM to judge another LLM introduces a reliability question rather than eliminating one. Debois treats that concern as unresolved, while recognizing model-based judging as a practical tool. The engineering approach combines these methods instead of relying on an informal impression alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Faster generation moves work into review
Enablement also has an emotional component. Some engineers initially fear AI; giving them a coding copilot lets them experience its usefulness directly. That puts the team building GenAI capabilities in a second role: helping colleagues become comfortable using AI in their own work.
Debois reports that copilots produced more code, larger AI-generated pull requests, and longer review times. He does not claim that review overhead entirely canceled the generation gains. The point is that speeding up one stage can create friction at the next, so code generation alone is an incomplete account of productivity.
He connects this tension to Ironies of Automation and a newer generative-AI treatment of the same problem, Ironies of Generative AI. The person using automation moves from producing work toward managing and reviewing it. Generation becomes faster while evaluation occupies more of the person's time.
That role change becomes dangerous when the reviewer no longer understands the work well enough to assess it. Without the domain model or hands-on experience, accepting an output can become easier than evaluating it. Debois illustrates this with a weekend Copilot acceptance-rate anecdote; his humorous explanation that developers simply care less is not an established cause. The deeper concern is declining situational awareness: less practice producing the work can leave people less prepared to recognize and handle failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automation creates a duty to prepare for failure
The early promise of DevOps could sound like replacing a person's work with a shell script. What grew around that automation was a much larger discipline of preparing for failure. Debois illustrates the balance as 20% automation and 80% failure preparation, an organizing metaphor rather than a measured industry allocation.
The progression matters:
- CI/CD and tests add checks around automated delivery.
- Production monitoring reveals whether the running system is healthy.
- Designing for failure begins by assuming things will break.
- Observability gives operators the situational awareness to understand what is happening.
- Chaos engineering injects failures so teams can keep practicing their response.
Automation removes some work, but these practices help people remain capable when it fails.
The purpose is also to reduce cognitive load during an incident, when people have the least capacity to reconstruct an unfamiliar system. Not every organization adopts every practice; the appropriate investment depends on its risk appetite. But the work of enablement extends to helping engineers feel prepared, not simply making them faster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Governance must survive everyday use
Governance starts with personal awareness. A tool that records the screen may promise productivity while also exposing everything visible on it. Teams need to understand what they are copying or transmitting, and how a service handles those inputs. Training opt-outs deserve particular attention because where and when to configure them may not be obvious.
A restricted set of approved models makes license review more manageable. The word open does not by itself establish permission to use a model; its license and origin still need examination. Debois also includes basic awareness of European workflow risk assessment: teams should ask whether a proposed use is allowed and appropriate without expecting every engineer to become a lawyer.
Operational controls require equal care. Debois treats prompt injection as an unsolved problem for which defenses are still necessary. He compares guardrails to intrusion detection and web application firewalls: a growing pile of rules and logs can eventually train people to ignore alerts. Installing a control can create reassurance without ensuring that anyone can operate it effectively.
More uncontrolled text crossing service boundaries also increases the need to scrutinize personally identifiable information, with useful metrics and alerting around what is transmitted. These responsibilities bring the platform's remit into focus: operate shared infrastructure, enable teams to use it, and establish company rules for that use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect the platform to the product experience
The platform's internal focus leaves another coordination problem: how AI appears to customers across the product. Debois introduces the unFIX Experience Crew as a complementary organizational idea from an alternative to Team Topologies. Applied to AI, the crew works across feature teams to understand what they are building and make the resulting user experience consistent.
That adds an experience layer to the combination of CloudOps, SecOps, developer experience, the data platform, and AI infrastructure. Debois advises against this structure for a 10-person company and suggests considering it when scaling to 10 or more teams. It is a judgment about coordination needs, not a requirement to create departments before there is enough work to justify them.
The benefit comes from collaboration between specialties. SecOps brings access-control and governance expertise; CloudOps knows provisioning and cloud vendors. Bringing those capabilities into a shared area gives teams a better chance of solving the whole problem together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who owns the guardrails?
In the Q&A, Debois first agrees to share the slides. The substantive follow-up asks how guardrails should be delivered: should they sit in front of every served model and be documented on its model card, or should product teams be responsible for consuming a separate guardrail service correctly?
Debois describes layered ownership. Central governance supplies the generic rules; individual teams add rules appropriate to their use cases. The shared component avoids duplicating common policy, while self-service lets teams address requirements specific to their products. His answer establishes those responsibilities without prescribing a particular enforcement topology. The platform provides a common baseline, and product teams build their own protections on top.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The earlier organizational patterns that helped inform Team Topologies.
An organizational pattern for improving the customer journey across products, services and teams.
IFAC's access page for Lisanne Bainbridge's foundational 1983 paper on human operators and automation.
Research connecting generative AI's evaluation burden and workflow disruptions with earlier human-factors findings.
Further reading
A practical guide to mapping team responsibilities and collaboration, service and facilitation interactions.
Research on AI-assisted programming, including code checking, debugging, developer experience and Copilot acceptance patterns.
Full transcript
Loading transcript…