AI Engineer World's Fair 2024

AI Platform Engineering

Loading slides…

Read the talk

AI Platform Engineering: From Pilot Teams to Production

Scaling GenAI across an organization requires shared infrastructure, practical enablement, and governance that helps application teams build and operate useful products.

From a talk by Patrick Debois

Before you start: Familiarity with application development, APIs, and basic deployment practices will help; no model-training background is required.

What happens after the first AI pilot?

You have a team experimenting with GenAI. What will it take for the rest of the organization to use it? Patrick Debois approaches that question through an earlier transformation: he organized the first DevOpsDays in 2009, with about 60 attendees, and subsequently watched DevOps spread through enterprises. That experience supplies a useful lens, with a qualification he makes explicit: applying an old paradigm to a new technology can reveal a pattern—or get it completely wrong.

His interest in GenAI starts with automation intelligence. Much of his DevOps work involved automating pipelines, making them robust, and ensuring that they delivered value. AI extends that work, but it also raises the familiar question of how a promising capability becomes something an organization can reliably use.

The adoption pattern begins with one pilot team. Two or three more teams try the technology; their experience exposes repeatable patterns; those patterns then support wider adoption. Cloud infrastructure followed this path. Shared abstractions eventually let application teams move faster within their own domains without understanding every part of the underlying infrastructure.

Orange slide with three groups of blue and yellow circles labeled Incubate, Repeat, and Scale above a Maturity line.Open full source frame
Introducing organizational and technical change: Incubate, Repeat, Scale.

Even the unsettled name of the practice has a function. Debois recalls the arguments over DevOps, alongside earlier movements around agile and ITIL. A label such as AI engineer gives people a way to find emerging stories, job postings, and one another. Defining it too rigidly, too early, can restrict what the practice becomes.

0:190:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:19 · section reference included

Production work changes the team

In Debois's experience building GenAI applications over two years, the first organizational friction appeared when the work landed with data scientists who were not accustomed to running production services. Engineers began moving into the data science team. As integration became a larger part of the work, engineering's relative share increased. Capabilities then spread to other teams and eventually moved into the platform.

That movement also reconnected data that had been isolated in a data lake with the applications that needed it. Debois calls this a shift right, contrasting it with security's shift left: responsibilities move as teams discover where the friction actually lies.

Bringing development and operations into the same room was a start, but permanent, intensive collaboration did not scale across an organization. Kubernetes pipelines, infrastructure, and security tooling became shared capabilities. Team Topologies, which grew out of DevOps Topologies, provides a vocabulary for this arrangement: feature teams build and run their applications on top of a platform.

The platform team can interact with those teams in three ways:

InteractionWhat the platform team does
Service consumptionProvides an API or service the team can use
CollaborationWorks with the team to shape the capability it needs
FacilitationHelps the team overcome problems and learn

The platform is a product, and application teams are its customers. Its purpose is to help traditional application developers, supported by data science expertise, deliver AI features within their own teams.

3:594:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:59 · section reference included

Model access and reusable data infrastructure

The first platform capability is model access. Instead of every team independently investigating providers and obtaining access, the platform team determines what is appropriate for the company. Existing cloud relationships often influence that choice, making CloudOps a natural collaborator. Debois presents the vendors in this part of the talk as examples, without claiming affiliation.

Retrieval-augmented generation adds a second layer: vector storage and search. Dedicated vector databases appeared early, while traditional database vendors increasingly incorporated vector capabilities. That makes the infrastructure less exotic, but teams still need to understand vectors and embeddings to use it well. The slide places products such as Chroma, Vespa, and Milvus alongside PostgreSQL, Redis, and SingleStore.

Pink slide titled Unstructured data - Vector Databases, with database logos arranged in two columns, including Chroma, Vespa, Milvus, PostgreSQL, Redis, and SingleStore.Open full source frame
Dedicated vector databases alongside databases that support vector search.

A vector database alone does not connect an application to company knowledge. The platform also needs data connectors that expose internal sources to multiple teams. Building these once avoids having each feature team rediscover how to integrate the same systems. This ongoing responsibility for connecting and operating retrieval infrastructure is sometimes called RAGOps.

Model version control completes this initial foundation. A central model repository or registry makes versions visible and reusable, much as a shared code library makes existing work discoverable. Teams should not have to invent their own model-storage and discovery practices independently.

7:417:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:41 · section reference included

Control access and observe quality

Larger enterprises often want independence from a particular model provider. A shared proxy can mediate model access while enforcing who may use which services. Debois tentatively compares the OpenAI API's possible role to S3's role in storage: a common interface around which providers might converge. This is a historical forecast in the talk, not a claim of a formal standard or verified compatibility.

Once applications run in production, ordinary request monitoring is insufficient. The platform needs to capture prompts and trace their relationships. Debois describes an initial prompt leading to five or six iterations or questions, so the useful unit of observation is often a sequence rather than a single API call. Providing that tracing centrally saves each team from assembling it independently.

An available API can still produce poor results. Production monitoring therefore needs checks on data and output quality, extending practices familiar from MLOps. Continuously running evaluations act as another kind of health check: they can help reveal model changes or unusual user behavior that an uptime check would miss.

Caching and feedback collection are further shared services. Feedback should extend beyond thumbs up, thumbs down, and retry: inline edits reveal how a user would correct the result. Building and managing these capabilities well is expensive enough that duplicating them across teams is wasteful. The growing ecosystem resembles Kubernetes in its breadth; an AI platform is already more than cloud access and a few API keys.

9:449:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:44 · section reference included

Help teams experiment—and find a use case

Providing infrastructure does not automatically make it usable. Like any product team, the platform team must guide its customers and learn from their experience. Enablement is both instruction and a feedback channel: it reveals which capabilities help and which get in the way.

Several forms of support make experimentation easier:

  • Prototyping tools: Include product owners, who need to explore potential use cases as much as engineers do. Connect those tools securely to company data.
  • Frameworks: Give developers a way to discover what is possible, supported by education and documentation suited to their existing stacks.
  • Local development: Let developers experiment in a familiar environment, including while traveling. Improving laptop models can shorten iteration loops, although local stacks have tradeoffs.

These are ways to help teams learn, not merely additional items in a service catalog.

The next obstacle is choosing work worth doing. Mandating GenAI in every part of a product can put implementation ahead of a real use case; Debois compares some of this activity to spending a marketing budget. He also warns that an excessive focus on model training and fine-tuning can postpone getting anything into production.

Premature cost optimization creates a similar distraction. Insisting on a local or cheaper model before establishing the business case can consume engineering effort without producing value. Debois's ordering is to establish the return first, then reduce the cost of delivering it.

Once users provide feedback, someone must interpret it. Engineers accustomed to error stacks may not know what a qualitative correction means for the product. The product owner often holds the domain knowledge needed to distinguish an acceptable answer from a misleading one. Collecting feedback without involving that expertise leaves the improvement loop unfinished.

12:1512:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:15 · section reference included

Model changes make evaluation essential

Developer experience includes some unusually difficult choices: which model to start with, how to obtain representative test data, and how much to depend on a fast-changing framework. Debois describes developers first learning through frameworks, then abandoning them after disruptive changes and rebuilding against lower-level APIs. A prompt call may be simple, but tracing, monitoring, and the surrounding integrations still have to work together. His hope is for a dependable tool ecosystem rather than every team rebuilding that machinery.

Debois recalls using approximately eight LLM models over two years while pursuing better quality. Each change required rewriting and reworking application prompts, and the work was manual. From an engineering-management perspective, that combined substantial cost with little assurance that the replacement would behave correctly. Establish evaluations before refactoring prompts for a new model.

Evaluation becomes easier to discuss when it starts with checks engineers already understand. A character-count requirement or regular expression has a definite result. For example, a TypeScript check for the talk's twenty-character constraint can be expressed directly:

typescript

function hasTwentyCharacters(output: string): boolean {
  return Array.from(output).length === 20;
}

const outputs = ["Your order is ready.", "Ready."];
const results = outputs.map(output => ({
  output,
  passes: hasTwentyCharacters(output),
}));

console.log(results);

This checks a format requirement; it does not establish whether the answer is useful or correct.

The evaluation ladder then moves beyond deterministic rules:

MethodWhat it checks
Sentiment helper modelThe answer's sentiment
Semantic comparisonRelatedness, such as question–answer relevance
LLM judgeQualities assessed by another language model
Human feedbackHuman assessment of the result

Using an LLM to judge another LLM introduces a reliability question rather than eliminating one. Debois treats that concern as unresolved, while recognizing model-based judging as a practical tool. The engineering approach combines these methods instead of relying on an informal impression alone.

15:3415:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:34 · section reference included

Faster generation moves work into review

Enablement also has an emotional component. Some engineers initially fear AI; giving them a coding copilot lets them experience its usefulness directly. That puts the team building GenAI capabilities in a second role: helping colleagues become comfortable using AI in their own work.

Debois reports that copilots produced more code, larger AI-generated pull requests, and longer review times. He does not claim that review overhead entirely canceled the generation gains. The point is that speeding up one stage can create friction at the next, so code generation alone is an incomplete account of productivity.

He connects this tension to Ironies of Automation and a newer generative-AI treatment of the same problem, Ironies of Generative AI. The person using automation moves from producing work toward managing and reviewing it. Generation becomes faster while evaluation occupies more of the person's time.

That role change becomes dangerous when the reviewer no longer understands the work well enough to assess it. Without the domain model or hands-on experience, accepting an output can become easier than evaluating it. Debois illustrates this with a weekend Copilot acceptance-rate anecdote; his humorous explanation that developers simply care less is not an established cause. The deeper concern is declining situational awareness: less practice producing the work can leave people less prepared to recognize and handle failures.

18:3018:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:30 · section reference included

Automation creates a duty to prepare for failure

The early promise of DevOps could sound like replacing a person's work with a shell script. What grew around that automation was a much larger discipline of preparing for failure. Debois illustrates the balance as 20% automation and 80% failure preparation, an organizing metaphor rather than a measured industry allocation.

The progression matters:

  1. CI/CD and tests add checks around automated delivery.
  2. Production monitoring reveals whether the running system is healthy.
  3. Designing for failure begins by assuming things will break.
  4. Observability gives operators the situational awareness to understand what is happening.
  5. Chaos engineering injects failures so teams can keep practicing their response.

Automation removes some work, but these practices help people remain capable when it fails.

The purpose is also to reduce cognitive load during an incident, when people have the least capacity to reconstruct an unfamiliar system. Not every organization adopts every practice; the appropriate investment depends on its risk appetite. But the work of enablement extends to helping engineers feel prepared, not simply making them faster.

Teal slide maps infrastructure as code, CI/CD, monitoring, resilient engineering, observability, and chaos engineering to automation and failure preparedness, with Reduce Cognitive Load below.Open full source frame
20% automation, 80% prepare for failure.
20:5621:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:56 · section reference included

Governance must survive everyday use

Governance starts with personal awareness. A tool that records the screen may promise productivity while also exposing everything visible on it. Teams need to understand what they are copying or transmitting, and how a service handles those inputs. Training opt-outs deserve particular attention because where and when to configure them may not be obvious.

A restricted set of approved models makes license review more manageable. The word open does not by itself establish permission to use a model; its license and origin still need examination. Debois also includes basic awareness of European workflow risk assessment: teams should ask whether a proposed use is allowed and appropriate without expecting every engineer to become a lawyer.

Operational controls require equal care. Debois treats prompt injection as an unsolved problem for which defenses are still necessary. He compares guardrails to intrusion detection and web application firewalls: a growing pile of rules and logs can eventually train people to ignore alerts. Installing a control can create reassurance without ensuring that anyone can operate it effectively.

More uncontrolled text crossing service boundaries also increases the need to scrutinize personally identifiable information, with useful metrics and alerting around what is transmitted. These responsibilities bring the platform's remit into focus: operate shared infrastructure, enable teams to use it, and establish company rules for that use.

22:1222:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:12 · section reference included

Connect the platform to the product experience

The platform's internal focus leaves another coordination problem: how AI appears to customers across the product. Debois introduces the unFIX Experience Crew as a complementary organizational idea from an alternative to Team Topologies. Applied to AI, the crew works across feature teams to understand what they are building and make the resulting user experience consistent.

That adds an experience layer to the combination of CloudOps, SecOps, developer experience, the data platform, and AI infrastructure. Debois advises against this structure for a 10-person company and suggests considering it when scaling to 10 or more teams. It is a judgment about coordination needs, not a requirement to create departments before there is enough work to justify them.

The benefit comes from collaboration between specialties. SecOps brings access-control and governance expertise; CloudOps knows provisioning and cloud vendors. Bringing those capabilities into a shared area gives teams a better chance of solving the whole problem together.

24:4225:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:42 · section reference included

Who owns the guardrails?

In the Q&A, Debois first agrees to share the slides. The substantive follow-up asks how guardrails should be delivered: should they sit in front of every served model and be documented on its model card, or should product teams be responsible for consuming a separate guardrail service correctly?

Debois describes layered ownership. Central governance supplies the generic rules; individual teams add rules appropriate to their use cases. The shared component avoids duplicating common policy, while self-service lets teams address requirements specific to their products. His answer establishes those responsibilities without prescribing a particular enforcement topology. The platform provides a common baseline, and product teams build their own protections on top.

26:4226:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:42 · section reference included

Resources

From the talk

  • The earlier organizational patterns that helped inform Team Topologies.

  • An organizational pattern for improving the customer journey across products, services and teams.

  • IFAC's access page for Lisanne Bainbridge's foundational 1983 paper on human operators and automation.

  • Research connecting generative AI's evaluation burden and workflow disruptions with earlier human-factors findings.

Transcript download
Full transcript

Loading transcript…