Contents
  1. Purpose and development
    1. The shared AI platform
    2. Service boundaries
      1. Where a rule takes effect
      2. Model capabilities and their providers
      3. What survives a failed process
    3. How platform obligations developed
  2. Configuration and authority
    1. From configuration to running work
    2. Tenant authority and resource isolation
  3. Model access and artifact lifecycle
    1. The model-access contract
    2. Release identity
    3. Approval and retirement
  4. Change, capacity, and reliability
    1. Deployment, exposure and recovery
    2. Shared capacity and admission
    3. Service objectives and shared failures
  5. Developer evidence and usefulness
    1. Execution evidence
    2. The supported developer path
    3. Adoption and platform evolution
  6. Accountability and reference
    1. Operating responsibility
    2. Frequently asked questions
      1. Is a model gateway an AI platform?
      2. How does AI platform engineering relate to MLOps and internal developer platforms?
      3. Can teams using only hosted models need a platform?
      4. When is a library or purchased service sufficient?
      5. What does an AI platform engineer own?
      6. Does successful deployment establish application quality?
      7. Does model access authorize a downstream action?
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

AI platform engineering

AI platform engineering is the practice of building and operating shared services that other teams use to develop, release and run AI systems. Patrick Debois describes those teams as customers in his 2024 talk AI Platform Engineering. In his LiteLLM example, an application sends a model request to a shared proxy. The proxy forwards the request to a provider and tracks spending across models. The platform team maintains that connection for its callers. Each application team decides whether it can use the answer it receives. The Cloud Native Computing Foundation’s Platforms White Paper makes the teams’ recurring tasks the starting point for choosing what to offer as a service.

Purpose and development

The shared AI platform

Two teams can request the same operation and need different promises about its performance. A sandbox runs submitted code in an environment that restricts what the code can affect. In his sandbox platform talk, Abhishek Bhardwaj of OpenAI compares two callers. A research team submits many trial runs, called rollouts, in parallel. An interactive product submits code for a person waiting for one result. The slide’s “Primary optimization” row pairs research with throughput, the number of trials the fleet finishes per unit of time. It pairs product use with latency, the time from the person’s request to their result.

Table comparing research and product sandbox platforms on optimization, reliability and security.
Research vs. product sandbox platforms — From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI · Abhishek Bhardwaj · Watch at 6:43

The reliability and security rows change the service design too. Restarting a failed research rollout spends compute again. The research service must also protect internal resources and prevent submitted code from manipulating the test that rewards it. In the product, a failure interrupts someone’s task, and one customer’s code must be kept away from other customers’ work and data. A count of completed runs is informative for the researcher but could conceal a poor interactive experience. A product owner therefore needs a user-visible timing measurement as well as evidence of isolation.

For the proxy in Debois’s example, the corresponding questions are which applications may send calls, which providers may receive them and who investigates a failed call. Debois distinguishes working alongside feature teams to discover a need, offering a stable operation through an API and helping a team learn to use it. The resulting service can incorporate infrastructure bought from a provider. What its users depend on is the supported operation and the team that stands behind it.

Service boundaries

A service boundary separates the operation a team can request from the work it still owns. The table compares three different requests. Tailscale’s Aperture forwards a call to a model provider. OpenAI’s 2026 sandbox example runs code for a caller. Uber’s Michelangelo, described in 2017, supports a longer workflow. A developer trains a predictor on examples and releases it. An application can then send new inputs to that predictor for estimates, an operation called serving. Machine Learning Fundamentals

ServiceConsumer and requestWork the platform takes onDecision the consumer retains
Tailscale · ApertureAn application requests a named model.Identify the caller, resolve the provider and forward the call with provider credentials.Decide whether the answer is suitable for the application’s next step.
OpenAI · research and product sandboxesA research system or interactive product submits code.Run the code within the specified execution boundary and operate the capacity for that workload.Define a useful trial or user result and decide what to do with it.
Uber · MichelangeloA model developer releases a predictor. UberEATS requests a delivery-time estimate.Provide shared training, evaluation, deployment and prediction services.Assess the delivery-time model for its intended use and decide how to use its estimate in the product.

Where a rule takes effect

Where a check runs determines how a change reaches existing users. Suppose permission to send data to one provider ends on Friday. In a copied project template, editing the template changes future copies. In a library shipped with each application, a fix reaches callers as they deploy the new version. At an operated endpoint, the operator can change the check for existing clients and verify the rule on serving instances before Friday. Coverage depends on every affected call passing through that endpoint. Buying the endpoint or building it internally is a separate choice.

Write the contract in terms a caller can act on. Specify eligible users, supported inputs and outputs, permissions, operating limits, failure behavior and the first support contact. Assign custody of credentials and authority to release changes. Several teams might need only approved defaults and a credential setup tool. Others might need a shared model registry, a deployment service or a runtime that controls how long work runs and how it stops. Sharing any of these removes repeated work while introducing a dependency that someone must maintain. A need found during one customer integration merits shared support when other teams can use the same operation. Runtime responsibilities · Validating a shared capability

Model capabilities and their providers

Andrej Karpathy’s November 2023 LLM OS sketch helps locate capabilities around a large language model (LLM). The model occupies the CPU’s place. The context window resembles RAM containing the input the application has assembled for this call. Files and embeddings, numerical representations used for lookup, occupy the disk role. Storage and retrieval services have to supply their contents. The host supplies conventional tools such as a calculator, Python interpreter and terminal. Adapters provide audio and video, and network integrations connect a browser or other models. Drawing these connections grants no filesystem access, credentials or isolation. The host and service operators decide what a proposed call can reach. Original sketch · Context Engineering · Embeddings and Representation Learning

What survives a failed process

Karpathy’s sketch identifies capabilities a model can call on. In Anthropic’s Claude Managed Agents, a practical design question is which parts of that work must survive a process failure. A durable session log records events by appending them in order. The harness is the software that drives Claude’s loop of requesting tools and interpreting results. Tools and on-demand sandboxes execute the requests. Keeping the log separate lets the runtime replace a harness process or discard a sandbox while preserving the session history. The service manages credentials and grants outside the untrusted sandbox. Anthropic’s engineering account · What a harness controls

Priyanka Phatak and Gabe Cemaj’s workshop slide shows two arrows at the “brain.” One connects it to the session for reading and writing events. The other sends tool calls to the “hands.” Suppose the log records a requested write, a tool sends it, and the harness fails before recording a reply. A replacement harness can recover the request from the log even after the sandbox is gone. The missing reply leaves open whether the target applied the write. Check the operation’s status or resulting state at the target before issuing it again. If the outcome remains unknown, retry only if the target’s contract makes repetition safe. Claude Managed Agents workshop · Resolving uncertain effects

Diagram connecting a durable session log, a stateless brain, and on-demand sandboxes and tools.
The agent became three separate interfaces — Decoupling the Brain from the Hands: Claude Managed Agents (Part 1 of 4) — Anthropic · Priyanka Phatak · Gabe Cemaj · Watch at 7:51

How platform obligations developed

The tension between interactive response and efficient sharing predates AI services. Batch processing made programmers wait to discover whether a fix worked, while exclusive console access left an expensive computer idle between commands. In 1962, Fernando Corbató, Marjorie Merwin-Daggett and Robert Daley described an MIT experiment that interleaved several users’ programs in short intervals. Programmers could interact with a shared machine, though the processing time available to each still depended on competing work. An Experimental Time-Sharing System

Release automation addressed a different source of delay. Code could pass its unit tests while its integration and deployment remained untested. Dave Farley’s 2007 deployment pipeline stored the compiled output of a commit so that successive checks could examine the same candidate. For a learned model, that principle extends beyond the executable. The data preparation used in production also has to agree with what the model encountered during training. The Deployment Pipeline

Before Michelangelo, Uber application engineers commonly built project-specific containers to serve predictions. Development of its shared training and serving machinery began in mid-2015 and expanded into common data pipelines. By the time Uber published its account in 2017, it reported production use across dozens of teams. That same year, Google described TensorFlow Extended (TFX), which connected data preparation, validation, training and serving. Google Play’s serving logs lacked some features, or model inputs, used to train its app-ranking model. An online experiment attributed a 2% improvement in homepage app-install rate to correcting that particular mismatch. TFX paper

Sharing infrastructure also raised a question of ownership. The 2019 first edition of Team Topologies distinguished building something together, helping another team learn to do the work, and providing an ongoing service. Debois later applied those distinctions to AI application teams. Hosted models put a similar boundary between companies. On June 11, 2020, OpenAI invited developers to request access to an API that accepted text and returned completions from GPT-3-family models. OpenAI operated the serving infrastructure and controlled upgrades and access. A team training its own predictor needs a way to release the model together with its input preparation. A team buying completions needs to know where its inputs go and whether the application still works after a provider change. Both need a way to verify the behavior their users receive. Team Topologies · OpenAI API announcement

Configuration and authority

From configuration to running work

Deploying a new predictor and asking for its next prediction exercise different parts of a service. The control plane manages resources and configuration. The data plane executes requests or jobs using those resources. A deployment request records a desired state. A controller repeatedly observes the current state and asks for changes that bring it closer to the desired one, a process called reconciliation. For example, the Kubernetes Job controller requests units of execution called Pods. Other components place and run them. Control and data planes · Kubernetes controllers

Consider a change from release A to release B. The caller needs a way to follow it after the configuration write succeeds. A long-running API can return an operation ID. Invalid input can fail immediately, while a failure to initialize B later belongs on the accepted operation. Document how to inspect progress, when that record expires and where to read the lasting deployment result. In the figure, follow the two workers separately. One can answer with A while the other reports that it has applied B’s configuration and is loading its model. A readiness check makes the second worker eligible for traffic. An invocation linked to B establishes which version answered that caller. Long-running operations · Readiness probes

In this proposed architecture, accepted R2 is still loading while application traffic goes only to ready R1. Runtime and routing observations return to the controller for another comparison; they do not guarantee successful rollout or answer quality.

Which worker answered?

Example

An accepted change and a live prediction have different witnesses.

1 / 3 · Accept the change

The caller can inspect the operation for B. Worker 1 remains in service.

New this stepRequested release B
New this step09:00 · deployment acceptedThe control plane recorded the desired release.
New this stepWorker 1
An illustrative deployment requests B. Worker 1 still answers with A while worker 2 prepares B. The final two observations establish eligibility and one actual reply on B. The earlier reply remains part of the history.
Read the diagram as text
  • Requested release B.
  • 09:00 · deployment accepted. The control plane recorded the desired release.
  • Worker 1.
  • Worker 2.
  • 09:01 · B applied, model loading. Provisioned worker reports the new configuration.
  • 09:01 · reply from A.
  • 09:02 · ready for traffic. The configured readiness check passed.
  • 09:03 · reply from B.
  • 09:00 · deployment accepted → Requested release B: names desired version.
  • 09:00 · deployment accepted → Worker 2: controller requests worker.
  • Worker 2 → 09:01 · B applied, model loading: reports applied version.
  • Worker 1 → 09:01 · reply from A: answers during loading.
  • Worker 2 → 09:02 · ready for traffic: passes check.
  • Worker 2 → 09:03 · reply from B: answers verification call.
  1. Accept the change. The caller can inspect the operation for B. Worker 1 remains in service. Active: Requested release B, 09:00 · deployment accepted, Worker 1. New: Requested release B, 09:00 · deployment accepted, Worker 1.
  2. Observe both workers. The new worker has applied B but is loading. The observed caller still received A. Active: Requested release B, 09:00 · deployment accepted, Worker 1, Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A. New: Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A.
  3. Verify a reply. The readiness result and the later reply record different facts about worker 2. Active: Requested release B, 09:00 · deployment accepted, Worker 1, Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A, 09:02 · ready for traffic, 09:03 · reply from B. New: 09:02 · ready for traffic, 09:03 · reply from B.

A useful consequence of the separation is that existing work can sometimes continue when configuration updates stop. In OpenAI’s inference-routing design, a request handler chooses from a locally installed snapshot of engines and routing weights, using current engine health as a local check. New snapshots arrive asynchronously. Specify how long the service can use an old route and what happens when a change cannot propagate. Permission has a separate freshness requirement. If the service can no longer establish that a caller is authorized, a usable route alone cannot justify sending their data. OpenAI’s routing design · Static stability

Tenant authority and resource isolation

A tenant is a customer, project or group whose data and resource allowances the platform separates. An actor is the person or workload performing an operation. Two actors in one tenant may have different privileges. Authentication establishes identity. Authorization decides whether the identified actor may perform a particular action on a particular resource. Derive tenant scope from verified credentials or trusted session state. A model can propose a record ID, but the service must look up that ID within the scope it independently established. OWASP tenant isolation guidance

Consider the two reads associated with a queued job. When a worker starts the job, it needs the initiating user’s tenant scope, its own identity and a current grant to read the input artifact. If the grant expired while the job waited, the read fails. Later, a support engineer may need to inspect why it failed. The same job ID identifies the target, but the responder needs a separate grant for its diagnostic fields. Carry the trusted scope into cache lookups and administrative operations as well. Uber’s maintained client illustrates how to preserve delegation across hops. It exchanges credentials for short-lived tokens addressed to each recipient, preserving the human and agent identities so the recipient can apply its policy. Uber’s identity design · Enforcing current authority

One job, two protected reads

Example

The initiating user and a responder need grants on different resources.

Job with verified tenant and user
Authorize delegated artifact readIdentify the worker and verify its grant for this artifact.
Return artifact bytes
Authenticated responder
Authorize diagnostic readCheck tenant scope, role and permitted fields.
Return permitted fields
Deny requested read
In this example both lookups use the verified job scope. A delegated worker may read the artifact only while its grant is current. A separately authenticated responder receives only authorized diagnostic fields. Either lookup can fail independently.
Read the diagram as text
  • Job with verified tenant and user.
  • Authorize delegated artifact read. Identify the worker and verify its grant for this artifact.
  • Return artifact bytes.
  • Authenticated responder.
  • Authorize diagnostic read. Check tenant scope, role and permitted fields.
  • Return permitted fields.
  • Deny requested read.
  • Job with verified tenant and user → Authorize delegated artifact read: supplies delegated scope.
  • Authorize delegated artifact read → Return artifact bytes: current grant covers artifact.
  • Authorize delegated artifact read → Deny requested read: artifact grant missing or expired.
  • Job with verified tenant and user → Authorize diagnostic read: identifies target tenant.
  • Authenticated responder → Authorize diagnostic read: requests job diagnostics.
  • Authorize diagnostic read → Return permitted fields: field grant covers request.
  • Authorize diagnostic read → Deny requested read: no permitted field access.

A shared worker pool can give one tenant capacity that another leaves idle. Dedicated workers reduce contention on those machines at the expense of managing capacity for each pool. Both arrangements require access checks on shared storage and services. A namespace groups resources, and a quota bounds specified kinds of consumption. Uncovered resources, such as a shared network, can still carry interference. Retention determines how long records remain available at all. Give diagnostic fields a purpose, authorized readers and a disposal rule when designing their storage. Execution isolation · Artifact lifetimes

Model access and artifact lifecycle

The model-access contract

A model is a learned computation that produces a prediction or generated output. A catalog describes the models available to a developer. An endpoint accepts requests to execute one. A gateway mediates the call and can provide authentication, forwarding and metering even when the caller has already chosen the model. State how precisely the service can identify that caller. In Aperture, several people using a shared bridge can appear as one identity, so a usage entry for the bridge cannot by itself allocate consumption to a person. How Aperture works

Who operates the endpoint affects what its catalog entry can promise. In Microsoft’s 2024 Azure demonstration, customers using managed compute obtained graphics processor (GPU) quota, managed capacity and paid for virtual-machine hours. For selected models, the serverless option instead exposed a provider-operated API billed by input and output token, a unit of model input or output. A price per token leaves a developer with practical questions about limits, data handling and support. Azure model catalog demonstration

Questions to answer for each supported model
Contract questionProvider operates servingInternal team operates serving
Which inputs and outputs work?Publish the formats and parameters supported by the specific endpoint.Test the interface implemented by the installed model and server build.
Who may send which data?Name the credential custodian, grant and revocation procedure, destination and provider processing terms.Name the workload identity, permitted storage and processing locations, and any external recipients.
What identifies the running version?Record the model ID returned when available and the provider’s guarantee for that ID.Inspect the loaded package, serving build and applied settings.
What happens under load or failure?Publish internal allowances, upstream limits, the first support contact and provider escalation route.State ready-capacity assumptions, queue and rejection behavior, and who operates the pool.

Even endpoints with the same request syntax can differ in what they honor. OpenRouter documents a require_parameters setting that limits routing to providers supporting every parameter supplied. Without it, a provider may ignore an unsupported parameter. Passing that check still leaves the question of whether the model does the task well. Optiver pairs a new model with test cases supplied by trading teams, client integration, usage recommendations and regional support. Its same-day release objective includes getting those pieces ready. OpenRouter’s parameter policy · Optiver’s rollout process · Checking a route · Authorizing its recipients

Release identity

An artifact is an identified item retained for reuse. An artifact registry stores or references versions and their metadata. Model weights are one artifact, the numerical values learned during training. A release also depends on how the server interprets those values, how inputs are prepared and how the application invokes the model. A release manifest records the combination. For a text application it can identify the weights or hosted model reference, tokenizer that encodes text, serving build, prompts, tool definitions and application settings. The test report can then name exactly which release it assessed. Tokenization · Identifying the system that ran

In MLflow a client can request a numbered model version or a mutable alias such as champion. If an operator reassigns champion, the next load resolves to the new target. A worker that already loaded the previous target can still answer from memory. Compare the resolved reference in the manifest with the loaded version recorded for an invocation. For downloadable content, an Open Container Initiative descriptor supplies a byte length and a digest, a hash of the content. A package from another mirror can be checked against the same trusted reference. MLflow registry workflows · OCI content descriptors

The test report identifies a release

Example

A movable alias can resolve to one dependency without identifying the whole tested system.

Evaluation report
Release manifest
championMutable alias.
Model packageResolved version with a trusted content reference.
Input encoding
Serving build
Prompts, tools and settings
Source run
In this example the release pins four dependencies. Its model package has a source run and is also the current alias target. Moving the alias changes future lookups while the test report continues to refer to the same manifest.
Read the diagram as text
  • Evaluation report.
  • Release manifest.
  • champion. Mutable alias.
  • Model package. Resolved version with a trusted content reference.
  • Input encoding.
  • Serving build.
  • Prompts, tools and settings.
  • Source run.
  • Evaluation report → Release manifest: identifies tested release.
  • Release manifest → Model package: pins content.
  • champion → Model package: currently resolves to.
  • Release manifest → Input encoding: pins transformation.
  • Release manifest → Serving build: pins executable.
  • Release manifest → Prompts, tools and settings: pins configuration.
  • Model package → Source run: records origin.

A digest match establishes byte identity. Provenance describes who or what produced the package and from which inputs. Behavioral evidence records what happened on particular tests. Those distinctions matter when reusing a package in a new application. Loading some model formats can execute code with the worker’s permissions. A trusted digest can identify those bytes perfectly while leaving loading safety and suitability for the new task unresolved. Give permission to publish a candidate separately from permission to approve its use. W3C provenance model · PyTorch security policy · AI Security

A hosted dependency may expose only a provider’s identifier. Anthropic documents that a pinned model ID retains its weights and model configuration, while routing, safety classifiers and sampling infrastructure around it can change. Record what the provider identifies, the settings under application control and the response metadata available at execution. Those records narrow an investigation if behavior changes, even when exact replay is unavailable. Model IDs and versioning

Approval and retirement

Track registration, approval, deployment and support separately for each version. The registry identifies what is available. An approval names the release and the use a reviewer authorized. A serving record identifies the version that handled a request. A support deadline tells consumers how long they have to migrate. If the replacement has approval but requests still reach the retiring version, the outstanding work is the cutover.

Before promotion, check that the dependencies can be fetched and used together. An evaluation then systematically tests the identified system against its intended use. The application owner supplies representative tasks and acceptance criteria. The platform can execute them and retain the configuration, results, reviewer and approval scope. In SageMaker’s documented project templates, marking a registered version Approved triggers deployment automation. Approval records the decision to proceed. The deployment operation and a subsequent invocation tell the operator what happened. SageMaker approval transitions · Evals and Benchmarks

Deprecation gives consumers notice while they can still migrate. Anthropic recommends using model and API-key usage to locate consumers and testing replacements on application tasks. Requests fail after retirement, and a hosting partner may use a different deadline. A practical procedure is to join observed usage to owner records, inspect manifests for infrequent jobs, notify each owner and verify calls to the replacement. A quiet week can hide a monthly job. Withdraw obsolete access after cutover and retain release evidence for its authorized lifetime. Provider retirement guidance

Change, capacity, and reliability

Deployment, exposure and recovery

Deployment makes a release executable. Exposure directs work to it. Bringing up a new process can leave a substantial interval between the two. The process may still need to import libraries, download weights and load them into memory. Modal allows initialization before a container becomes ready, so the delay occurs before assignment instead of inside the first invocation. Keeping spare workers initialized costs money while they sit idle. During replacement, count the workers that can actually answer, including the old ones that must remain available while replacements load. Modal cold starts

A canary assigns a limited live population to the candidate while the existing version serves a control population. Define a representative cohort, an observation window and conditions for promotion or stopping. Separate measurements by release so an error affecting the small cohort remains visible. Check infrastructure behavior such as latency alongside application behavior such as incorrect answers. A shadow instead receives permitted copies of requests while production supplies the answer. Budget its capacity, isolate its state and disable production writes. Discarding a shadow’s answer comes too late if its tool has already sent a message. Canarying releases

A route change leaves an earlier assignment alive

Example

Withdrawing a candidate from new traffic can leave work to drain.

1 / 3 · Load B alongside A

Existing traffic can use A during candidate startup.

New this stepRelease APreviously deployed version.
New this stepRelease BCandidate version.
New this stepInitial check · B loading
A serves while B loads. Once B is ready it receives a canary assignment while A receives control traffic. After the route is restored, a new request reaches A and the earlier assignment on B still requires a terminal outcome.
Read the diagram as text
  • Release A. Previously deployed version.
  • Release B. Candidate version.
  • Initial check · B loading.
  • Later check · B ready.
  • Control request.
  • Canary request.
  • First verified call after rollback.
  • Still running at rollback. Status of the earlier canary assignment.
  • Release B → Initial check · B loading: initially reports.
  • Release B → Later check · B ready: later passes check.
  • Control request → Release A: assigned to control.
  • Canary request → Release B: assigned after readiness.
  • First verified call after rollback → Release A: answered on restored route.
  • Canary request → Still running at rollback: latest observed status.
  1. Load B alongside A. Existing traffic can use A during candidate startup. Active: Release A, Release B, Initial check · B loading. New: Release A, Release B, Initial check · B loading.
  2. Compare live cohorts. After the readiness check passes, the candidate and control receive distinct assignments. Active: Release A, Release B, Initial check · B loading, Later check · B ready, Control request, Canary request. New: Later check · B ready, Control request, Canary request.
  3. Stop new exposure. One new call confirms the route to A. Inspect or drain the assigned work on B independently. Active: Release A, Release B, Initial check · B loading, Later check · B ready, Control request, Canary request, First verified call after rollback, Still running at rollback. New: First verified call after rollback, Still running at rollback.

Draining stops new assignments while allowing earlier ones to finish. In Modal’s rolling strategy, existing containers handle inputs during the build, then new containers accept traffic as they become ready. Its recreate strategy terminates existing containers and retries their inputs, which requires the application to handle repetition safely. Specify what happens to queued jobs, long-lived streams and unfinished work at the drain deadline. A client that has already displayed part of an answer needs to know whether the stream ended prematurely before it can decide how to continue. Modal deployment strategies · Job recovery mechanisms · Interrupted serving

Rolling back directs new work to the earlier release. Verify which version answers a new call, then inspect work the candidate had already accepted. To stop further actions in a running session, Sachin Gupta recommends checking a kill switch at each decision point, including in child workers. For an action already sent to an external service, a timeout can leave the outcome unknown. Use its operation ID to find out whether the write committed. Retry only if repetition is safe. Reversing a completed effect requires a separate authorized action where one is available. Both the restored release and any remaining workers must respect current permissions. Agents Need Feature Flags · Cancellation semantics

Shared capacity and admission

Capacity is the work a service can finish under specified conditions. Consider a batch client submitting a model call for every row in a database while interactive clients wait for their first token. The batch can become a noisy neighbor by occupying resources the interactive clients also need. Counting requests hides differences in input size, output length, calls per task and deadlines. A platform can bound concurrency, reserve room for a workload class or reject work that would otherwise sit too long in a queue. Each choice changes which callers bear the wait. Inference Engineering

A quota sets a tenant’s allowance. A reservation holds an allocation for particular work. Admission permits work to proceed once the required checks pass. Placement assigns it to actual resources. In Kueue, batch jobs can wait for quota, borrow unused quota under a configured policy or displace admitted jobs through preemption. An admitted job still needs its workers placed. Similarly, a model caller below its internal quota may encounter an upstream limit or a worker still loading weights. Kueue concepts · Distributed placement

Where Ray Serve counts work

Concurrency belongs to the replica and waiting work belongs to the handle.

One replica
2 assigned requests
One handle
2 waiting requests
5 further arrivals
Ray Serve’s published fixture specifies two requests assigned to the replica and two waiting at one handle. It expects that handle to reject five further arrivals while its queue is full. The arrows show where each count applies, without specifying an order of checks.
Read the diagram as text
  • One replica.
  • 2 assigned requests.
  • One handle.
  • 2 waiting requests.
  • 5 further arrivals.
  • One replica → 2 assigned requests: has assigned.
  • One handle → 2 waiting requests: holds in its queue.
  • One handle → 5 further arrivals: rejects while queue is full.

Ray Serve’s published overload example separates two limits that a dashboard might otherwise call queue size. One slow replica has two assigned requests. One handle, the caller-side object used to submit work, has two more requests waiting. Five further arrivals at that handle are rejected. Look at where each count attaches in the figure. A full queue explains those rejections, but another handle has its own queue. This example supplies no division of service time between tenants. Ray Serve load shedding

To compare allocation policies, replay the same arrivals and request sizes with the same ready capacity. Record deadlines met, jobs completed, work deferred and requests rejected for each workload class. An interactive request may meet its deadline because the scheduler interrupted a batch job. The cost to the batch depends on how much work it must repeat and whether it ever gets to finish. Before reclaiming its worker, verify where the job can resume. When it does resume, check that it still has permission to read its inputs. Provisioning for bursts and recovery

Attribute usage to the workload and tenant, documenting how idle or shared costs are assigned. Showback reports those assigned costs to the teams using the service. Chargeback posts the expense to their budgets or accounts. A budget can explain who pays even when every ready worker is occupied. FinOps terminology · Paying for the required service

Service objectives and shared failures

A service-level indicator measures a defined property of eligible events. A service-level objective sets a target for that indicator over a time window. For instance, a proposed target of 99.9% complete, timely replies over 30 days allows 0.1% of eligible calls to miss. That allowance is the error budget. Count valid calls within published limits even if the service rejects them under load. Identify calls outside those limits separately. Measure at the caller if the promise includes delivery, and agree on a decision the owner will take if misses exhaust the budget, such as pausing discretionary releases. Implementing service-level objectives

Example success conditions to pair with a target and reporting window
Operation and eligible populationObservation that counts as success
Model access · authorized calls within contractCaller receives the complete reply by the deadline. Answer quality has its own acceptance criteria.
Batch execution · accepted jobs in a stated classThe authorized consumer can retrieve the promised result by the job deadline.
Registry read · authorized reads of supported artifactsRetrieved bytes match the trusted content descriptor.
Deployment · accepted changes within supported conditionsA verification call reaches the intended release within the promised interval.

A common dependency in Cloudflare’s June 2025 outage

Separate product operations depended on the same storage path.

AI Gateway configuration
KV-backed asset delivery
Workers KV
Backing storage
Third-party infrastructure
Arrows point toward dependencies. AI Gateway configuration and KV-backed asset delivery both relied on Workers KV. Some of its backing storage depended on a third-party provider.
Read the diagram as text
  • AI Gateway configuration.
  • KV-backed asset delivery.
  • Workers KV.
  • Backing storage.
  • Third-party infrastructure.
  • AI Gateway configuration → Workers KV: reads configuration from.
  • KV-backed asset delivery → Workers KV: reads assets from.
  • Workers KV → Backing storage: depends on.
  • Backing storage → Third-party infrastructure: partly depends on.

Distinct operations can share a failure domain, a dependency whose loss affects them together. On June 12, 2025, Cloudflare experienced an outage in the storage behind Workers KV, its key-value service. AI Gateway needed KV for configuration. Other services needed it for functions including asset delivery. AI Gateway’s error rate peaked at 97% of requests. Switching between two product endpoints offers little protection if both still require the same failing store. Cloudflare incident report

Separate tenant compute retains shared initialization dependencies. Already loaded work can continue only with sufficient resources and the declared authorization-freshness contract; new initialization remains blocked while those dependencies are unavailable.

Exercise recovery at the boundary of each promise. With configuration updates disabled, try an authorized call against existing workers. After restoring a registry, fetch and verify a supported artifact. For a job, interrupt execution after an external write but before the reply is recorded. Check whether the replacement can determine what committed and finish without duplicating it. These exercises test three different things: continuity of serving, durability of release contents and recovery of unfinished work. Signals and outcomes · Testing interruption boundaries

Developer evidence and usefulness

Execution evidence

A useful request ID lets a developer find out what happened with ordinary tenant permissions. A trace links records of related operations, called spans. Consider a proposed diagnostic view for a model call. Starting with the ID, the caller can see each attempt, its elapsed time, the model requested, the model reported by the endpoint and the applicable release and policy revisions. A retry gets a separate attempt record. In the OpenTelemetry conventions, requested and reported model already have separate attributes. Release and policy identity need application instrumentation. GenAI span conventions · Traces and spans

Follow the release reference to the assessment run before deployment. Its report describes the tests, tools, limits and retry policy used there. A judgment about today’s answer, if one exists, belongs to today’s request and needs a source and time of its own. Follow the attempt reference to metered usage instead. A failed first attempt can consume resources even if the second succeeds. Reconcile those entries with provider billing before treating them as posted charges. A single invocation can therefore complete on time, receive an unfavorable user judgment and incur charges for two attempts. Attributable cost ledgers

Which record answers which question?

Example

The unit being assessed can be a release, an answer or a single billable attempt.

RequestEntry point visible to its authorized caller.
Model attemptRecords requested and reported model and the wait visible at the service boundary.
Applied release
Policy revision
Usage entryCan be reconciled against the provider’s bill.
Earlier release assessment
Optional answer feedback
Proposed diagnostic view. A request identifies its attempt. The attempt records release, policy and metered usage. A release refers to a prior assessment. Optional feedback refers to the answer from the individual request. These links can be exposed with tenant-scoped permissions.
Read the diagram as text
  • Request. Entry point visible to its authorized caller.
  • Model attempt. Records requested and reported model and the wait visible at the service boundary.
  • Applied release.
  • Policy revision.
  • Usage entry. Can be reconciled against the provider’s bill.
  • Earlier release assessment.
  • Optional answer feedback.
  • Request → Model attempt: initiates.
  • Model attempt → Applied release: runs under.
  • Model attempt → Policy revision: checked under.
  • Model attempt → Usage entry: incurs usage.
  • Applied release → Earlier release assessment: refers to assessment.
  • Optional answer feedback → Request: judges its answer.

An external service sets a limit on what a trace can explain. In the Weave remote-tool demonstration, a client called a tool through the Model Context Protocol, an interface for discovering and invoking tools. The client timed the call as a whole. To distinguish the time spent fetching a page from time spent preparing the result, the tool server had to record its own operations and link them to the initiating call. A slow fetch overlapping the call could belong to another request. Hand the tool owner the observed wait and request identity so they can return authorized, correlated evidence. Remote-tool demonstration · Model Context Protocol

Explain gaps at the point of lookup. A restricted field, a sampled trace and a delayed export leave different questions open. Show collection health and the last available observation before interpreting the absence of a record. Decide which identifiers and decision fields an investigation needs before capturing request bodies, who may inspect them and when they expire. Sensitive telemetry

The supported developer path

A golden path is a documented, supported way to finish a common task. For someone shipping a model-backed feature, the task extends from obtaining access to diagnosing a real invocation. The table proposes a path a team can test with a new developer. At each handoff, use only the credentials, instructions and outputs the preceding step actually supplied. This catches gaps such as a deployed application that returns a request ID its own developer cannot look up.

Developer actionUsable outputWho helps if it fails?
Present identity and intended useScoped access that works at the documented test endpointAccess owner checks the grant and destination.
Validate the client, then submit a test inputLocal configuration result and an authorized integration responseClient maintainer checks the interface. Application owner assesses the response.
Submit an approved manifestDeployment operation ID, followed by a verified invocation on the intended versionRelease owner locates the failing deployment step.
Look up the invocationTenant-readable attempts, usage and any stated visibility limitDiagnostics owner investigates access or collection.
Send the affected operation, impact and request IDAn acknowledged investigation with a named responderService on-call accepts or coordinates the handoff.

A portal, command-line client and agent tool can expose the same well-defined API. Juan Herreros Elorza argues for making its validation callable early, while a developer or agent can still correct an input before a remote build. Modal’s edit-run demonstration makes a related distinction visible. A command issued from a laptop packages the edited code, runs it in the cloud and streams the output back. A local mock exercises less of the path. Even the remote test exercises only the identity, dependencies and data present in its test environment. Platforms for Humans and Machines · Modal’s demonstration

Show the ordinary operation first and expose extension settings where they become necessary. For example, an application can supply its own evaluator while using the same release identity, permissions and diagnostic records. Name who maintains it and include it in interface-change tests. When a team needs an exception, specify its scope, maintainer, security controls, escalation contact and a condition for review or retirement. That gives the team a supported route back when the common service evolves.

Adoption and platform evolution

Adoption means teams repeatedly use a capability to accomplish its intended task. Count verified deployments if deployment is the task. Count signups if access is the question. For a pilot, observe comparable releases before and after introducing the path, from a candidate ready to submit through an invocation on its intended version. Separate independent completions, assisted completions and abandoned attempts. Name the eligible teams and the paths the measurement misses. Mandatory enrollment can coexist with a release procedure that still requires a platform engineer every time.

Observe what happens during the wait. Ask a developer to show the last release where they needed help. Did they spend time editing configuration, waiting for approval or finding someone who could explain an error? Ask the operator what the successful release required from them. Include support interruptions, ongoing compatibility work and time to verify recovery. A shorter setup for every application can justify substantial common maintenance. Measuring both sides makes that judgment possible. DevEx research on workflow measures and developer perceptions

Michelangelo addressed repeated work building serving containers. Gitpod encountered a different pattern in development environments. Users expected persistent, mutable state, interactive bursts and the ability to run arbitrary tools. Supporting Docker and varied customer installations required additional compatibility and isolation machinery. In 2024 Gitpod described moving to a workload-specific control plane while retaining declarative APIs. The comparison suggests what to inspect in a pilot: how often the same operation recurs and how much variation the operator must absorb to offer it. Gitpod engineering retrospective

DORA’s 2024 survey associated internal developer platform use with higher individual productivity and team performance, alongside lower delivery throughput and change stability. Extra handoffs were one proposed explanation. The associations leave causation open. Use observed repeat success to decide where to expand, and repeated calls for help to locate the next fix. When a better interface makes a path obsolete, identify its consumers, support their migration and verify the replacement before withdrawal. DORA 2024 report

Accountability and reference

Operating responsibility

Taking a service on call requires agreement about objectives, staffing, training and authority to change it. The platform engineer supports the shared operation. The application engineer owns the business logic and judges whether the product works. A customer-facing engineer learns the customer’s process, integrates the service and establishes who can accept its operation locally. One person can cover several roles. In a failure, the distinction identifies who can repair the service and who can confirm that the affected task works again. Shared production responsibility · Customer operating ownership

An example assignment of decision rights
DecisionAccountable ownerExecutor and evidenceEscalation
Contain a shared outageIncident commanderOperations lead coordinates changes against impact and observed versions.Service owner for additional authority or coverage
Engage an upstream providerPlatform service ownerProvider liaison supplies permitted request IDs, times and failure scope.Provider support, with updates to command
Approve a model or artifactRelease authorityRelease operator checks dependency identity, assessment and authorized use.Application owner for disputed task criteria
Accept a product resultApplication ownerDomain reviewer verifies the affected task on the observed release.Application or platform on-call according to the defect
Grant an access exceptionResource ownerAuthorized administrator records actor, purpose, resource, action and expiry.Policy owner if the request exceeds delegated authority
Commit capacityServing ownerCapacity operator checks reservations, ready workers and competing deadlines.Owners of affected service objectives
Retire a serviceService ownerMigration lead verifies cutovers, remaining dependencies and evidence retention.Owners of unmigrated applications

A runbook specifies a trigger, authorized actions, expected observations and an escalation route. Rehearse one failed rollout through a shift change. Can the responder actually redirect traffic or add capacity? In a Keycard demonstration, the user clicked approval for a production restart, but the service denied it because the user lacked that role. A scaling action permitted for the same user succeeded after approval. An exercise can reveal exactly where a runbook needs another executor.

At handoff, brief the incoming commander on impact, actions taken, observed releases and unfinished work. Name the remaining recovery test, such as an authorized request reaching the restored version followed by an application owner checking its result. Keep command until the successor acknowledges it, then announce the change to responders. An upstream provider may have repaired its fault while both of those checks remain outstanding. Google SRE incident management

Frequently asked questions

Is a model gateway an AI platform?

A gateway can supply an entire narrow platform service if teams need supported model access. It mediates calls, for example by supplying credentials and reporting usage. Teams that also need to release models or run code require other operations with their own contracts.

How does AI platform engineering relate to MLOps and internal developer platforms?

Machine learning operations (MLOps) includes preparing data, validating models and keeping predictions running. Large language model operations (LLMOps) includes concerns such as tracing prompt interactions and checking generated answers after a change. An internal developer platform gives engineering teams supported workflows. An AI platform can offer either set of operations through those workflows.

Can teams using only hosted models need a platform?

Yes. The provider can operate every model server while its customers still need credential setup, approved data destinations, usage attribution and help migrating before a model retires. Multiple teams doing that work can justify a common access service.

When is a library or purchased service sufficient?

A library suits an operation whose updates can wait for each consumer to redeploy. A rule that must change across existing traffic by a common deadline needs an enforcement point covering those calls. A purchased service can provide it when the update, authorization and support terms meet the requirement.

What does an AI platform engineer own?

The engineer owns the agreed shared interface and the work that keeps it usable, including operation, diagnostics, support and consumer migration. That can include coordinating a provider repair. The application team retains its product decisions and acceptance criteria.

Does successful deployment establish application quality?

Verifying a deployment establishes which release can answer. Evaluating quality requires inspecting what the answer means. In a research-agent example, a tool retrieved monthly US consumer price data for a question about five quarters. The call succeeded, but the returned periods made the data unsuitable for the requested analysis.

Does model access authorize a downstream action?

A model-access grant covers the model request. If the reply proposes a database read or a restart, the service protecting that resource must authorize the current actor, action and target. A worker acting for a user must preserve that delegation through the downstream call.

Open questions

  1. How long can an installed configuration remain usable during an outage? A route may still work after a caller’s grant changes. An exercise that revokes access during a control-plane failure could measure the exposure interval alongside the availability preserved for other callers.

  2. How much capacity should interactive users be able to reclaim? Preempting a batch can save a deadline or destroy hours of progress. Matched workload tests should report both classes’ completion times, rejected and deferred work, and the progress each displaced job can recover.

  3. How can an application owner localize a behavioral change behind a pinned hosted model ID? Variation in output and changes to serving infrastructure can look alike. Request-linked serving revisions and repeated task tests under recorded conditions would give the owner evidence for distinguishing them.

  4. When does shared maintenance reduce total effort? Setup improvements appear quickly, while support, migration and compatibility work accrue over many releases. Following comparable releases and recoveries across both consumer and operator teams would show where time was saved, where it moved and which workloads benefited.

Follow the curated reading path through the speakers and demonstrations behind this entry.

28 min

AI Engineer World's Fair 2024 · 2024

AI Platform Engineering

Patrick Debois

Cited in this entry

Patrick Debois explains how to discover a recurring need with feature teams, turn it into a service and help those teams operate their applications. His model proxy makes the division concrete.

Watch talk
19 min

AI Engineer World's Fair 2026 · 2026

Agents Need Feature Flags

Sachin Gupta

Cited in this entry

Sachin Gupta shows where stop checks belong so that a live session and its child workers can observe a change in permission.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

55 matching talks

Every catalogued talk on this subject: Infrastructure & deployment →

TalkSpeakerEventYear
Juan Herreros ElorzaAI Engineer Europe 20262026
Christopher Lovejoy, Saul HowardAI Engineer World's Fair 20262026
Nishant GuptaAI Engineer World's Fair 20262026
DottaAI Engineer Europe 20262026
Remy GuercioAI Engineer Europe 20262026
Harshil AgrawalAI Engineer Europe 20262026
Lovina DmelloAI Engineer World's Fair 20262026
Sonny Merla, Mauro Luchetti, Mattia RedaelliAI Engineer Europe 20262026
Uday Kiran Medisetty, Adam HudaAI Engineer World's Fair 20262026
AI SDK v6

Transcript reviewed

Nico AlbaneseAI Engineer Europe 20262026
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
Conquering Agent Chaos

Transcript reviewed

Rick BlalockAI Engineer World's Fair 20252025
Building security around ML

Cited in this entry

Dr. Andrew DavisAI Engineer World's Fair 20242024
Carter Abdallah, Vincent Weisser, Lucas Atkins, Chris AlexiukAI Engineer World's Fair 20262026
Andrew ThompsonAI Engineer World's Fair 20252025
Sandipan BhaumikAI Engineer Europe 20262026
Anju KambadurAI Engineer Summit 20252025
Jia WuAI Engineer World's Fair 20262026
Fryderyk WiatrowskiAI Engineer Europe 20262026
Skills are the New SDKs

Transcript reviewed

Elvin AghammadzadaAI Engineer World's Fair 20262026
Nachiket Paranjape, Swaroop Chitlur HaridasAI Engineer World's Fair 20262026
Jetashree RaviAI Engineer World's Fair 20262026
Kamil PotrecEvent unknownUnknown
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Peter WerryAI Engineer World's Fair 20262026
Lou BichardAI Engineer World's Fair 20252025
Sharmila Chokalingam, ShubhiAI Engineer World's Fair 20242024
Wolfram RavenwolfAI Engineer World's Fair 20262026
Patrick SmythEvent unknownUnknown
Annabell Schäfer, Doneyli De JesusEvent unknownUnknown
Jared JoselowitzAI Engineer World's Fair 20262026
Nishant Gupta, Naman AhujaAI Engineer World's Fair 20262026
Dylan PatelAI Engineer World's Fair 20242024
Matt BrockmanAI Engineer World's Fair 20262026
Marina PetzelEvent unknownUnknown
Eno ReyesAI Engineer World's Fair 20252025
Harish GaggarAI Engineer NYC 2026 Online Talks2026
Gautam KorlamAI Engineer NYC 2026 Online Talks2026
AI Engineer Summit 20252025
Amy Boyd, Nitya NarasimhanAI Engineer Europe 20262026
Kim MaidaAI Engineer NYC 2026 Online Talks2026
Rafal Wilinski, Vitor BaloccoAI Engineer World's Fair 20252025
Yegor Denisov-BlanchAI Engineer World's Fair 20252025
Charles FryeAI Engineer Summit 20232023
Kyle MisteleAI Engineer World's Fair 20262026
Omar KhattabAI Engineer World's Fair 20252025
Gergely Orosz, swyxAI Engineer Europe 20262026
Lei ZhangAI Engineer Code 20252025
Remy GuercioAI Engineer World's Fair 20262026
Felippe Felisola Caso, Francesco GallettaEvent unknownUnknown
Abdul DakkakAI Engineer World's Fair 20262026
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
John FeminellaEvent unknownUnknown
Safia AbdallaAI Engineer World's Fair 20262026
Sam Green, Aaron LinskyEvent unknownUnknown

References

Coverage and source review
Processed transcripts
47 processed in full · 7 in the curated path
Automated source review
Passed
Metadata candidates
13 unreviewed; not verified topic membership
Corpus version
c76f4395d1050f81b2bfd27df900224591ef94b026792d1b65896097daa9f77c

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI — Abhishek Bhardwaj

    https://www.youtube.com/watch?v=OqM67QG_Ikk&t=430s

  2. Research vs. product sandbox platforms — From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

    https://ai.engineer/talks/OqM67QG_Ikk-from-fork-fleet-designing-agent-sandbox-cloud?view=slides#transcript-OqM67QG_Ikk-asset-5

  3. AI Platform Engineering

    https://www.youtube.com/watch?v=5qNXdLbEdew&t=384

  4. AI Platform Engineering — Patrick Debois

    https://www.youtube.com/watch?v=5qNXdLbEdew&t=610s

  5. CNCF Platforms White Paper

    https://tag-app-delivery.cncf.io/whitepapers/platforms/

  6. Tailscale: How Aperture works

    https://tailscale.com/docs/aperture/how-aperture-works

  7. Meet Michelangelo: Uber’s Machine Learning Platform

    https://www.uber.com/us/en/blog/michelangelo-machine-learning-platform/

  8. Team Topologies: Thinnest Viable Platform Examples

    https://github.com/TeamTopologies/Thinnest-Viable-Platform-examples

  9. Agentic SDLC at Uber - Building Blocks for Uber’s Software Factory

    https://www.youtube.com/watch?v=17-YSUHo6Lk&t=106

  10. The Protection of Information in Computer Systems: Basic Principles

    https://web.mit.edu/Saltzer/www/publications/protection/Basic.html

  11. AI Platform Engineering

    https://www.youtube.com/watch?v=5qNXdLbEdew&t=483

  12. Google SRE: The Evolving SRE Engagement Model

    https://sre.google/sre-book/evolving-sre-engagement-model/

  13. LLM OS — Andrej Karpathy’s original diagram

    https://x.com/karpathy/status/1723140519554105733

  14. OWASP Access Control

    https://owasp.org/www-community/Access_Control

  15. Scaling Managed Agents with stable interfaces — Anthropic

    https://www.anthropic.com/research/managed-agents

  16. Decoupling the Brain from the Hands: Claude Managed Agents (Part 1 of 4) — Anthropic

    https://www.youtube.com/watch?v=Uv0nAf_7tyU&t=473s

  17. The agent became three separate interfaces — Decoupling the Brain from the Hands: Claude Managed Agents (Part 1 of 4) — Anthropic

    https://ai.engineer/talks/Uv0nAf_7tyU-decoupling-brain-from-hands-claude-managed-agents?view=slides#transcript-Uv0nAf_7tyU-asset-10

  18. gRPC lifecycle: cancellation is not rollback

    https://grpc.io/docs/what-is-grpc/core-concepts/

  19. An Experimental Time-Sharing System

    https://cseweb.ucsd.edu/classes/wi19/cse221-a/papers/corbato62.pdf

  20. The Deployment Pipeline — Dave Farley

    https://continuousdelivery.com/wp-content/uploads/2010/01/The-Deployment-Pipeline-by-Dave-Farley-2007.pdf

  21. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform

    https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2020_2021/papers/baylor_KDD_2017.pdf

  22. Team Topologies — Official Book Page

    https://teamtopologies.squarespace.com/book

  23. OpenAI API — June 11, 2020

    https://openai.com/index/openai-api/

  24. AWS Advanced Multi-AZ Resilience Patterns: Control planes and data planes

    https://docs.aws.amazon.com/whitepapers/latest/advanced-multi-az-resilience-patterns/control-planes-and-data-planes.html

  25. Kubernetes: Controllers

    https://kubernetes.io/docs/concepts/architecture/controller/

  26. Google AIP-151: Long-running operations

    https://google.aip.dev/151

  27. Kubernetes: Liveness, Readiness, and Startup Probes

    https://kubernetes.io/docs/concepts/workloads/pods/probes/

  28. Modal: Cold Start Performance

    https://modal.com/docs/guide/cold-start

  29. OpenTelemetry GenAI semantic conventions: spans

    https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

  30. Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

    https://www.youtube.com/watch?v=sOB3HSiG8vo&t=558

  31. AWS Fault Isolation Boundaries: Static stability

    https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/static-stability.html

  32. Kubernetes: Multi-tenancy

    https://kubernetes.io/docs/concepts/security/multi-tenancy/

  33. OWASP Multi-Tenant Security: tenant context and resource access

    https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html

  34. Uber Engineering: Solving the Identity Crisis for AI Agents

    https://www.uber.com/us/en/blog/solving-the-agent-identity-crisis/

  35. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.01162020.pdf

  36. Google Codelabs: Prototype to Production—Getting predictions from custom trained models

    https://codelabs.developers.google.com/vertex-p2p-predictions

  37. Accelerate your AI journey with Azure AI model catalog

    https://www.youtube.com/watch?v=gL9kfxt6uo0&t=585

  38. OpenRouter: Provider Routing

    https://openrouter.ai/docs/guides/routing/provider-selection

  39. How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver

    https://www.youtube.com/watch?v=aGIUg8zKI8A&t=430

  40. NIST Generative AI Profile: Third-Party Risk

    https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

  41. AWS: Shared Responsibility Model

    https://aws.amazon.com/compliance/shared-responsibility-model/

  42. Claude Platform: Model IDs and versioning

    https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions

  43. Ray Serve: Best Practices in Production

    https://docs.ray.io/en/latest/serve/production-guide/best-practices.html

  44. MLflow: Model Registry Workflows

    https://www.mlflow.org/docs/latest/ml/model-registry/workflow/

  45. Two-tower retrieval, approximation and compatible deployment artifacts

    https://www.tensorflow.org/recommenders/examples/efficient_serving

  46. From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

    https://www.youtube.com/watch?v=YIxzNZDDy1A&t=2946

  47. OCI Image Specification v1.1.0: Content Descriptors

    https://raw.githubusercontent.com/opencontainers/image-spec/v1.1.0/descriptor.md

  48. W3C: PROV-DM — The PROV Data Model

    https://www.w3.org/TR/prov-dm/

  49. PyTorch Security Policy: using models securely

    https://github.com/pytorch/pytorch/security/policy

  50. Amazon SageMaker AI: Update the Approval Status of a Model

    https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-approve.html

  51. Claude Platform: Model deprecations

    https://platform.claude.com/docs/en/about-claude/model-deprecations

  52. Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

    https://www.youtube.com/watch?v=obfPzbapMIM&t=2093

  53. One Registry to Rule them All - Sonny Merla, Mauro Luchetti, & Mattia Redaelli, Quantyca

    https://www.youtube.com/watch?v=VXfRt_H-V08&t=593

  54. Canarying Releases

    https://sre.google/workbook/canarying-releases/

  55. The Site Reliability Workbook — Canarying Releases

    https://sre.google/workbook/canarying-releases/

  56. Agents Need Feature Flags

    https://www.youtube.com/watch?v=zU4EagB311U&t=369

  57. Modal: Managing Deployments

    https://modal.com/docs/guide/managing-deployments

  58. Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

    https://www.youtube.com/watch?v=7c9FSUVcXR0&t=444

  59. Agents Need Feature Flags

    https://www.youtube.com/watch?v=zU4EagB311U&t=579

  60. Compensating Transaction pattern

    https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction

  61. RFC 7009: OAuth 2.0 Token Revocation

    https://www.rfc-editor.org/rfc/rfc7009.html

  62. Compute & System Design for Next Generation Frontier Models

    https://www.youtube.com/watch?v=gFyBdBm0AGo&t=470

  63. Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

    https://www.youtube.com/watch?v=7c9FSUVcXR0&t=126

  64. Google SRE: Handling Overload

    https://sre.google/sre-book/handling-overload/

  65. Kueue: Concepts

    https://kueue.sigs.k8s.io/docs/concepts/

  66. Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

    https://www.youtube.com/watch?v=7c9FSUVcXR0&t=541

  67. FinOps Foundation: FinOps Terminology

    https://www.finops.org/assets/terminology/

  68. Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

    https://www.youtube.com/watch?v=rTojoVotlD8&t=433

  69. Implementing SLOs — Google SRE Workbook

    https://sre.google/workbook/implementing-slos/

  70. Google SRE: Service Level Objectives

    https://sre.google/sre-book/service-level-objectives/

  71. AI Platform Engineering

    https://www.youtube.com/watch?v=5qNXdLbEdew&t=621

  72. Cloudflare Service Outage June 12, 2025

    https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/

  73. NIST SP 800-61r3: incident response and verified recovery

    https://doi.org/10.6028/NIST.SP.800-61r3

  74. Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

    https://www.youtube.com/watch?v=tZycGA_Fbfo&t=1172

  75. LangSmith: Feedback data format

    https://docs.langchain.com/langsmith/feedback-data-format

  76. Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

    https://www.youtube.com/watch?v=7c9FSUVcXR0&t=707

  77. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    https://www.youtube.com/watch?v=Lcqat4iP_lE&t=419

  78. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    https://www.youtube.com/watch?v=Lcqat4iP_lE&t=466

  79. Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

    https://www.youtube.com/watch?v=tZycGA_Fbfo&t=422

  80. Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

    https://www.youtube.com/watch?v=cCRO3ChaYhM&t=629

  81. Google SRE: Managing Incidents

    https://sre.google/sre-book/managing-incidents/

  82. Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

    https://www.youtube.com/watch?v=cCRO3ChaYhM&t=535

  83. Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

    https://www.youtube.com/watch?v=qeDPKbWjsuk&t=336

  84. Challenges to Scaling Agents for Generative AI Products

    https://www.youtube.com/watch?v=b2GqTDWtg6s&t=859

  85. Platform Engineering Maturity Model, version 1

    https://github.com/cncf/tag-app-delivery/blob/main/platforms-maturity-model/v1/index.md

  86. Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

    https://www.youtube.com/watch?v=cCRO3ChaYhM&t=1051

  87. DevEx: What Actually Drives Productivity

    https://www.michaelagreiler.com/wp-content/uploads/2024/06/DevEx-WhatDrivesProductivity.pdf

  88. Building CISO-approved agent fleet architecture

    https://www.youtube.com/watch?v=d9rsC6_VLoA&t=333

  89. We’re Leaving Kubernetes — Gitpod Engineering

    https://ona.com/stories/we-are-leaving-kubernetes

  90. DORA Accelerate State of DevOps Report 2024

    https://dora.dev/research/2024/dora-report/2024-dora-accelerate-state-of-devops-report.pdf

  91. Your Agent Is a Lethal Trifecta (And What to Do About It) — Kim Maida, Keycard

    https://www.youtube.com/watch?v=ws9XcGGPBDk&t=363

  92. One Registry to Rule them All - Sonny Merla, Mauro Luchetti, & Mattia Redaelli, Quantyca

    https://www.youtube.com/watch?v=VXfRt_H-V08&t=344

  93. Challenges to Scaling Agents for Generative AI Products

    https://www.youtube.com/watch?v=b2GqTDWtg6s&t=811

  94. The Update Framework: authenticated update workflow

    https://theupdateframework.github.io/specification/latest/

  95. Building security around ML

    https://www.youtube.com/watch?v=GVbPiq3Pet0&t=1285

  96. Agents Need Feature Flags

    https://www.youtube.com/watch?v=zU4EagB311U&t=1097

  97. The Protection of Information in Computer Systems

    https://www.cs.virginia.edu/~evans/cs551/saltzer/

  98. Monitor Your Application — Ray Serve

    https://docs.ray.io/en/master/serve/monitoring.html