Purpose and development
The shared AI platform
Two teams can request the same operation and need different promises about its performance. A sandbox runs submitted code in an environment that restricts what the code can affect. In his sandbox platform talk, Abhishek Bhardwaj of OpenAI compares two callers. A research team submits many trial runs, called rollouts, in parallel. An interactive product submits code for a person waiting for one result. The slide’s “Primary optimization” row pairs research with throughput, the number of trials the fleet finishes per unit of time. It pairs product use with latency, the time from the person’s request to their result.

The reliability and security rows change the service design too. Restarting a failed research rollout spends compute again. The research service must also protect internal resources and prevent submitted code from manipulating the test that rewards it. In the product, a failure interrupts someone’s task, and one customer’s code must be kept away from other customers’ work and data. A count of completed runs is informative for the researcher but could conceal a poor interactive experience. A product owner therefore needs a user-visible timing measurement as well as evidence of isolation.
For the proxy in Debois’s example, the corresponding questions are which applications may send calls, which providers may receive them and who investigates a failed call. Debois distinguishes working alongside feature teams to discover a need, offering a stable operation through an API and helping a team learn to use it. The resulting service can incorporate infrastructure bought from a provider. What its users depend on is the supported operation and the team that stands behind it.
Service boundaries
A service boundary separates the operation a team can request from the work it still owns. The table compares three different requests. Tailscale’s Aperture forwards a call to a model provider. OpenAI’s 2026 sandbox example runs code for a caller. Uber’s Michelangelo, described in 2017, supports a longer workflow. A developer trains a predictor on examples and releases it. An application can then send new inputs to that predictor for estimates, an operation called serving. Machine Learning Fundamentals
| Service | Consumer and request | Work the platform takes on | Decision the consumer retains |
|---|---|---|---|
| Tailscale · Aperture | An application requests a named model. | Identify the caller, resolve the provider and forward the call with provider credentials. | Decide whether the answer is suitable for the application’s next step. |
| OpenAI · research and product sandboxes | A research system or interactive product submits code. | Run the code within the specified execution boundary and operate the capacity for that workload. | Define a useful trial or user result and decide what to do with it. |
| Uber · Michelangelo | A model developer releases a predictor. UberEATS requests a delivery-time estimate. | Provide shared training, evaluation, deployment and prediction services. | Assess the delivery-time model for its intended use and decide how to use its estimate in the product. |
Where a rule takes effect
Where a check runs determines how a change reaches existing users. Suppose permission to send data to one provider ends on Friday. In a copied project template, editing the template changes future copies. In a library shipped with each application, a fix reaches callers as they deploy the new version. At an operated endpoint, the operator can change the check for existing clients and verify the rule on serving instances before Friday. Coverage depends on every affected call passing through that endpoint. Buying the endpoint or building it internally is a separate choice.
Write the contract in terms a caller can act on. Specify eligible users, supported inputs and outputs, permissions, operating limits, failure behavior and the first support contact. Assign custody of credentials and authority to release changes. Several teams might need only approved defaults and a credential setup tool. Others might need a shared model registry, a deployment service or a runtime that controls how long work runs and how it stops. Sharing any of these removes repeated work while introducing a dependency that someone must maintain. A need found during one customer integration merits shared support when other teams can use the same operation. Runtime responsibilities · Validating a shared capability
Model capabilities and their providers
Andrej Karpathy’s November 2023 LLM OS sketch helps locate capabilities around a large language model (LLM). The model occupies the CPU’s place. The context window resembles RAM containing the input the application has assembled for this call. Files and embeddings, numerical representations used for lookup, occupy the disk role. Storage and retrieval services have to supply their contents. The host supplies conventional tools such as a calculator, Python interpreter and terminal. Adapters provide audio and video, and network integrations connect a browser or other models. Drawing these connections grants no filesystem access, credentials or isolation. The host and service operators decide what a proposed call can reach. Original sketch · Context Engineering · Embeddings and Representation Learning
What survives a failed process
Karpathy’s sketch identifies capabilities a model can call on. In Anthropic’s Claude Managed Agents, a practical design question is which parts of that work must survive a process failure. A durable session log records events by appending them in order. The harness is the software that drives Claude’s loop of requesting tools and interpreting results. Tools and on-demand sandboxes execute the requests. Keeping the log separate lets the runtime replace a harness process or discard a sandbox while preserving the session history. The service manages credentials and grants outside the untrusted sandbox. Anthropic’s engineering account · What a harness controls
Priyanka Phatak and Gabe Cemaj’s workshop slide shows two arrows at the “brain.” One connects it to the session for reading and writing events. The other sends tool calls to the “hands.” Suppose the log records a requested write, a tool sends it, and the harness fails before recording a reply. A replacement harness can recover the request from the log even after the sandbox is gone. The missing reply leaves open whether the target applied the write. Check the operation’s status or resulting state at the target before issuing it again. If the outcome remains unknown, retry only if the target’s contract makes repetition safe. Claude Managed Agents workshop · Resolving uncertain effects

How platform obligations developed
The tension between interactive response and efficient sharing predates AI services. Batch processing made programmers wait to discover whether a fix worked, while exclusive console access left an expensive computer idle between commands. In 1962, Fernando Corbató, Marjorie Merwin-Daggett and Robert Daley described an MIT experiment that interleaved several users’ programs in short intervals. Programmers could interact with a shared machine, though the processing time available to each still depended on competing work. An Experimental Time-Sharing System
Release automation addressed a different source of delay. Code could pass its unit tests while its integration and deployment remained untested. Dave Farley’s 2007 deployment pipeline stored the compiled output of a commit so that successive checks could examine the same candidate. For a learned model, that principle extends beyond the executable. The data preparation used in production also has to agree with what the model encountered during training. The Deployment Pipeline
Before Michelangelo, Uber application engineers commonly built project-specific containers to serve predictions. Development of its shared training and serving machinery began in mid-2015 and expanded into common data pipelines. By the time Uber published its account in 2017, it reported production use across dozens of teams. That same year, Google described TensorFlow Extended (TFX), which connected data preparation, validation, training and serving. Google Play’s serving logs lacked some features, or model inputs, used to train its app-ranking model. An online experiment attributed a 2% improvement in homepage app-install rate to correcting that particular mismatch. TFX paper
Sharing infrastructure also raised a question of ownership. The 2019 first edition of Team Topologies distinguished building something together, helping another team learn to do the work, and providing an ongoing service. Debois later applied those distinctions to AI application teams. Hosted models put a similar boundary between companies. On June 11, 2020, OpenAI invited developers to request access to an API that accepted text and returned completions from GPT-3-family models. OpenAI operated the serving infrastructure and controlled upgrades and access. A team training its own predictor needs a way to release the model together with its input preparation. A team buying completions needs to know where its inputs go and whether the application still works after a provider change. Both need a way to verify the behavior their users receive. Team Topologies · OpenAI API announcement
Configuration and authority
From configuration to running work
Deploying a new predictor and asking for its next prediction exercise different parts of a service. The control plane manages resources and configuration. The data plane executes requests or jobs using those resources. A deployment request records a desired state. A controller repeatedly observes the current state and asks for changes that bring it closer to the desired one, a process called reconciliation. For example, the Kubernetes Job controller requests units of execution called Pods. Other components place and run them. Control and data planes · Kubernetes controllers
Consider a change from release A to release B. The caller needs a way to follow it after the configuration write succeeds. A long-running API can return an operation ID. Invalid input can fail immediately, while a failure to initialize B later belongs on the accepted operation. Document how to inspect progress, when that record expires and where to read the lasting deployment result. In the figure, follow the two workers separately. One can answer with A while the other reports that it has applied B’s configuration and is loading its model. A readiness check makes the second worker eligible for traffic. An invocation linked to B establishes which version answered that caller. Long-running operations · Readiness probes
Which worker answered?
ExampleAn accepted change and a live prediction have different witnesses.
The caller can inspect the operation for B. Worker 1 remains in service.
Read the diagram as text
- Requested release B.
- 09:00 · deployment accepted. The control plane recorded the desired release.
- Worker 1.
- Worker 2.
- 09:01 · B applied, model loading. Provisioned worker reports the new configuration.
- 09:01 · reply from A.
- 09:02 · ready for traffic. The configured readiness check passed.
- 09:03 · reply from B.
- 09:00 · deployment accepted → Requested release B: names desired version.
- 09:00 · deployment accepted → Worker 2: controller requests worker.
- Worker 2 → 09:01 · B applied, model loading: reports applied version.
- Worker 1 → 09:01 · reply from A: answers during loading.
- Worker 2 → 09:02 · ready for traffic: passes check.
- Worker 2 → 09:03 · reply from B: answers verification call.
- Accept the change. The caller can inspect the operation for B. Worker 1 remains in service. Active: Requested release B, 09:00 · deployment accepted, Worker 1. New: Requested release B, 09:00 · deployment accepted, Worker 1.
- Observe both workers. The new worker has applied B but is loading. The observed caller still received A. Active: Requested release B, 09:00 · deployment accepted, Worker 1, Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A. New: Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A.
- Verify a reply. The readiness result and the later reply record different facts about worker 2. Active: Requested release B, 09:00 · deployment accepted, Worker 1, Worker 2, 09:01 · B applied, model loading, 09:01 · reply from A, 09:02 · ready for traffic, 09:03 · reply from B. New: 09:02 · ready for traffic, 09:03 · reply from B.
A useful consequence of the separation is that existing work can sometimes continue when configuration updates stop. In OpenAI’s inference-routing design, a request handler chooses from a locally installed snapshot of engines and routing weights, using current engine health as a local check. New snapshots arrive asynchronously. Specify how long the service can use an old route and what happens when a change cannot propagate. Permission has a separate freshness requirement. If the service can no longer establish that a caller is authorized, a usable route alone cannot justify sending their data. OpenAI’s routing design · Static stability
Tenant authority and resource isolation
A tenant is a customer, project or group whose data and resource allowances the platform separates. An actor is the person or workload performing an operation. Two actors in one tenant may have different privileges. Authentication establishes identity. Authorization decides whether the identified actor may perform a particular action on a particular resource. Derive tenant scope from verified credentials or trusted session state. A model can propose a record ID, but the service must look up that ID within the scope it independently established. OWASP tenant isolation guidance
Consider the two reads associated with a queued job. When a worker starts the job, it needs the initiating user’s tenant scope, its own identity and a current grant to read the input artifact. If the grant expired while the job waited, the read fails. Later, a support engineer may need to inspect why it failed. The same job ID identifies the target, but the responder needs a separate grant for its diagnostic fields. Carry the trusted scope into cache lookups and administrative operations as well. Uber’s maintained client illustrates how to preserve delegation across hops. It exchanges credentials for short-lived tokens addressed to each recipient, preserving the human and agent identities so the recipient can apply its policy. Uber’s identity design · Enforcing current authority
One job, two protected reads
ExampleThe initiating user and a responder need grants on different resources.
Read the diagram as text
- Job with verified tenant and user.
- Authorize delegated artifact read. Identify the worker and verify its grant for this artifact.
- Return artifact bytes.
- Authenticated responder.
- Authorize diagnostic read. Check tenant scope, role and permitted fields.
- Return permitted fields.
- Deny requested read.
- Job with verified tenant and user → Authorize delegated artifact read: supplies delegated scope.
- Authorize delegated artifact read → Return artifact bytes: current grant covers artifact.
- Authorize delegated artifact read → Deny requested read: artifact grant missing or expired.
- Job with verified tenant and user → Authorize diagnostic read: identifies target tenant.
- Authenticated responder → Authorize diagnostic read: requests job diagnostics.
- Authorize diagnostic read → Return permitted fields: field grant covers request.
- Authorize diagnostic read → Deny requested read: no permitted field access.
A shared worker pool can give one tenant capacity that another leaves idle. Dedicated workers reduce contention on those machines at the expense of managing capacity for each pool. Both arrangements require access checks on shared storage and services. A namespace groups resources, and a quota bounds specified kinds of consumption. Uncovered resources, such as a shared network, can still carry interference. Retention determines how long records remain available at all. Give diagnostic fields a purpose, authorized readers and a disposal rule when designing their storage. Execution isolation · Artifact lifetimes
Model access and artifact lifecycle
The model-access contract
A model is a learned computation that produces a prediction or generated output. A catalog describes the models available to a developer. An endpoint accepts requests to execute one. A gateway mediates the call and can provide authentication, forwarding and metering even when the caller has already chosen the model. State how precisely the service can identify that caller. In Aperture, several people using a shared bridge can appear as one identity, so a usage entry for the bridge cannot by itself allocate consumption to a person. How Aperture works
Who operates the endpoint affects what its catalog entry can promise. In Microsoft’s 2024 Azure demonstration, customers using managed compute obtained graphics processor (GPU) quota, managed capacity and paid for virtual-machine hours. For selected models, the serverless option instead exposed a provider-operated API billed by input and output token, a unit of model input or output. A price per token leaves a developer with practical questions about limits, data handling and support. Azure model catalog demonstration
| Contract question | Provider operates serving | Internal team operates serving |
|---|---|---|
| Which inputs and outputs work? | Publish the formats and parameters supported by the specific endpoint. | Test the interface implemented by the installed model and server build. |
| Who may send which data? | Name the credential custodian, grant and revocation procedure, destination and provider processing terms. | Name the workload identity, permitted storage and processing locations, and any external recipients. |
| What identifies the running version? | Record the model ID returned when available and the provider’s guarantee for that ID. | Inspect the loaded package, serving build and applied settings. |
| What happens under load or failure? | Publish internal allowances, upstream limits, the first support contact and provider escalation route. | State ready-capacity assumptions, queue and rejection behavior, and who operates the pool. |
Even endpoints with the same request syntax can differ in what they honor. OpenRouter documents a require_parameters setting that limits routing to providers supporting every parameter supplied. Without it, a provider may ignore an unsupported parameter. Passing that check still leaves the question of whether the model does the task well. Optiver pairs a new model with test cases supplied by trading teams, client integration, usage recommendations and regional support. Its same-day release objective includes getting those pieces ready. OpenRouter’s parameter policy · Optiver’s rollout process · Checking a route · Authorizing its recipients
Release identity
An artifact is an identified item retained for reuse. An artifact registry stores or references versions and their metadata. Model weights are one artifact, the numerical values learned during training. A release also depends on how the server interprets those values, how inputs are prepared and how the application invokes the model. A release manifest records the combination. For a text application it can identify the weights or hosted model reference, tokenizer that encodes text, serving build, prompts, tool definitions and application settings. The test report can then name exactly which release it assessed. Tokenization · Identifying the system that ran
In MLflow a client can request a numbered model version or a mutable alias such as champion. If an operator reassigns champion, the next load resolves to the new target. A worker that already loaded the previous target can still answer from memory. Compare the resolved reference in the manifest with the loaded version recorded for an invocation. For downloadable content, an Open Container Initiative descriptor supplies a byte length and a digest, a hash of the content. A package from another mirror can be checked against the same trusted reference. MLflow registry workflows · OCI content descriptors
The test report identifies a release
ExampleA movable alias can resolve to one dependency without identifying the whole tested system.
Read the diagram as text
- Evaluation report.
- Release manifest.
- champion. Mutable alias.
- Model package. Resolved version with a trusted content reference.
- Input encoding.
- Serving build.
- Prompts, tools and settings.
- Source run.
- Evaluation report → Release manifest: identifies tested release.
- Release manifest → Model package: pins content.
- champion → Model package: currently resolves to.
- Release manifest → Input encoding: pins transformation.
- Release manifest → Serving build: pins executable.
- Release manifest → Prompts, tools and settings: pins configuration.
- Model package → Source run: records origin.
A digest match establishes byte identity. Provenance describes who or what produced the package and from which inputs. Behavioral evidence records what happened on particular tests. Those distinctions matter when reusing a package in a new application. Loading some model formats can execute code with the worker’s permissions. A trusted digest can identify those bytes perfectly while leaving loading safety and suitability for the new task unresolved. Give permission to publish a candidate separately from permission to approve its use. W3C provenance model · PyTorch security policy · AI Security
A hosted dependency may expose only a provider’s identifier. Anthropic documents that a pinned model ID retains its weights and model configuration, while routing, safety classifiers and sampling infrastructure around it can change. Record what the provider identifies, the settings under application control and the response metadata available at execution. Those records narrow an investigation if behavior changes, even when exact replay is unavailable. Model IDs and versioning
Approval and retirement
Track registration, approval, deployment and support separately for each version. The registry identifies what is available. An approval names the release and the use a reviewer authorized. A serving record identifies the version that handled a request. A support deadline tells consumers how long they have to migrate. If the replacement has approval but requests still reach the retiring version, the outstanding work is the cutover.
Before promotion, check that the dependencies can be fetched and used together. An evaluation then systematically tests the identified system against its intended use. The application owner supplies representative tasks and acceptance criteria. The platform can execute them and retain the configuration, results, reviewer and approval scope. In SageMaker’s documented project templates, marking a registered version Approved triggers deployment automation. Approval records the decision to proceed. The deployment operation and a subsequent invocation tell the operator what happened. SageMaker approval transitions · Evals and Benchmarks
Deprecation gives consumers notice while they can still migrate. Anthropic recommends using model and API-key usage to locate consumers and testing replacements on application tasks. Requests fail after retirement, and a hosting partner may use a different deadline. A practical procedure is to join observed usage to owner records, inspect manifests for infrequent jobs, notify each owner and verify calls to the replacement. A quiet week can hide a monthly job. Withdraw obsolete access after cutover and retain release evidence for its authorized lifetime. Provider retirement guidance
Change, capacity, and reliability
Deployment, exposure and recovery
Deployment makes a release executable. Exposure directs work to it. Bringing up a new process can leave a substantial interval between the two. The process may still need to import libraries, download weights and load them into memory. Modal allows initialization before a container becomes ready, so the delay occurs before assignment instead of inside the first invocation. Keeping spare workers initialized costs money while they sit idle. During replacement, count the workers that can actually answer, including the old ones that must remain available while replacements load. Modal cold starts
A canary assigns a limited live population to the candidate while the existing version serves a control population. Define a representative cohort, an observation window and conditions for promotion or stopping. Separate measurements by release so an error affecting the small cohort remains visible. Check infrastructure behavior such as latency alongside application behavior such as incorrect answers. A shadow instead receives permitted copies of requests while production supplies the answer. Budget its capacity, isolate its state and disable production writes. Discarding a shadow’s answer comes too late if its tool has already sent a message. Canarying releases
A route change leaves an earlier assignment alive
ExampleWithdrawing a candidate from new traffic can leave work to drain.
Existing traffic can use A during candidate startup.
Read the diagram as text
- Release A. Previously deployed version.
- Release B. Candidate version.
- Initial check · B loading.
- Later check · B ready.
- Control request.
- Canary request.
- First verified call after rollback.
- Still running at rollback. Status of the earlier canary assignment.
- Release B → Initial check · B loading: initially reports.
- Release B → Later check · B ready: later passes check.
- Control request → Release A: assigned to control.
- Canary request → Release B: assigned after readiness.
- First verified call after rollback → Release A: answered on restored route.
- Canary request → Still running at rollback: latest observed status.
- Load B alongside A. Existing traffic can use A during candidate startup. Active: Release A, Release B, Initial check · B loading. New: Release A, Release B, Initial check · B loading.
- Compare live cohorts. After the readiness check passes, the candidate and control receive distinct assignments. Active: Release A, Release B, Initial check · B loading, Later check · B ready, Control request, Canary request. New: Later check · B ready, Control request, Canary request.
- Stop new exposure. One new call confirms the route to A. Inspect or drain the assigned work on B independently. Active: Release A, Release B, Initial check · B loading, Later check · B ready, Control request, Canary request, First verified call after rollback, Still running at rollback. New: First verified call after rollback, Still running at rollback.
Draining stops new assignments while allowing earlier ones to finish. In Modal’s rolling strategy, existing containers handle inputs during the build, then new containers accept traffic as they become ready. Its recreate strategy terminates existing containers and retries their inputs, which requires the application to handle repetition safely. Specify what happens to queued jobs, long-lived streams and unfinished work at the drain deadline. A client that has already displayed part of an answer needs to know whether the stream ended prematurely before it can decide how to continue. Modal deployment strategies · Job recovery mechanisms · Interrupted serving
Rolling back directs new work to the earlier release. Verify which version answers a new call, then inspect work the candidate had already accepted. To stop further actions in a running session, Sachin Gupta recommends checking a kill switch at each decision point, including in child workers. For an action already sent to an external service, a timeout can leave the outcome unknown. Use its operation ID to find out whether the write committed. Retry only if repetition is safe. Reversing a completed effect requires a separate authorized action where one is available. Both the restored release and any remaining workers must respect current permissions. Agents Need Feature Flags · Cancellation semantics
Shared capacity and admission
Capacity is the work a service can finish under specified conditions. Consider a batch client submitting a model call for every row in a database while interactive clients wait for their first token. The batch can become a noisy neighbor by occupying resources the interactive clients also need. Counting requests hides differences in input size, output length, calls per task and deadlines. A platform can bound concurrency, reserve room for a workload class or reject work that would otherwise sit too long in a queue. Each choice changes which callers bear the wait. Inference Engineering
A quota sets a tenant’s allowance. A reservation holds an allocation for particular work. Admission permits work to proceed once the required checks pass. Placement assigns it to actual resources. In Kueue, batch jobs can wait for quota, borrow unused quota under a configured policy or displace admitted jobs through preemption. An admitted job still needs its workers placed. Similarly, a model caller below its internal quota may encounter an upstream limit or a worker still loading weights. Kueue concepts · Distributed placement
Where Ray Serve counts work
Concurrency belongs to the replica and waiting work belongs to the handle.
Read the diagram as text
- One replica.
- 2 assigned requests.
- One handle.
- 2 waiting requests.
- 5 further arrivals.
- One replica → 2 assigned requests: has assigned.
- One handle → 2 waiting requests: holds in its queue.
- One handle → 5 further arrivals: rejects while queue is full.
Ray Serve’s published overload example separates two limits that a dashboard might otherwise call queue size. One slow replica has two assigned requests. One handle, the caller-side object used to submit work, has two more requests waiting. Five further arrivals at that handle are rejected. Look at where each count attaches in the figure. A full queue explains those rejections, but another handle has its own queue. This example supplies no division of service time between tenants. Ray Serve load shedding
To compare allocation policies, replay the same arrivals and request sizes with the same ready capacity. Record deadlines met, jobs completed, work deferred and requests rejected for each workload class. An interactive request may meet its deadline because the scheduler interrupted a batch job. The cost to the batch depends on how much work it must repeat and whether it ever gets to finish. Before reclaiming its worker, verify where the job can resume. When it does resume, check that it still has permission to read its inputs. Provisioning for bursts and recovery
Attribute usage to the workload and tenant, documenting how idle or shared costs are assigned. Showback reports those assigned costs to the teams using the service. Chargeback posts the expense to their budgets or accounts. A budget can explain who pays even when every ready worker is occupied. FinOps terminology · Paying for the required service
Service objectives and shared failures
A service-level indicator measures a defined property of eligible events. A service-level objective sets a target for that indicator over a time window. For instance, a proposed target of 99.9% complete, timely replies over 30 days allows 0.1% of eligible calls to miss. That allowance is the error budget. Count valid calls within published limits even if the service rejects them under load. Identify calls outside those limits separately. Measure at the caller if the promise includes delivery, and agree on a decision the owner will take if misses exhaust the budget, such as pausing discretionary releases. Implementing service-level objectives
| Operation and eligible population | Observation that counts as success |
|---|---|
| Model access · authorized calls within contract | Caller receives the complete reply by the deadline. Answer quality has its own acceptance criteria. |
| Batch execution · accepted jobs in a stated class | The authorized consumer can retrieve the promised result by the job deadline. |
| Registry read · authorized reads of supported artifacts | Retrieved bytes match the trusted content descriptor. |
| Deployment · accepted changes within supported conditions | A verification call reaches the intended release within the promised interval. |
A common dependency in Cloudflare’s June 2025 outage
Separate product operations depended on the same storage path.
Read the diagram as text
- AI Gateway configuration.
- KV-backed asset delivery.
- Workers KV.
- Backing storage.
- Third-party infrastructure.
- AI Gateway configuration → Workers KV: reads configuration from.
- KV-backed asset delivery → Workers KV: reads assets from.
- Workers KV → Backing storage: depends on.
- Backing storage → Third-party infrastructure: partly depends on.
Distinct operations can share a failure domain, a dependency whose loss affects them together. On June 12, 2025, Cloudflare experienced an outage in the storage behind Workers KV, its key-value service. AI Gateway needed KV for configuration. Other services needed it for functions including asset delivery. AI Gateway’s error rate peaked at 97% of requests. Switching between two product endpoints offers little protection if both still require the same failing store. Cloudflare incident report
Exercise recovery at the boundary of each promise. With configuration updates disabled, try an authorized call against existing workers. After restoring a registry, fetch and verify a supported artifact. For a job, interrupt execution after an external write but before the reply is recorded. Check whether the replacement can determine what committed and finish without duplicating it. These exercises test three different things: continuity of serving, durability of release contents and recovery of unfinished work. Signals and outcomes · Testing interruption boundaries
Developer evidence and usefulness
Execution evidence
A useful request ID lets a developer find out what happened with ordinary tenant permissions. A trace links records of related operations, called spans. Consider a proposed diagnostic view for a model call. Starting with the ID, the caller can see each attempt, its elapsed time, the model requested, the model reported by the endpoint and the applicable release and policy revisions. A retry gets a separate attempt record. In the OpenTelemetry conventions, requested and reported model already have separate attributes. Release and policy identity need application instrumentation. GenAI span conventions · Traces and spans
Follow the release reference to the assessment run before deployment. Its report describes the tests, tools, limits and retry policy used there. A judgment about today’s answer, if one exists, belongs to today’s request and needs a source and time of its own. Follow the attempt reference to metered usage instead. A failed first attempt can consume resources even if the second succeeds. Reconcile those entries with provider billing before treating them as posted charges. A single invocation can therefore complete on time, receive an unfavorable user judgment and incur charges for two attempts. Attributable cost ledgers
Which record answers which question?
ExampleThe unit being assessed can be a release, an answer or a single billable attempt.
Read the diagram as text
- Request. Entry point visible to its authorized caller.
- Model attempt. Records requested and reported model and the wait visible at the service boundary.
- Applied release.
- Policy revision.
- Usage entry. Can be reconciled against the provider’s bill.
- Earlier release assessment.
- Optional answer feedback.
- Request → Model attempt: initiates.
- Model attempt → Applied release: runs under.
- Model attempt → Policy revision: checked under.
- Model attempt → Usage entry: incurs usage.
- Applied release → Earlier release assessment: refers to assessment.
- Optional answer feedback → Request: judges its answer.
An external service sets a limit on what a trace can explain. In the Weave remote-tool demonstration, a client called a tool through the Model Context Protocol, an interface for discovering and invoking tools. The client timed the call as a whole. To distinguish the time spent fetching a page from time spent preparing the result, the tool server had to record its own operations and link them to the initiating call. A slow fetch overlapping the call could belong to another request. Hand the tool owner the observed wait and request identity so they can return authorized, correlated evidence. Remote-tool demonstration · Model Context Protocol
Explain gaps at the point of lookup. A restricted field, a sampled trace and a delayed export leave different questions open. Show collection health and the last available observation before interpreting the absence of a record. Decide which identifiers and decision fields an investigation needs before capturing request bodies, who may inspect them and when they expire. Sensitive telemetry
The supported developer path
A golden path is a documented, supported way to finish a common task. For someone shipping a model-backed feature, the task extends from obtaining access to diagnosing a real invocation. The table proposes a path a team can test with a new developer. At each handoff, use only the credentials, instructions and outputs the preceding step actually supplied. This catches gaps such as a deployed application that returns a request ID its own developer cannot look up.
| Developer action | Usable output | Who helps if it fails? |
|---|---|---|
| Present identity and intended use | Scoped access that works at the documented test endpoint | Access owner checks the grant and destination. |
| Validate the client, then submit a test input | Local configuration result and an authorized integration response | Client maintainer checks the interface. Application owner assesses the response. |
| Submit an approved manifest | Deployment operation ID, followed by a verified invocation on the intended version | Release owner locates the failing deployment step. |
| Look up the invocation | Tenant-readable attempts, usage and any stated visibility limit | Diagnostics owner investigates access or collection. |
| Send the affected operation, impact and request ID | An acknowledged investigation with a named responder | Service on-call accepts or coordinates the handoff. |
A portal, command-line client and agent tool can expose the same well-defined API. Juan Herreros Elorza argues for making its validation callable early, while a developer or agent can still correct an input before a remote build. Modal’s edit-run demonstration makes a related distinction visible. A command issued from a laptop packages the edited code, runs it in the cloud and streams the output back. A local mock exercises less of the path. Even the remote test exercises only the identity, dependencies and data present in its test environment. Platforms for Humans and Machines · Modal’s demonstration
Show the ordinary operation first and expose extension settings where they become necessary. For example, an application can supply its own evaluator while using the same release identity, permissions and diagnostic records. Name who maintains it and include it in interface-change tests. When a team needs an exception, specify its scope, maintainer, security controls, escalation contact and a condition for review or retirement. That gives the team a supported route back when the common service evolves.
Adoption and platform evolution
Adoption means teams repeatedly use a capability to accomplish its intended task. Count verified deployments if deployment is the task. Count signups if access is the question. For a pilot, observe comparable releases before and after introducing the path, from a candidate ready to submit through an invocation on its intended version. Separate independent completions, assisted completions and abandoned attempts. Name the eligible teams and the paths the measurement misses. Mandatory enrollment can coexist with a release procedure that still requires a platform engineer every time.
Observe what happens during the wait. Ask a developer to show the last release where they needed help. Did they spend time editing configuration, waiting for approval or finding someone who could explain an error? Ask the operator what the successful release required from them. Include support interruptions, ongoing compatibility work and time to verify recovery. A shorter setup for every application can justify substantial common maintenance. Measuring both sides makes that judgment possible. DevEx research on workflow measures and developer perceptions
Michelangelo addressed repeated work building serving containers. Gitpod encountered a different pattern in development environments. Users expected persistent, mutable state, interactive bursts and the ability to run arbitrary tools. Supporting Docker and varied customer installations required additional compatibility and isolation machinery. In 2024 Gitpod described moving to a workload-specific control plane while retaining declarative APIs. The comparison suggests what to inspect in a pilot: how often the same operation recurs and how much variation the operator must absorb to offer it. Gitpod engineering retrospective
DORA’s 2024 survey associated internal developer platform use with higher individual productivity and team performance, alongside lower delivery throughput and change stability. Extra handoffs were one proposed explanation. The associations leave causation open. Use observed repeat success to decide where to expand, and repeated calls for help to locate the next fix. When a better interface makes a path obsolete, identify its consumers, support their migration and verify the replacement before withdrawal. DORA 2024 report
Accountability and reference
Operating responsibility
Taking a service on call requires agreement about objectives, staffing, training and authority to change it. The platform engineer supports the shared operation. The application engineer owns the business logic and judges whether the product works. A customer-facing engineer learns the customer’s process, integrates the service and establishes who can accept its operation locally. One person can cover several roles. In a failure, the distinction identifies who can repair the service and who can confirm that the affected task works again. Shared production responsibility · Customer operating ownership
| Decision | Accountable owner | Executor and evidence | Escalation |
|---|---|---|---|
| Contain a shared outage | Incident commander | Operations lead coordinates changes against impact and observed versions. | Service owner for additional authority or coverage |
| Engage an upstream provider | Platform service owner | Provider liaison supplies permitted request IDs, times and failure scope. | Provider support, with updates to command |
| Approve a model or artifact | Release authority | Release operator checks dependency identity, assessment and authorized use. | Application owner for disputed task criteria |
| Accept a product result | Application owner | Domain reviewer verifies the affected task on the observed release. | Application or platform on-call according to the defect |
| Grant an access exception | Resource owner | Authorized administrator records actor, purpose, resource, action and expiry. | Policy owner if the request exceeds delegated authority |
| Commit capacity | Serving owner | Capacity operator checks reservations, ready workers and competing deadlines. | Owners of affected service objectives |
| Retire a service | Service owner | Migration lead verifies cutovers, remaining dependencies and evidence retention. | Owners of unmigrated applications |
A runbook specifies a trigger, authorized actions, expected observations and an escalation route. Rehearse one failed rollout through a shift change. Can the responder actually redirect traffic or add capacity? In a Keycard demonstration, the user clicked approval for a production restart, but the service denied it because the user lacked that role. A scaling action permitted for the same user succeeded after approval. An exercise can reveal exactly where a runbook needs another executor.
At handoff, brief the incoming commander on impact, actions taken, observed releases and unfinished work. Name the remaining recovery test, such as an authorized request reaching the restored version followed by an application owner checking its result. Keep command until the successor acknowledges it, then announce the change to responders. An upstream provider may have repaired its fault while both of those checks remain outstanding. Google SRE incident management
Frequently asked questions
Is a model gateway an AI platform?
A gateway can supply an entire narrow platform service if teams need supported model access. It mediates calls, for example by supplying credentials and reporting usage. Teams that also need to release models or run code require other operations with their own contracts.
How does AI platform engineering relate to MLOps and internal developer platforms?
Machine learning operations (MLOps) includes preparing data, validating models and keeping predictions running. Large language model operations (LLMOps) includes concerns such as tracing prompt interactions and checking generated answers after a change. An internal developer platform gives engineering teams supported workflows. An AI platform can offer either set of operations through those workflows.
Can teams using only hosted models need a platform?
Yes. The provider can operate every model server while its customers still need credential setup, approved data destinations, usage attribution and help migrating before a model retires. Multiple teams doing that work can justify a common access service.
When is a library or purchased service sufficient?
A library suits an operation whose updates can wait for each consumer to redeploy. A rule that must change across existing traffic by a common deadline needs an enforcement point covering those calls. A purchased service can provide it when the update, authorization and support terms meet the requirement.
What does an AI platform engineer own?
The engineer owns the agreed shared interface and the work that keeps it usable, including operation, diagnostics, support and consumer migration. That can include coordinating a provider repair. The application team retains its product decisions and acceptance criteria.
Does successful deployment establish application quality?
Verifying a deployment establishes which release can answer. Evaluating quality requires inspecting what the answer means. In a research-agent example, a tool retrieved monthly US consumer price data for a question about five quarters. The call succeeded, but the returned periods made the data unsuitable for the requested analysis.
Does model access authorize a downstream action?
A model-access grant covers the model request. If the reply proposes a database read or a restart, the service protecting that resource must authorize the current actor, action and target. A worker acting for a user must preserve that delegation through the downstream call.
Open questions
How long can an installed configuration remain usable during an outage? A route may still work after a caller’s grant changes. An exercise that revokes access during a control-plane failure could measure the exposure interval alongside the availability preserved for other callers.
How much capacity should interactive users be able to reclaim? Preempting a batch can save a deadline or destroy hours of progress. Matched workload tests should report both classes’ completion times, rejected and deferred work, and the progress each displaced job can recover.
How can an application owner localize a behavioral change behind a pinned hosted model ID? Variation in output and changes to serving infrastructure can look alike. Request-linked serving revisions and repeated task tests under recorded conditions would give the owner evidence for distinguishing them.
When does shared maintenance reduce total effort? Setup improvements appear quickly, while support, migration and compatibility work accrue over many releases. Following comparable releases and recoveries across both consumer and operator teams would show where time was saved, where it moved and which workloads benefited.





























































