Contents
  1. Purpose and workload
    1. Optimize acceptable work
    2. Foundational developments
    3. Describe the demand
  2. Workload economics
    1. Price accepted outcomes
      1. A cheaper call, a dearer result
    2. Separate savings from allocation
  3. Time and consequential delay
    1. Budget the completion path
    2. Find the delay worth removing
  4. Avoided work and valid reuse
    1. Remove unnecessary execution
    2. Define valid cache reuse
    3. Measure the value of reuse
      1. A bounded break-even calculation
  5. Usable capacity
    1. Distinguish activity from throughput
    2. Provision for bursts and recovery
      1. Ready capacity must exceed new demand
    3. Pay for the required service
  6. Optimization experiments
    1. Test the complete workload
    2. Compare interventions across layers
  7. Adoption and maintenance
    1. Choose and revisit the operating point
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

AI cost & performance

AI cost and performance engineering reduces the resources needed to deliver acceptable results within the time users can wait. It examines the complete workflow: model calls, tools, verification, retries, human review, and the capacity kept ready to serve them. The goal is to remove unnecessary work and consequential delays while preserving quality and operating requirements. A cheaper call matters only if the resulting workflow remains useful and economical.

Purpose and workload

Optimize acceptable work

Start by defining acceptance criteria: observable conditions that make a completed task useful. A generated database query may need to return the intended records, respect access restrictions, and finish while someone is waiting. Producing valid query syntax meets only part of that contract. An evaluation systematically assesses behavior against its intended use; defining worthwhile improvement establishes the baseline and checks that this chapter takes as inputs.

Separate a constraint, which a candidate must satisfy, from an objective, which you improve among acceptable candidates. Quality, completion deadlines, failure rates, and review burden can constrain a cost-reduction study. Interactive completion and background processing need different operating points because the user waits differently. Hacking the Inference Pareto Frontier develops this application-first framing: establish the required quality and latency before minimizing cost.

A service-level indicator, or SLI, defines a quantitative observation; a service-level objective, or SLO, gives it a target over a specified population and window. A response-time target needs start and end events, an observation location, and an eligible population. Receiving an acknowledgement, seeing useful partial output, and receiving a completed usable result are different boundaries. A system can improve one while leaving another unchanged.

Foundational developments

Whole-workload optimization brings together several older engineering traditions. Each addresses a different source of expense or delay; their coexistence explains why improving model execution alone rarely settles the complete decision.

  • Critical-path scheduling — 1959 — Kelley and Walker's Critical-Path Planning and Scheduling represented activities and dependencies as a network. Completion depended on the longest required path, making the placement of work as consequential as its duration.
  • Amdahl's argument — 1967 — Gene Amdahl's whole-problem analysis challenged acceleration claims that neglected sequential data management and irregular work. Unchanged portions of a computation constrain the benefit of faster arithmetic.
  • The working-set model — May 1968 — Peter Denning's working-set paper connected scheduling with recently referenced information. Too little retained memory caused repeated retrieval; too much consumed capacity that other work needed.
  • Cloud elasticity — February 2009 — Armbrust and colleagues' Berkeley report, Above the Clouds, explained how acquiring and releasing resources with demand could reduce both idle provisioning and insufficient capacity. Hourly price alone could not describe workload economics.
  • Joint agent optimization — 2024 — Kapoor and colleagues' AI Agents That Matter separated one-time optimization expense from recurring execution cost and studied quality and cost together. Deployment volume therefore became part of judging an optimization's value.

These developments are complementary, not a replacement sequence. Dependency analysis identifies consequential waiting; memory analysis explains reuse; elasticity changes what capacity must be purchased; task-level optimization asks whether the resulting work remains worthwhile.

Describe the demand

Use three nested units. A task is the requested outcome. An attempt is one execution trying to achieve it. A call invokes a model, tool, or service within an attempt. One task may require several attempts, and each attempt may contain many calls. Preserve these relationships in the execution records used for analysis; counting API calls alone cannot describe task volume or expense.

Tokens are model-specific units used to encode inputs and outputs; count the complete request, including its formatting and supporting context. Keep input and output counts separate. Prefill processes supplied input, whose tokens are already known and can be processed together. Decode incrementally produces output: each new token depends on the preceding choices. Swapping a long output for an equally long input therefore changes execution demand even when total token count stays constant. Inference Engineering explains those mechanics.

Build a profile for each meaningful task class, then compare its share of task volume, model work, and total expense.
Profile dimensionKeep visible
ExecutionAttempts per task; model and tool calls per attempt; failures and cancellations
SizeInput and output distributions, not only their averages
TimingArrival bursts, pauses between related calls, and completion deadlines
ReuseRepeated inputs, compatible prefixes, and actual reuse outcomes
OutcomeAcceptance, review effort, and unresolved assessment

Observed traffic can depart sharply from an average request model. BurstGPT found working-hour patterns in its conversation-service traces and more irregular bursts in API traffic. Separately, the KV-cache testing suite preserves multi-request sessions, client delays, and nested subagents. A session trace is not necessarily an accepted task, and matching text hashes indicate potential reuse rather than actual cache residency. Use representative sampling to establish which workload these records describe.

Workload economics

Price accepted outcomes

Unit economics measures cost and value per meaningful unit of delivered work. Start with the attributable cost ledger, then state the task cohort, follow-up horizon, included costs, and allocation rules. Include model calls, tools, retrieval, storage, unsuccessful attempts, cancellation-related work, and material human review. Do not add provider charges twice through both request records and reconciled billing totals.

cattempt=CA,ccompleted=CD,caccepted=CK.c_{\mathrm{attempt}}=\frac{C}{A},\qquad c_{\mathrm{completed}}=\frac{C}{D},\qquad c_{\mathrm{accepted}}=\frac{C}{K}. Here CC is the entire scoped cohort cost, AA is executed attempts, DD is terminally completed tasks, and KK is known acceptable outcomes. Terminal completion includes finalized failures, not just success. Cancelled and unfinished tasks remain separately recorded, with their incurred costs included. Each ratio requires a nonzero denominator; no accepted outcomes makes cost per accepted outcome undefined, not zero.

A cheaper call, a dearer result

Consider this constructed comparison of the same 100 tasks. Each attempt makes one model call. Both configurations finish all tasks within the required time, all outcomes are assessed, and 90 are accepted. Review costs $1 per reviewed task; the remaining included cost is $10 per configuration.
QuantityConfiguration AConfiguration B
Attempts120150
Charge per model call$0.10$0.06
Model charges$12$9
Reviewed tasks / review cost8 / $820 / $20
Other included cost$10$10
Total scoped cost$30$39
Cost per attempt$0.25$0.26
Cost per completed task$0.30$0.39
Cost per accepted outcome$0.333$0.433

B saves $3 in model charges but adds $12 in review, increasing total cost by $9. Its cheaper calls do not compensate for the changed workflow. The review amount is an assigned labor cost; whether reducing those hours would reduce payroll is a separate decision developed next.

Keep assessment coverage beside every outcome ratio. Recent tasks may not yet have downstream results, and selectively reviewed tasks may differ from unreviewed ones. Absence of a recorded failure is not acceptance. Compare cohorts at the same follow-up horizon and report unresolved counts. A definitive deadline miss can already be known even while eventual correctness remains unresolved. Incomplete-feedback analysis explains the limits of inference from such records.

Separate savings from allocation

Marginal cost is the added cost of more work. Fully allocated cost assigns a workload its share of all included expenses. Avoidable cost is future expenditure that changes because of a decision. These quantities differ: freeing already-paid staff time or committed compute creates capacity, but does not necessarily reduce cash expenditure. Scarce released capacity can still have value because it can perform other work.

The same reduction in processing work has different consequences under different purchasing arrangements.
ArrangementImmediate consequenceWhen expenditure falls
Metered consumptionFewer chargeable unitsAs avoided usage reduces charges under the applicable tariff
Existing capacity commitmentMore spare capacityWhen capacity can be reduced or a future purchase avoided
Removable serving unitsLess work per unitWhen enough capacity becomes removable without violating service requirements

Define the planning horizon before calling a cost fixed or variable. A server may be fixed for the current contract but removable at renewal; adding another required replica creates a step in cost. Allocate shared resources transparently, using measured consumption or staff activity where appropriate, and retain estimated or unallocated amounts. Ownership accounting can include acquisition, energy, support, software, and labor. Depreciation spreads an asset's cost over its useful life; do not also charge its full purchase price to the same period.

Use dated price rules and reconcile usage with billing, while preserving their coverage differences. Anthropic's documented Usage and Cost API, for example, has separate reports with different inclusions and reporting delays. Provider totals cannot supply the application's task-to-attempt mapping or external review costs. Include startup, idle capacity, telemetry, evaluation, and maintenance when they materially change the decision.

Time and consequential delay

Budget the completion path

A latency budget allocates an end-to-end allowance across necessary execution and waiting. Reuse the observation boundaries already established for the service. Time to first token, or TTFT, concerns the first token or content arrival at a named observation point; it need not be the first useful result. Streaming can improve early responsiveness without shortening final completion.

The critical path is the longest required dependency path determining earliest completion under the represented constraints. An operation starts no earlier than the latest finish among its prerequisites. Shared-resource serialization and additional waiting must also be represented; do not count prerequisite waiting again inside execution duration. Slack is the delay an activity can absorb without delaying completion under that schedule.

For example, suppose a request waits 100 ms for admission, then spends 600 ms generating a candidate. A 200 ms format check and a 400 ms evidence check start together once generation finishes. Release requires both checks and takes another 100 ms. Completion takes 100 + 600 + 400 + 100 = 1,200 ms: the checks overlap, so only the longer check determines when release can start. A 1,500 ms allowance leaves 300 ms of reserve without omitting either required check.

Parallel checks, one release boundary

Parallel checks, one release boundaryAdmission 0–100, generation100–700, format700–900 and evidence700–1100 in parallel, release1100–1200 milliseconds. Arrows show prerequisites; the1500 millisecond allowance leaves300 reserve, not executed work.030060090012001500Admission wait0–100 msGenerate candidate100–700 msFormat check700–900 msEvidence check700–1100 msRelease result1100–1200 ms300 ms reserveElapsed time (ms) · arrows are prerequisites; rows are not sequential
The example finishes release at 1,200 ms, leaving 300 ms before the 1,500 ms allowance. Both checks require generation; release waits for both checks. The reserve is not executed work, and the schedule does not establish when a first useful output is exposed.

The placement of verification is part of the service contract. Hippocratic AI's June 2026 architecture account describes parallel supervisors: some block current outputs, including medication-guidance checks, while others can intervene on later turns. Moving a check off the current response path changes when protection applies. It does not make verification free or establish equivalent protection. Likewise, when human approval is required before completion, its queue and review time belong inside the budget.

Later operations inherit the remaining deadline, not a fresh allowance. Before another attempt, account for backoff, execution, and the work still needed afterward. Propagating the original deadline helps avoid consuming resources for a caller that has already stopped waiting.

Find the delay worth removing

First estimate how much of the completion path an intervention can change. Amdahl's central warning remains useful: acceleration of one portion leaves the rest. For a fixed workload, the familiar speedup bound makes that limitation explicit.

S=1(1−f)+f/s.S=\frac{1}{(1-f)+f/s}. Here ff is the fraction of baseline duration accelerated, ss is that portion's speedup, and SS is whole-workload speedup. If 40% of a 10-second path becomes twice as fast, duration becomes 6+4/2=86+4/2=8 seconds: a 1.25× speedup. Even eliminating that portion leaves six seconds. This assumes unchanged remaining work, dependencies, overhead, and contention.

Dependency structure can impose a tighter limit. Accelerating one of two equally long critical paths leaves the other intact. Conversely, apparently noncritical work may matter if finishing it releases a scarce resource. After an improvement, bottleneck migration means a different constraint now determines completion; remeasure rather than applying the original fraction repeatedly.

Tail latency describes slow responses; p95, the 95th percentile, is a threshold at or above roughly 95% of observations. Better averages can hide deadline failures. Dean and Barroso's 2013 The Tail at Scale explains fan-out: parallel requests expose completion to slow dependencies. Which dependencies must finish, and whether their delays coincide, determines the effect.

Each example has 100 two-stage tasks, timed in milliseconds. A contains 94 pairs (1,1), three (10,1), and three (1,10); B contains 97 pairs (1,1) and three (10,10). Stage distributions match, each with p95 of 1 ms. Total p95 differs: A is 11 ms; B is 2 ms. Component percentiles cannot be added.

Same stages, different pairing

Same stages, different pairingIdentical stage distributions, different pairing. A has94 totals at2 and6 at11 milliseconds: p95 is11. B has97 totals at2 and3 at20 milliseconds: p95 is2 despite a larger maximum. Step curves use matching axes.A · slow stages on different tasks0%50%100%02112094% at 2 ms < 95%p95 = 11 ms · max = 11 msTotal latency (ms)B · slow stages on the same tasks0%50%100%02112097% at 2 ms ≥ 95%p95 = 2 ms · max = 20 msTotal latency (ms)Cumulative fraction · dashed line = 95% · steps, not interpolated observations
Example datasets with 100 tasks each: A has 94 totals of 2 ms and 6 of 11 ms; B has 97 totals of 2 ms and 3 of 20 ms. Stage distributions match. At the first observed total covering at least 95% of tasks, A has p95 = 11 ms and B p95 = 2 ms; B still has the larger maximum.

Prioritize an intervention by the delay it can remove, the tasks exposed to that delay, and its attainable improvement. Then check the complete latency distribution again. Reducing a frequent short operation and removing a rare deadline-breaking wait solve different problems.

Avoided work and valid reuse

Remove unnecessary execution

Additional computation can be valuable: another attempt may solve a difficult task, and verification may prevent a bad result. The target is work that does not justify its resources through better accepted outcomes. Distinguish removing an unnecessary model call, retaining a completed computation, sharing one already underway, and allowing another independent attempt.

Memoization retains results for later matching calls. Request coalescing shares an execution still in flight. Python's lru_cache can execute overlapping misses twice despite thread-safe bookkeeping. Go's singleflight runs one same-key call within a Group; duplicate callers wait for its result. The application must establish safe sharing. Neither persistent caching nor cross-process coordination follows from this contract.

Sharing a result requires equivalent result obligations. It does not authorize replaying an external action: safe repetition needs its own contract. Retries also add demand when a dependency may already be overloaded. Independent loops can multiply attempts across layers. Use retryable-error distinctions, backoff, per-request limits, and shared retry budgets; an attempt budget is not a monetary budget when operations have different prices.

Stopping the caller's wait is not necessarily stopping execution. The gRPC deadline contract makes the server application responsible for terminating activities it spawned after cancellation. Measure abandoned work rather than assuming its computation or billing vanished. Harness Engineering owns those controls; Reasoning and Test-Time Compute owns decisions to spend additional computation deliberately.

Changing where models are used can matter more than changing models. Fink and colleagues' May 2026 radiology report-labelling study retained rule-valid fields and sent only rule-invalid fields to a language model. Across paired comparisons of 22 models on 2,923 structured computed tomography (CT) pulmonary angiography reports from one regional reporting network, median cohort processing time fell from 6.7 hours to about one hour, and reported marginal cost from €1.19 to €0.17. The reference standard retained rule-valid fields, potentially favoring hybrids; costs excluded development and full operating overhead. This supports investigating call elimination, not assuming rules always win.

Define valid cache reuse

A cache entry retains an artifact for possible reuse. A cache hit finds an entry that satisfies both a matching rule and a validity rule. Matching proposes that the work corresponds; validity establishes that reusing it is still permitted and suitable. First identify the artifact, because different caches avoid different operations.

Reuse typeRetained artifactWork avoided
Deterministic preprocessingTransformation resultRepeating the same valid transformation
Exact response reuseCompleted answerGenerating another answer for an equivalent request
Semantic response reuseAnswer plus matching representationsGeneration, if approximate matching preserves answer validity
Compatible prefix reuseNumerical model stateProcessing the already-computed prefix, not new generation

A key-value cache, or KV cache, stores numerical attention state derived from processed tokens. Cross-request prefix reuse requires compatible preceding state, not similar meaning. vLLM's versioned prefix-cache design includes preceding-block identity, exact token IDs, and additional identifiers such as adapters and multimodal content. Reusing this state skips some prefill; the model must still process new input and generate the requested continuation.

Embeddings are learned vector representations that can propose semantically similar requests. Similarity does not establish identical answers: the same question can refer to different records or conversation contexts. MeanCache compares conversation context after matching query representations, illustrating why the latest sentence alone is insufficient. Neither test establishes authorization or freshness. Representation-learning limits explain why nearby vectors can preserve the wrong distinctions.

An application-level validity contract must account for result-affecting inputs, transformation and model versions, configuration, source revisions, and required variability. Tenant and permission context must come from verified identity, not a model-proposed identifier. Recheck changed authority before disclosing retained results. A request requiring a fresh random sample or a new external effect cannot generally be satisfied by replaying an old return value.

Invalidation makes an entry ineligible after a relevant change. Eviction removes retained state to manage capacity. A time to live, or TTL, limits retention or reuse age under a specified policy; it does not promise that the source stayed unchanged. In the reported credit-decision incident, a database write succeeded but failed invalidation left a downstream agent using the old score. Faster retrieval of that value was not a successful optimization.

Measure the value of reuse

Temporal locality means information is reused near its previous use. A working set is the information referenced over a relevant interval. Denning's original model used recent process execution: retaining too little caused repeated retrieval, while retaining too much consumed memory. Replacement policy alone could not repair excessive aggregate demand. The modern application is to balance avoided recomputation against the space and time occupied by retained state.

A miss penalty is the work or delay incurred when reuse is unavailable. Request hit rate counts each request equally, regardless of the expense avoided. Facebook's 2013 Scaling Memcache distinguished access frequency from miss expense when allocating cache pools. It also addressed a cache stampede: many readers simultaneously fall back to expensive storage after invalidation. A lease allowed one client to refill a missing entry while competing readers briefly waited. If another invalidation occurred before that refill finished, the stale refill was rejected.

Measure avoided execution expense against lookup, fill, retention, refresh, transfer, and validation expense over the same interval. Track saved input processing, generated output, transferred bytes, charges, and completion time separately. A hit can save expensive computation without shortening a different critical path. Semantic reuse that returns the wrong answer remains a quality failure, even when it avoids every model call.

A bounded break-even calculation

For a fixed eligible prefix with ordinary input charge BB, one cache write at multiplier ww followed by rr reads at multiplier aa costs B(w+ar)B(w+ar). Without caching it costs B(1+r)B(1+r). Prefix-charge savings require r>w−11−a,a<1.r>\frac{w-1}{1-a},\qquad a<1. In Claude's pricing inspected August 30, 2026, a=0.1a=0.1, while five-minute and one-hour writes used w=1.25w=1.25 and w=2w=2. These require respectively one and two subsequent reads. This excludes suffixes, outputs, tools, and cache-management overhead; expiry or changed prefixes can require another write.

Longer retention helps only when it spans enough valid reuse. In Context Platform Engineering, longer lifetimes bridge more pauses but increase the retained working set. GPU high-bandwidth memory (HBM), host memory (DRAM), and external storage offer different places to retain compatible KV state. A lower storage tier can hold reusable state yet deliver it too slowly to outperform recomputation. Test cold misses, warm reuse, transfer overhead, and mixed-tier concurrency separately; neither a larger cache nor a warm single-request result establishes a serving improvement.

Compatible prefix state can already be resident, arrive from another tier, or be recomputed from prefix tokens. Saved prefill is useful only after accounting for management and transfer; remaining-input processing and generation still run. This is a conceptual architecture, not a universal serving topology.

Usable capacity

Distinguish activity from throughput

Utilization needs a named resource, definition, and interval. For a graphics processing unit (GPU), the NVIDIA Management Library (NVML) reports the percentage of a sampling interval with at least one kernel—a function executing on the device—running. This measures activity, not the fraction of peak arithmetic throughput achieved. Its memory-utilization field likewise measures memory activity over time, not allocated capacity. Neither metric establishes how much useful work completed: a busy device can be executing retries or producing late results.

Throughput is completed units per time. Goodput adds a compliance rule that must be stated: a benchmark may count requests meeting configured latency objectives, while an application may require accepted tasks delivered before a deadline. Those are not interchangeable. Report quality-assessment coverage separately when only some completions have known outcomes.

Concurrency can spread shared execution costs across requests, but every active request also needs state and competes for resources. The inference capacity envelope connects request sizes, memory, batching, and queueing. Bursts temporarily create more work than the service can finish, while unusually slow requests occupy capacity longer. Spare capacity lets the service catch up. Near saturation, little remains for recovery, so greater arrival or service-time variability can increase waiting even at unchanged average demand. There is no universal utilization percentage that guarantees acceptable latency.

John Little's 1961 queueing result connects consistent long-run averages: L=λW.L=\lambda W. Here LL is the average number of items inside the chosen boundary, λ\lambda is their arrival rate, and WW is average residence time. If residence includes waiting and execution, occupancy must include both. At 10 requests per second and two seconds of average residence, average occupancy is 20 requests. This accounting relationship is not a prediction of tail latency or a remedy for an unstable overloaded queue.

Little's 2011 retrospective also treats finite observation windows. If work remains at either boundary, ordinary completed-request counts and full completion latencies cannot simply be substituted into the identity. The boundary population and time spent inside the window require explicit accounting.

The useful unit depends on the application. For streaming speech, generation must sustain playback. Once that requirement is met, faster token production may matter less than startup delay and the number of simultaneous playable streams. Optimizing Inference for Voice Models uses this distinction to explain why maximizing raw token speed can miss the economic objective.

Provision for bursts and recovery

Arrival rate describes incoming work per time. Ready capacity describes work the available service can actually process. Their mismatch accumulates a backlog. Headroom is spare capacity beyond current demand, while scaling lag is the delay before added resources become usable. Translate each task class into its model calls, input and output work, tool demand, and review demand; a single requests-per-second limit cannot represent unequal tasks.

Keep offered, admitted, completed, rejected, cancelled, and unfinished work separate. Rejection protects capacity but does not erase demand. Hosted quotas add further ceilings. Claude's rate-limit documentation, inspected September 1, 2026, separates request, input-token, and output-token limits and warns about bursts and acceleration limits. For most documented models, cache reads avoid input-token quota consumption, with exceptions; losing reuse can therefore raise both charges and quota demand. A permitted rate is not a throughput or latency guarantee.

Ready capacity must exceed new demand

Consider a simple model that treats work as continuously divisible and processes it in arrival order: first in, first out (FIFO). Two work units arrive per second, with no service until second three. Six units accumulate. Service at four units per second clears the backlog by second six; service at two preserves it. Backlog BB drains in B/(C−r)B/(C-r) because only capacity beyond new arrivals clears waiting work. This requires constant service C>rC>r and continuing arrivals rr; average benchmark throughput alone does not guarantee that service rate.

Readiness and headroom determine recovery

A lossless FIFO fluid model: arrivals stay at 2 work units/s, with no service before readiness. Work units are continuously divisible; this is not a forecast of AI request percentiles.

At readiness: 6 queued units. Drain duration after readiness: 3 s; absolute catch-up: 6 s.
FIFO delay at cumulative work unit 2: 2.5 s. Backlog at 10 s: 0 units.
Cumulative work: vertical backlog and horizontal FIFO delayArrivals rise at2units/s. Completion begins at readiness, follows available capacity while backlogged, and cannot exceed arrivals. The vertical marker at readiness is backlog. The horizontal marker follows cumulative work unit2 from arrival to completion.Cumulative work (units)Dashed blue: arrived · solid green: completed0102002.557.510Elapsed time (seconds)Purple vertical: 6 units at readiness (3 s)Orange horizontal: work unit 2 arrives at 1 s; completes at 3.5 s
Time (s)ArrivedCompletedBacklog
0000
3606
612120
1020200

With positive backlog and C > 2, drain duration is B/(C−2). Once caught up, completed work follows arrivals even though ready capacity can be higher. Default: lag 3 s and service 4 units/s create 6 queued units, drain them over 3 s, and catch up at 6 s; work unit 2 waits 2.5 s.

Arrivals remain fixed at 2 work units/s. Vertical separation is unfinished work; horizontal separation at equal cumulative work is FIFO delay. Available service must exceed arrivals to drain positive backlog. The model does not infer request percentiles or warm-capacity prices.

Good warm execution cannot compensate for arbitrary startup delay. One RunPod deployment demonstration reported about 41 seconds queued and 1.5 seconds executing, attributing part of the wait to initialization. That is one request, not a latency distribution. Test actual scaling events using the startup boundary, including recovery after capacity loss.

Warm capacity exchanges idle expense for lower startup exposure. Modal's scaling controls distinguish minimum warm containers, an active-service buffer, and idle scale-down duration. Their value depends on demand gaps and startup behavior. Deferrable work can wait for capacity; interactive work may need ready headroom before the burst arrives.

Pay for the required service

Compare purchasing arrangements only after establishing the required operating envelope: workload mix, quality, latency, availability, and recovery behavior. Total cost of ownership includes acquiring and operating the resources, not just their headline rate. Use measured effective capacity under those requirements, carrying forward necessary headroom, redundancy, startup, labor, maintenance, commitments, and billing granularity.

Elasticity changes the amount purchased as well as its price. The Berkeley cloud report illustrated demand peaking at 500 servers but averaging 300: peak provisioning buys 12,000 server-hours daily for 7,200 server-hours of demand. Its hypothetical example explains how a higher metered hourly price can still produce lower expenditure. Modern AI comparisons must additionally include readiness and workload-specific capacity rather than assuming instantaneous scaling.

Commitments change what can be avoided. Bedrock Provisioned Throughput documents hourly-billed model capacity and commitments that cannot be deleted before their term ends. Lower usage can leave the bill unchanged. Conversely, an asynchronous service can sell scheduling flexibility: Claude's batch-processing documentation, inspected August 30, 2026, describes half-standard API pricing and expiry of unfinished batches after 24 hours. This is neither faster interactive completion nor guaranteed success for every request.

Sensitivity analysis varies uncertain assumptions to determine when a decision changes. For each arrangement, recompute cost only after checking feasibility in these scenarios.
Demand scenarioFeasibility checkEconomic sensitivity
Sparse interactive trafficCold-start exposure versus deadlineMinimum warm capacity and operating labor
Stable sustained trafficMeasured service capacity with required headroomCommitment terms and removable capacity units
Bursty interactive trafficReadiness lag and backlog recoveryPaid reserve versus metered burst capacity
Deferrable background workCompletion window and failure handlingScheduling discount versus delay

There is no universal demand threshold at which self-management wins. A comparison can change when operating labor increases, a replica becomes removable, or required headroom rises. Local and On-Device AI adds device-specific constraints; here those constraints enter as feasibility and cost inputs.

Optimization experiments

Test the complete workload

Write the expected mechanism before the comparison: what work disappears, what wait shortens, or what capacity becomes usable. Specify the baseline, cost scope, minimum worthwhile improvement, acceptable quality loss, timing targets, and rejection conditions. A paired comparison evaluates baseline and candidate on the same sampled tasks. Preserve the task pairing and relevant groups when analyzing differences; matched-work evaluation develops the method. Similar average scores alone do not establish that a quality-loss limit was met.

An optimization result should retain enough context to distinguish a mechanism gain from a changed workload.
RecordWhat must remain interpretable
Workload and systemTask mix, allowed attempts, versions, resource guarantees and limits
Reuse conditionsCold versus warm state, prefix overlap, working-set size, and replay order
Load coverageConfigured arrivals, started work, generator omissions, and server rejections
ResultsAccepted outcomes, assessment coverage, cost, timing distribution, failures, cancellations, and unfinished work

Isolated tests answer narrower causal questions. Callan Fox's October 2025 cache-testing method first compares cold and cached requests, then tests external-manager overhead while data still fits GPU memory, then external retrieval, and finally mixed tiers under concurrency. That sequence separates reuse benefit from background-transfer cost. It must still be followed by the whole workload under representative load.

Load generation can hide overload. In a closed model, a user waits for its previous operation before starting another, so slower service reduces offered traffic. An open model schedules independent arrivals. Either can represent a real workload, but they answer different questions; see open-loop and closed-loop load. An arrival-rate test also needs enough generator capacity: k6 records dropped iterations when it cannot start scheduled work. Those are omissions by the generator, not automatically server rejections.

Repeatability can narrow realism. MLPerf's end-to-end RAG benchmark measures tasks per second across a multi-step workflow. Retrieval-augmented generation, or RAG, retrieves evidence for generating an answer. The benchmark's performance runs fix intermediate inputs and hop counts while still generating outputs, with accuracy evaluated separately. This controls execution variation but does not measure unconstrained live trajectories; its initial offline workload also does not establish interactive tail latency.

Finally, keep environment effects visible. Anthropic's coding-evaluation infrastructure study held model, harness, and tasks fixed while varying resources. Extra headroom could prevent infrastructure failures; larger allocations could also enable different strategies. Record both guarantees and hard limits, repeat comparisons, and retain task-class differences. A result produced under more permissive conditions is not automatically an improvement to the same system.

Compare interventions across layers

Routing selects a model or provider. A cascade calls stages sequentially, using an earlier result to decide whether to escalate. The available decision information differs: RouteLLM chooses from the query before generation, whereas FrugalGPT scores generated answers before deciding to continue. Strong-model call share is a cost proxy, not total expenditure. Model Routing and LLM Gateways owns the selection and fallback policies.

For a two-stage cascade, let c1c_1 be first-stage cost, cvc_v the cost of scoring its answer, c2c_2 second-stage cost, and ee the escalated fraction. With fixed per-stage costs: E[Ccalls]=c1+cv+ec2.\mathbb{E}[C_{\mathrm{calls}}]=c_1+c_v+e c_2. Initial-only tasks pay c1+cvc_1+c_v; escalated tasks pay all three terms. Escalation does not refund the first attempt. Variable request sizes require per-task accounting. Review, failures, and deadline outcomes remain separate parts of the whole-workload comparison.

Other interventions change different quantities. Context Engineering selects and reduces model inputs. Quantization changes numerical representation and execution requirements. Distillation trains a student using a teacher's behavior, potentially replacing later teacher calls. These are not interchangeable discounts: each changes its own computation, quality risks, and development cost. Test combinations end to end because their benefits and overheads can interact.

Optimization itself consumes resources. In AI Agents That Matter, joint prompt and configuration optimization reduced variable cost by 53% for the tested GPT-3.5 pipeline and 41% for Llama-3-70B versus default DSPy approaches, with similar measured retrieval performance. The held-out experiment used 200 cases across five runs. Retrieving the required documents is not final-answer correctness, and these historical costs omit a complete production workflow. The enduring distinction is upfront optimization expense versus recurring run cost.

Estimate payback over a plausible service life, including maintenance. In a simple declared scenario, a $2,000 implementation plus $500 of additional maintenance requires 50,000 tasks to break even; more tasks produce net savings if avoidable savings remain $0.05 per task. That calculation is useful only while quality, workload mix, savings, and demand assumptions hold. Released capacity under an unchanged commitment is not the same as those cash savings.

Adoption and maintenance

Choose and revisit the operating point

First reject candidates that violate the workload contract. Among feasible choices, one dominates another if it is no worse on every stated objective and better on at least one. The Pareto frontier contains evaluated choices with no dominator. It need not identify a single winner.

The candidate plot uses declared example values at one fixed workload. All conditions other than the labeled quality and deadline failures are assumed satisfied. E has some accepted outcomes, so its cost per accepted task is defined, but it fails the required quality threshold. The p95 target of at most 1.5 seconds concerns the population distribution, not every individual task. A and B trade cost against latency; C is worse than an available alternative. D and E are ineligible before preference enters the decision. Real estimates also have uncertainty: overlapping or unstable measurements may not establish dominance.

Feasibility precedes preference

Feasibility precedes preferenceA0.30USD and1.2seconds, B0.45 and0.8, C0.50 and1.4, D0.20 and1.8, E0.15 and1.0. A and B dominate C but not each other. D misses p95 target; E fails quality. Feasible choices are circles; infeasible choices are crosses.Completion latency p95 (s)00.511.520.000.150.300.450.60p95 target ≤ 1.5 sABC · dominatedD · timing failsE · quality failsCost per accepted task (USD/task)● feasible nondominated · ○ feasible dominated · × infeasible
Among these evaluated feasible choices, A and B each dominate C but neither dominates the other. D misses the population p95 target and E fails quality. Discrete points do not establish an unmeasured frontier between configurations.

Use a canary—a bounded, time-limited live deployment compared with a control—to test remaining operational uncertainty before wider adoption. Define exposure, acceptance and stop conditions, and rollback in advance. A small canary can miss rare failures or absent task classes. Choosing a live experiment distinguishes this from shadow execution, which does not apply the candidate's results to the real workflow.

A reusable decision procedure follows from the chapter's measurements.

  • Confirm the contract — Identify the workload, accepted outcome, timing boundaries, review limit, cost scope, and unresolved measurements.
  • Identify the mechanism — Name the computation, delay, retained state, or purchased capacity that the intervention changes.
  • Reject infeasible changes — Keep quality, authorization, deadlines, capacity, and review constraints ahead of cost preference.
  • Compare the remaining choices — Use whole-workload results, uncertainty, operating effort, and payback sensitivity rather than nominal prices.
  • Validate bounded exposure — Check live outcomes against the control and stop when prespecified limits are breached.
  • Record revisit triggers — Reassess when task mix, demand, model behavior, prices, commitments, cache reuse, or review burden changes.

Unit cost and total spending answer different questions. In a simple example, halving cost per accepted task while tripling accepted volume raises total cost by 50%. That can be successful efficiency with growing demand, not a regression. The durable achievement is an understood operating point: what it delivers, what it costs, which conditions make it feasible, and when those conditions must be checked again.

Open questions

  1. Adaptive stopping must estimate the value of another attempt while task difficulty and failure causes change. A promising stopping policy can save computation yet abandon cases that remain recoverable. Progress would mean held-out, shifted-workload evidence connecting incremental attempts to accepted outcomes, review burden, and deadline compliance—not confidence statements alone.

  2. Cache placement must balance reuse against congestion. The worker with the most compatible state may also have the longest queue, while moving that state has its own cost. Progress would mean policies evaluated under changing task mixes that improve completion latency and cost together without relying on privileged knowledge of future requests.

  3. Forecasting capacity for autonomous workflows remains difficult because the workflow creates part of its own demand: retries, delegation, and growing histories change future calls. Useful progress would combine task-level trace structure with validated forecasts of bursts and recovery, retaining uncertainty instead of treating an average session as a fixed request multiplier.

  4. Whole-workflow economics depends on delayed and selectively observed outcomes. Cheap automatic completions may receive less scrutiny than expensive escalations, making their apparent efficiency difficult to compare. Progress would include comparable follow-up and assessment coverage, with unresolved outcomes visible and conclusions tested against plausible missing-outcome assumptions.

Follow the curated reading path through the speakers and demonstrations behind this entry.

20 min

AI Engineer World's Fair 2025 · 2025

Hacking the Inference Pareto Frontier

Kyle Kranen

Cited in this entry

Connects application-specific quality and responsiveness to an operating point, then shows why serving techniques interact rather than provide independent discounts.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

70 matching talks

TalkSpeakerEventYear
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Rishabh BhargavaAI Engineer Europe 20262026
Juan PeredoAI Engineer Summit 20252025
Vivek MuppallaAI Engineer World's Fair 20262026
Ben FlastAI Engineer World's Fair 20242024
Allen PikeAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Harrison ChaseAI Engineer Summit 20232023
Adrian BertagnoliAI Engineer Europe 20262026
Nishant GuptaAI Engineer World's Fair 20262026
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
Rhythm Garg, Linden LiAI Engineer Code 20252025
Francesco Bonacci, Dillon DuPont, Robert WendtAI Engineer World's Fair 20262026
Keegan McCallumAI Engineer World's Fair 20252025
Filip MakraduliAI Engineer World's Fair 20252025
Audry HsuAI Engineer Europe 20262026
Ankur Goyal, Olmo MaldonadoAI Engineer World's Fair 20242024
Anna Marie BenzonAI Engineer World's Fair 20262026
Patricija ŽemaitytėAI Engineer World's Fair 20262026
Samuel ColvinAI Engineer Europe 20262026
Philipp KrennAI Engineer World's Fair 20252025
How to Kill the Code Review

Transcript reviewed

Ankit JainAI Engineer World's Fair 20262026
Sarah ChiengAI Engineer Europe 20262026
Vinoth GovindarajanAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
AI Engineer Summit 20252025
Benjamin CowenAI Engineer Europe 20262026
Bertrand CharpentierAI Engineer Europe 20262026
Alex GavrilescuAI Engineer Code 20252025
Gergely Orosz, Simon EskildsenAI Engineer World's Fair 20262026
Maximillian PirasEvent unknownUnknown
Jay Sahnan, Arik BirdEvent unknownUnknown
Ford PriorEvent unknownUnknown
Noyan TokgozogluEvent unknownUnknown
Nicholas ArcolanoEvent unknownUnknown
Marina PetzelEvent unknownUnknown
Soukaïna AbelhadAI Engineer NYC 2026 Online Talks2026
Xander SongEvent unknownUnknown
Katelyn Lesse, Angela JiangEvent unknownUnknown
Freddy LecueEvent unknownUnknown
Ryan AhernEvent unknownUnknown
Fuad AliEvent unknownUnknown
Shy RuparelAI Engineer New York 20262026
Tisha Chawla, Susheem KoulAI Engineer World's Fair 20262026
Pamela FoxAI Engineer World's Fair 20262026
Bohan LiEvent unknownUnknown
Mingsheng HongAI Engineer World's Fair 20262026
Yuval BelferAI Engineer World's Fair 20262026
Morgan SuoEvent unknownUnknown
Eugene CheahEvent unknownUnknown
Chris VelaEvent unknownUnknown
Louis-François BouchardEvent unknownUnknown
David AbitbolEvent unknownUnknown
Mikyo KingEvent unknownUnknown
Kevin HouEvent unknownUnknown
Oliver KennonEvent unknownUnknown
Aidan SochowskiEvent unknownUnknown
CJ Avilla, Harrison StallEvent unknownUnknown
Tariq ShaukatEvent unknownUnknown
Nikhil Benesch, Aaron LinskyEvent unknownUnknown
Archana Kamath, Tyler GillamAI Engineer World's Fair 20262026
Jesse HallEvent unknownUnknown
Jediah KatzEvent unknownUnknown
Evan ConradEvent unknownUnknown
Dor SassonAI Engineer World's Fair 20262026
Fuad AliEvent unknownUnknown
Daksh GuptaEvent unknownUnknown
Kanish ManujaAI Engineer World's Fair 20262026
Affaan Mustafa, Alejandro Soumah FernandezEvent unknownUnknown
Carmen LiEvent unknownUnknown

References

Coverage and source review
Processed transcripts
35 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
40 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. NIST/SEMATECH e-Handbook: Percentiles

    https://www.itl.nist.gov/div898/handbook/prc/section2/prc262.htm

  2. Le Boudec and Thiran: Network Calculus

    https://infoscience.epfl.ch/record/282/files/netCalBook.pdf?version=1

  3. Boyd and Vandenberghe: Convex Optimization Slides

    https://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf

  4. Backlog.md: Terminal Kanban Board for Managing Tasks with AI Agents — Alex Gavrilescu, Funstage

    https://www.youtube.com/watch?v=zMXKhhwiCIc&t=237

  5. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=181

  6. Google SRE: Service Level Objectives

    https://sre.google/sre-book/service-level-objectives/

  7. Google SRE Workbook: Implementing SLOs

    https://sre.google/workbook/implementing-slos/

  8. Kelley and Walker: Critical-Path Planning and Scheduling

    https://mosaicprojects.com.au/PDF/PM-History_Critical_Path_Planning_&_Scheduling_Kelley_and_Walker_1959.pdf

  9. Amdahl: Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities

    https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf

  10. Peter J. Denning: The Working Set Model for Program Behavior

    https://denninginstitute.com/pjd/PUBS/WSModel_1968.pdf

  11. Armbrust et al.: Above the Clouds—A Berkeley View of Cloud Computing

    https://www.cs.princeton.edu/courses/archive/spring13/cos448/web/docs/berkeley_cloud.pdf

  12. Kapoor et al.: AI Agents That Matter

    https://arxiv.org/html/2407.01502v1

  13. FinOps Foundation: How to Build a Generative AI Cost and Usage Tracker

    https://www.finops.org/wg/how-to-build-a-generative-ai-cost-and-usage-tracker/

  14. How fast are LLM inference engines anyway?

    https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=663

  15. Building Deterministic Infrastructure for Non-Deterministic AI Agents

    https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=153

  16. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems

    https://arxiv.org/html/2401.17644v3

  17. Inference Server Cache Performance Testing Suite

    https://github.com/callanjfox/kv-cache-tester

  18. Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

    https://www.youtube.com/watch?v=LrGCT7G_rU8&t=94

  19. FinOps Foundation: Unit Economics

    https://www.finops.org/framework/capabilities/unit-economics/

  20. Claude Platform: Usage and Cost API

    https://platform.claude.com/docs/en/manage-claude/usage-cost-api

  21. National Academies: informative censoring and sensitivity analysis

    https://www.ncbi.nlm.nih.gov/books/NBK209895/

  22. National Academies: inference with missing outcomes

    https://www.ncbi.nlm.nih.gov/books/NBK209900/

  23. ACCA: Relevant costs

    https://www.accaglobal.com/gb/en/student/exam-support-resources/fundamentals-exams-study-resources/f5/technical-articles/relevant-costs.html

  24. FinOps terminology: ownership, depreciation and utilization

    https://framework.finops.org/assets/terminology/

  25. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=3833

  26. Amazon Bedrock: Increase model invocation capacity with Provisioned Throughput

    https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html

  27. How Web Data Infrastructure Powers the Next Generation of AI

    https://www.youtube.com/watch?v=1UmZHb_E_SM&t=879

  28. FinOps Foundation: Usage Optimization

    https://www.finops.org/framework/capabilities/usage-optimization/

  29. AIPerf Metrics Reference

    https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference

  30. Metrics design — vLLM

    https://docs.vllm.ai/en/stable/design/metrics/

  31. Project Management for Construction: Fundamental Scheduling Procedures

    https://www.cmu.edu/cee/projects/PMbook/10_Fundamental_Scheduling_Procedures.html

  32. Muppalla, Perkari and Jhaveri: Building AI for Healthcare Conversations, Part 2

    https://hippocraticai.com/building-ai-for-healthcare-conversations-part-2/

  33. Google SRE: Addressing Cascading Failures

    https://sre.google/sre-book/addressing-cascading-failures/

  34. CUDA C++ Best Practices Guide

    https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

  35. Dean and Barroso: The Tail at Scale

    https://cs.brown.edu/courses/csci2950-u/s18/papers/p74-dean.pdf

  36. Lessons from building GenAI based applications — Juan Peredo

    https://www.youtube.com/watch?v=YYcNm2RexnY&t=567

  37. Python functools: lru_cache

    https://docs.python.org/3/library/functools.html

  38. Go singleflight package documentation

    https://pkg.go.dev/golang.org/x/sync/singleflight

  39. RFC 9111: HTTP Caching

    https://www.rfc-editor.org/rfc/rfc9111.html

  40. gRPC: Deadlines

    https://grpc.io/docs/guides/deadlines/

  41. Fink et al.: How green are large language models for radiology report labelling?

    https://link.springer.com/article/10.1186/s13244-026-02289-2

  42. MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Services

    https://arxiv.org/html/2403.02694v3

  43. Automatic prefix caching — vLLM

    https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/

  44. Automatic Prefix Caching — vLLM v0.14.1

    https://docs.vllm.ai/en/v0.14.1/design/prefix_caching/

  45. OWASP Multi-Tenant Security: tenant context and resource access

    https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html

  46. From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik

    https://www.youtube.com/watch?v=2czYyrTzILg&t=205

  47. Claude prompt caching: matching and eligibility

    https://platform.claude.com/docs/en/build-with-claude/prompt-caching

  48. Scaling Memcache at Facebook

    https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf

  49. Callan Fox: Evaluating management of KV Cache within an inference system

    https://medium.com/@callan.j.fox/evaluating-management-of-kv-cache-within-an-inference-system-2d7c3d266c3a

  50. Claude Platform pricing: Prompt caching

    https://platform.claude.com/docs/en/about-claude/pricing

  51. Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

    https://www.youtube.com/watch?v=NTBX-wxUhHs&t=698

  52. Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

    https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1287

  53. NVIDIA NVML: nvmlUtilization_t

    https://docs.nvidia.com/deploy/nvml-api/structnvmlUtilization__t.html

  54. Continuous batching — Transformers

    https://huggingface.co/docs/transformers/main/en/continuous_batching

  55. MIT OpenCourseWare: Review of Queueing Models

    https://ocw.mit.edu/courses/15-763j-manufacturing-system-and-supply-chain-design-spring-2005/568c857501f6785e7b328a3cf3990705_reviw_of_que_mol.pdf

  56. Notes on Little's Law — Karl Sigman

    https://www.columbia.edu/~ks20/4106-18-Fall/Notes-LL.pdf

  57. John D. C. Little: Little's Law as Viewed on Its 50th Anniversary

    https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/OS/Littles-Law-50-Years-Later.pdf

  58. Optimizing inference for voice models in production

    https://www.youtube.com/watch?v=gmTHs5T_YAE&t=211

  59. Google SRE: Handling Overload

    https://sre.google/sre-book/handling-overload/

  60. Modal: Scaling Out

    https://modal.com/docs/guide/scale

  61. Claude Platform: Rate limits

    https://platform.claude.com/docs/en/api/rate-limits

  62. Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

    https://www.youtube.com/watch?v=ILdE7FaAjVA&t=701

  63. Claude Platform: Batch processing

    https://platform.claude.com/docs/en/build-with-claude/batch-processing

  64. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=4320

  65. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=753

  66. MLCommons: MLPerf Inference Rules

    https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc

  67. Quantifying infrastructure noise in agentic coding evals

    https://www.anthropic.com/engineering/infrastructure-noise

  68. Grafana k6: Arrival-rate VU allocation

    https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/arrival-rate-vu-allocation/

  69. Grafana k6: Open and closed models

    https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/

  70. MLCommons: Introducing the MLPerf End-to-End RAG Inference Benchmark

    https://mlcommons.org/2026/08/endtoend-inference/

  71. RouteLLM: Learning to Route LLMs with Preference Data

    https://arxiv.org/html/2406.18665v2

  72. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

    https://arxiv.org/pdf/2305.05176

  73. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=275

  74. Google SRE Workbook: Canarying Releases

    https://sre.google/workbook/canarying-releases/

  75. RAG and the MongoDB Document Model

    https://www.youtube.com/watch?v=2Ey275TX4ZU&t=551

  76. AI Engineering 201: Inference

    https://www.youtube.com/watch?v=N7lJY5IKVLE&t=5143

  77. Selective Question Answering under Domain Shift

    https://arxiv.org/pdf/2006.09462

  78. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=668

  79. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=1136

  80. How fast are LLM inference engines anyway?

    https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=511

  81. The State of Model Routing — NVIDIA, Cognition, OpenRouter

    https://www.youtube.com/watch?v=QHBjufYK8TA&t=273

  82. The State of Model Routing — NVIDIA, Cognition, OpenRouter

    https://www.youtube.com/watch?v=QHBjufYK8TA&t=690