Contents
  1. Operational evidence
    1. Questions observability must answer
      1. Question-to-evidence contract
    2. From isolated records to executions
      1. Selected turning points
  2. Representing one execution
    1. Traces, spans, events, and links
    2. What each signal preserves
      1. Metric forms
  3. Capturing interpretable evidence
    1. Record meaning-changing boundaries
      1. Model invocation
      2. Retrieval, parsing, and tools
      3. Effects
      4. Boundary record
    2. Identify the system that ran
    3. Propagate context and observe telemetry
      1. Telemetry integrity
  4. Reconstruction and diagnosis
    1. Build a defensible execution account
      1. Evidence status
    2. Find the consequential divergence
      1. Investigation sequence
  5. Time, cost, and outcomes
    1. Attribute elapsed time
    2. Build an attributable cost ledger
      1. Accounting boundaries
    3. Connect operations to outcomes
  6. Evidence under constraints
    1. Sample for the question
      1. Sampling choices
    2. Govern telemetry as sensitive data
    3. Keep observability from failing the service
      1. Instrumentation decision record
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Observability & reliability

An AI system can return HTTP 200, emit no exception, and still give a useless answer or perform the wrong action. When that happens, engineers need more than the final response and a dashboard average. They need to identify the affected task attempt, reconstruct the path that actually ran, recover the inputs and versions that influenced it, distinguish proposed actions from confirmed effects, and determine where time and money went.

Observability is the ability to investigate system behavior through operational evidence. Telemetry is the recorded data—events, logs, metrics, traces, usage records, and effect confirmations—from which that investigation proceeds. The central design question is not “What can we log?” but “What will we need to establish later?” A complete trace can establish what instrumented operations occurred; it cannot by itself establish that the result was correct, useful, authorized, or safe. Those outcome claims require an intended-purpose evaluation or an authoritative check of external state.

Part I — Operational evidence

Questions observability must answer

Start with a concrete operational question. For a bad result, engineers usually need to ask: Which task and attempt produced it? Which components, model calls, retrievals, tools, retries, and asynchronous jobs participated? What information and configuration influenced each decision? Which effects were requested, attempted, confirmed, or left uncertain? How much elapsed time and expense belonged to the attempt? Which claims are directly recorded, which are derived from records, and which remain unavailable?

Question-to-evidence contract

Different questions require different evidence. No single signal answers all of them.
QuestionUseful evidenceWhat it does not establish
What executed?Task and attempt IDs; spans; events; tool-call recordsUninstrumented work or why a model chose an output
What influenced it?Input references; retrieved evidence; code, model, prompt, schema, policy, and configuration identitiesUndisclosed provider state or correctness of the inputs
What effect occurred?Operation ID; target-system receipt; authoritative state read; settlement or application acknowledgmentHuman understanding or reversal of an already completed effect
Where did time and cost go?Request-level timing relationships; provider usage; paid-tool records; applicable pricesCausation from aggregate correlation or missing provider usage
Was the outcome good?Task-specific checks, human review, or authoritative downstream outcomesA conclusion from transport success or an error-free span alone

This boundary prevents a common category error. A span with status “OK” says that the instrumented operation completed according to that instrumentation. It does not say the model grounded its answer, selected the intended tool, respected a business rule, or helped the user. Conversely, a later user correction may show that an outcome was bad without identifying which component caused it. Observability supplies the execution evidence; evaluation and authoritative outcome checks supply the criteria by which that execution is judged.

From isolated records to executions

Software observability developed around a recurring distributed-systems problem: a symptom can appear far from its source. A slow page may reflect a browser resource, an upstream service, a queue, or a dependency changed several calls earlier. Local logs show what one process reported. Aggregate metrics reveal that a population changed. Neither necessarily connects the operations that produced one affected request.

Selected turning points

These dates identify documented publications or project announcements, not universal invention dates.
DateDevelopmentContribution
July 1998NetLoggerUsed a common event format to follow data through application, operating-system, and network components.
2007X-TracePropagated a task identifier across application and network layers to connect reports into a task tree.
April 2010Dapper reportDescribed production tracing based on shared-library instrumentation and adaptive sampling at Google.
May 2019OpenTelemetry announcedMerged OpenTracing and OpenCensus work to reduce incompatible instrumentation and duplicated integrations.
November 2021W3C Trace ContextStandardized portable HTTP trace-parent propagation across tracing implementations.
2023OpenCensus transitionOpenTelemetry reported feature parity in several languages and scheduled most OpenCensus repositories for archival.

These mechanisms accumulated rather than replacing one another. Metrics remain efficient for detecting population changes; events retain local detail; traces connect operations. AI applications extend the execution record with model identity, prompt and agent versions, retrieved context, token usage, tool calls, and generated content when policy permits. LinkedIn, for example, described rich pre-production inspection and leaner production OpenTelemetry spans for model calls, tool invocations, and memory use—an illustration of capture detail changing with operating constraints, not a universal architecture.

Part II — Representing one execution

Traces, spans, events, and links

A trace correlates operations belonging to one logical execution. A span records one operation with an identity, start and end times, and optional attributes, events, status, and links. A span event marks an instant—such as a retry decision—inside or alongside duration-bearing work. A trace ID identifies the trace; a span ID identifies one operation; a parent span ID expresses a parent-child relationship. These identities describe telemetry structure, not business authority.

Parent-child relationships work when one operation starts or contains another causal continuation. Span links are better when strict nesting would misrepresent the relationship: a batch consumer may process messages from several producers, or a delayed job may begin in a new trace after the originating request ends. Messaging conventions therefore use links to associate consumer work with message-creation contexts. A separate application correlation ID can join several traces or requests into one order, conversation, or workflow, but it does not turn them into one trace.

This structure is a dependency graph, not necessarily a call stack or total timeline. Concurrent retrieval and model work can overlap. A queue can delay consumption. A retry creates another attempt. Different hosts can disagree about wall-clock timestamps, so communication relationships—send before receive, response before client receipt—can constrain order more reliably than sorting all timestamps together. Temporal overlap alone proves neither parentage nor causation.

One task across synchronous and asynchronous work

Example

A logical task can contain nested calls, linked asynchronous work, and an opaque boundary; one parent tree is not always an honest representation.

Application requestLogical task attempt entering the application.
RetrievalReturns evidence identifiers and selected content references.
Model boundaryThe application records the request and response, but provider internals are opaque.
Tool proposalA model output becomes a validated action request.
Queued messageCarries message identity and propagated creation context.
Delayed consumerStarts asynchronous work linked to message creation.
Authoritative effectTarget-system evidence confirms the defined effect boundary.
A request invokes retrieval and model work. The model proposes a tool call whose message creates asynchronously linked consumer work. The consumer receives authoritative effect confirmation, while the external model provider remains observable only at its boundary. Edge labels distinguish invocation, data, messaging linkage, and effect evidence; layout spacing is not a time scale.
Read the diagram as text
  • Application request. Logical task attempt entering the application.
  • Retrieval. Returns evidence identifiers and selected content references.
  • Model boundary. The application records the request and response, but provider internals are opaque.
  • Tool proposal. A model output becomes a validated action request.
  • Queued message. Carries message identity and propagated creation context.
  • Delayed consumer. Starts asynchronous work linked to message creation.
  • Authoritative effect. Target-system evidence confirms the defined effect boundary.
  • Application request → Retrieval: invokes.
  • Retrieval → Model boundary: evidence data.
  • Application request → Model boundary: invokes.
  • Model boundary → Tool proposal: proposes.
  • Tool proposal → Queued message: creates message.
  • Queued message → Delayed consumer: async trace link.
  • Delayed consumer → Authoritative effect: effect confirmation.

What each signal preserves

An event or structured log preserves an individual occurrence: a policy rejection, a provider response, a queue delivery, or an effect receipt. A metric aggregates numerical observations into streams over time. A trace preserves relationships among operations in a particular execution. Shared trace and span identifiers can correlate logs with execution context, while resource fields identify the emitting service or infrastructure.

Metric forms

  • Counter — Accumulates occurrences, such as attempts or failures. Interpret its rate over a stated interval and account for resets.
  • Gauge — Records a current absolute, non-additive value, such as queued work or occupied capacity. Use an asynchronous observable gauge when the value is obtained through an accessor rather than delivered as a change event.
  • Histogram — Places observations into compatible buckets while retaining count and sum, allowing aggregation before estimating fleet percentiles.
  • Exemplar — Associates a selected observation with trace context so an aggregate point can lead to an execution. It is selected evidence, not a representative sample by default.

Every distinct metric label set creates another time series. Request IDs, user IDs, arbitrary prompt text, and tool arguments can therefore make series counts grow with traffic. Keep metric dimensions bounded—operation, model, provider, deployment, disposition, or a controlled task class—and place request-level identity in protected logs or spans. High-cardinality trace data still has storage and privacy costs; the point is to use the representation suited to the question, not to move unlimited data elsewhere.

Each signal preserves different information.
SignalPreservesTypical questionMain loss or risk
Event or logOne occurrence and local fieldsWhat did this component report?Relationships require explicit correlation; payloads may be sensitive.
MetricAggregate values over a population and intervalDid rate, latency, usage, or saturation change?Individual execution detail is aggregated away.
TraceRelationships and timing within selected executionsWhich path and dependencies produced this attempt?Sampling, missing spans, and instrumentation gaps limit completeness.

Part III — Capturing interpretable evidence

Record meaning-changing boundaries

Model invocation

Capture evidence where information changes meaning. Stored conversation messages are not identical to the final model input: a chat template serializes roles and control tokens, retrieval may add passages, a policy layer may remove content, and a gateway may route to another model. Around a model boundary, record the operation and attempt identities, requested and returned model when available, prompt or template identity, ordered input references, relevant generation settings, response disposition, finish or cancellation reason, and provider-reported usage. If content is omitted, redacted, truncated, or stored externally, record that status explicitly.

Retrieval, parsing, and tools

At retrieval and transformation boundaries, preserve the query or query reference, collection and index identity, filters, returned evidence identifiers, ordering, and the parser or transformation result. At a tool boundary, distinguish the model's proposed call from schema validation, authorization, execution, and returned result. Tool name and call ID connect proposal to execution; arguments and results are sensitive opt-in content under current OpenTelemetry conventions. A tool span that finishes successfully only establishes its instrumented boundary.

Effects

External effects need their own evidence. A publisher confirmation can show that a broker accepted a message while saying nothing about whether a consumer processed it. A server response finishing can mean bytes were handed to the operating system, not that a client received or understood them. Record the requested action, stable operation ID, validation and authorization result, attempt, target-system receipt, and—when the claim requires it—an authoritative read or application acknowledgment showing the effect boundary actually reached. If the outcome is unknown, preserve it as unknown; a timeout does not prove that the effect failed.

Boundary record

The smallest useful record differs by boundary.
BoundaryBeforeAfterDisposition to preserve
ModelAttempt, requested model, input references, settingsReturned model, output reference, usageCompleted, truncated, cancelled, errored, usage unavailable
ToolCall ID, schema version, proposed argumentsValidated arguments, result referenceRejected, attempted, completed, timed out, result unavailable
External effectOperation ID, target, authorized intentTarget receipt or authoritative stateConfirmed, failed without effect, or unknown

Identify the system that ran

A trace becomes interpretable only when it identifies the system that produced it. Record resolvable immutable identities for the deployed code or artifact, agent or workflow, prompt or template, tool and response schemas, retrieval corpus or index, policy bundle, feature-flag assignment, and relevant tenant configuration. For models, separate provider, requested model, returned model, immutable snapshot or fine-tune where available, and routing layer. A model family or moving alias does not identify a deployment.

OpenTelemetry schema URLs provide one useful pattern: a versioned, immutable schema identity travels with telemetry groups so a schema-aware consumer can translate supported attribute changes. This helps distinguish a renamed field from changed system behavior. It does not identify code, prompts, models, policies, or business-data schemas; those remain application responsibilities. Likewise, an identifier does not prove that its artifact remains retrievable or that emitted values conformed to it.

The release identity chapter explains why source commits, packaged code, configuration, schemas, dependencies, and model selection can all be release artifacts. Evaluation manifests preserve the system and assessment used in an evaluation. A production trace should link to compatible identities, but it is not itself an evaluation manifest. Provider internals that cannot be observed should remain explicitly unavailable instead of being reconstructed from a product label.

Propagate context and observe telemetry

Context propagation carries trace and parent-span identity across a process boundary. With W3C Trace Context, a sender injects a traceparent header and a receiver extracts it so downstream spans join the same trace. Custom protocols and message envelopes need equivalent injection and extraction. Without propagation, two fully instrumented services can emit valid but disconnected traces. In an MCP demonstration, client context was carried through protocol metadata and restored by a controlled server so the backend could reconstruct the parent relationship; a third-party server outside the administrative domain remained opaque in that setup.

OpenTelemetry baggage carries application-defined properties with a distributed request. It is separate from trace identity and becomes a log, span, or metric attribute only when instrumentation copies it. Because baggage can cross process boundaries, keep it bounded, clear it before untrusted boundaries when appropriate, and never treat it as authorization. A tenant or workflow label carried in baggage is correlation context, not proof that the caller may access that tenant or workflow.

Telemetry integrity

The evidence pipeline is another distributed system. Application SDK queues can fill and drop spans. Collectors can retry and buffer exports, yet still lose records after capacity or retry limits are exhausted. Tail samplers can evict buffered traces, receive late spans after a decision, or make inconsistent decisions when related spans reach different collector instances. OTLP retries can duplicate telemetry when an acknowledgment is lost. Record queue occupancy, export failures, dropped and truncated records, sampling policy, collector and instrumentation versions, and schema identity. A quiet dashboard without these health signals is ambiguous.

Part IV — Reconstruction and diagnosis

Build a defensible execution account

Reconstruction begins with identity: the business task, attempt, trace or traces, provider requests, queue deliveries, tool calls, checkpoints, and external operation IDs. Join records through explicit parentage, links, call IDs, message identities, and application correlation fields. Use timestamps as observations from particular clocks, not as a universal total order. Communication and dependency relationships can constrain ordering when host clocks disagree.

Evidence status

Use evidence status to prevent plausible narration from filling a gap.
StatusMeaningExample
ObservedA retained record directly states the fact within its instrumentation boundary.Tool attempt call-7 carried quantity 1000.
DerivedA rule combines observed records without adding an unobserved event.The consumer work followed message creation because the recorded link connects them.
HypothesisA testable explanation is consistent with current records but not established.The parser change caused the wrong argument.
UnavailableRequired evidence was not captured, was lost, or lies outside the observation boundary.The third-party service's internal steps are unknown.

For agents, a trajectory is the ordered observations, selected actions, and results from one task attempt. A trajectory can be represented through spans and events, but generated reasoning text should not be treated as a faithful causal account of model computation. Experiments have shown models rationalizing answers while omitting input features that influenced them. Recorded inputs, actions, and outcomes show what was observed; controlled interventions test a causal explanation.

A runtime checkpoint is another evidence source, not a magical replay of the world. It can identify saved graph state, pending tasks, or a machine snapshot. Historical replay may re-execute later model calls and APIs, producing different results or repeated external effects. The runtime chapter owns persistence and recovery; for observability, the requirement is to record checkpoint identity and distinguish reading saved evidence from executing work again.

Find the consequential divergence

Begin with the affected outcome and its execution identity. First check whether the evidence path is complete enough for the proposed investigation: propagation, sampling, exporter health, content availability, and version context all matter. Then reconstruct the path and compare it with an explicit expectation or a successful execution under comparable conditions. Align equivalent stages rather than comparing arbitrary neighboring timestamps.

Investigation sequence

Each step narrows what the records can support before the investigation makes a causal claim.

  • Identify the affected attempt — Connect the reported outcome to its task, attempt, trace, provider request, tool call, and external operation identities.
  • Check evidence coverage — Determine whether propagation, sampling, export, content capture, and version records are complete enough to answer the question. Mark missing evidence instead of treating silence as normal behavior.
  • Reconstruct dependencies — Order recorded operations using parent relationships, links, messages, and effect confirmations rather than timestamps alone.
  • Align a comparable execution — Match equivalent stages from a successful execution or an explicit expected path so differences in workload or structure do not masquerade as faults.
  • Locate the first supported material difference — Find the earliest recorded divergence capable of explaining the downstream outcome, while keeping correlation distinct from causation.
  • Test competing explanations — Choose an observation or controlled change that could reject each remaining hypothesis.

Inspect boundaries separately: input assembly, retrieval, provider response, parsing, policy, tool selection, state, external effect, and infrastructure. Suppose a tool received an excessive quantity and executed it. The final tool call establishes where the effect was attempted, but adjacent records may show that an upstream model constructed the bad argument. In larger systems, a subtle earlier error can propagate through many prompts and tools, so the final failing interaction need not identify the component to change.

Align executions to find the first material difference

Example

The final bad effect can be downstream of an earlier recorded divergence; locating that difference narrows the investigation but does not by itself prove causation.

Comparable inputA successful attempt begins with the matched task conditions.
Retrieved evidenceThe comparable attempt records the selected evidence identifiers.
Proposed quantity: 10The model boundary directly records the proposed tool argument.
Validation passedThe recorded schema and policy checks accepted quantity 10.
Tool attempted 10The tool boundary records the validated quantity it received.
Expected effect confirmedAuthoritative target-system evidence confirms the defined effect boundary.
Affected inputThe bad attempt begins with task conditions matched to the comparable execution.
Retrieved evidenceThe affected attempt records the same selected evidence identifiers.
Proposed quantity: 1000This is the first recorded material difference between the aligned stages.
Validation passedThe recorded checks accepted the excessive quantity; whether a missing rule caused the outcome remains a hypothesis.
Tool attempted 1000The tool executed the validated argument it received; this is downstream evidence, not automatic proof of the originating defect.
Wrong effect confirmedAuthoritative target-system evidence confirms that the unintended item was changed.
Controlled causal testReplay a captured boundary or change one relevant rule while holding the comparison conditions fixed.
Hypothesis dispositionA changed downstream outcome supports the tested explanation; an unchanged outcome rejects or weakens it.
Equivalent stages expose the first recorded material difference: the affected execution proposes quantity 1000 where the comparable execution proposes 10. Validation, tool execution, and effect confirmation preserve the downstream path. A controlled replay or intervention must still test whether changing that proposal changes the outcome.
Read the diagram as text
  • Comparable input. A successful attempt begins with the matched task conditions.
  • Retrieved evidence. The comparable attempt records the selected evidence identifiers.
  • Proposed quantity: 10. The model boundary directly records the proposed tool argument.
  • Validation passed. The recorded schema and policy checks accepted quantity 10.
  • Tool attempted 10. The tool boundary records the validated quantity it received.
  • Expected effect confirmed. Authoritative target-system evidence confirms the defined effect boundary.
  • Affected input. The bad attempt begins with task conditions matched to the comparable execution.
  • Retrieved evidence. The affected attempt records the same selected evidence identifiers.
  • Proposed quantity: 1000. This is the first recorded material difference between the aligned stages.
  • Validation passed. The recorded checks accepted the excessive quantity; whether a missing rule caused the outcome remains a hypothesis.
  • Tool attempted 1000. The tool executed the validated argument it received; this is downstream evidence, not automatic proof of the originating defect.
  • Wrong effect confirmed. Authoritative target-system evidence confirms that the unintended item was changed.
  • Controlled causal test. Replay a captured boundary or change one relevant rule while holding the comparison conditions fixed.
  • Hypothesis disposition. A changed downstream outcome supports the tested explanation; an unchanged outcome rejects or weakens it.
  • Comparable input → Retrieved evidence: assembles request.
  • Retrieved evidence → Proposed quantity: 10: informs proposal.
  • Proposed quantity: 10 → Validation passed: submits quantity 10.
  • Validation passed → Tool attempted 10: authorizes attempt.
  • Tool attempted 10 → Expected effect confirmed: receives confirmation.
  • Affected input → Retrieved evidence: assembles request.
  • Retrieved evidence → Proposed quantity: 1000: informs proposal.
  • Proposed quantity: 1000 → Validation passed: submits quantity 1000.
  • Validation passed → Tool attempted 1000: authorizes attempt.
  • Tool attempted 1000 → Wrong effect confirmed: receives confirmation.
  • Proposed quantity: 1000 → Controlled causal test: defines tested change.
  • Controlled causal test → Hypothesis disposition: produces observation.

The earliest consequential divergence is the first supported difference capable of explaining downstream behavior—not merely the first unusual value or earliest timestamp. Correlation generates hypotheses; it does not prove root cause. Use a controlled change or a fixed replay boundary to distinguish explanations. Replay can hold a captured provider response fixed while exercising downstream code. Falsifiable debugging experiments determine what observation would reject each proposed cause.

Part V — Time, cost, and outcomes

Attribute elapsed time

Define the observation boundary before naming a latency. End-to-end client latency, server operation duration, queue time, model prefill, time to first response chunk, server time to first token, generation duration, inter-token latency, network transfer, and client-visible completion are different intervals. Current OpenTelemetry generative-AI metrics distinguish client time to first response chunk from server time to first token; chunks and tokens are not one-to-one, and the observation points differ.

A critical path is the longest weighted dependency path controlling completion. Total work is the sum of work across operations. If retrieval and model preparation overlap, their durations both contribute work but their overlap is counted once in elapsed wall time. The slowest individual span is not necessarily the critical path, and subtracting overlapping child durations independently from a parent can produce nonsense. Trace relationships and interval unions are required.

Elapsed latency is not summed span time

Example timings

Overlapping span durations describe concurrent recorded work and must not be added to obtain wall-clock latency.

Request0–120 msDuration 120 ms
Retrieval5–35 msDuration 30 msWithin Request
Model operation10–80 msDuration 70 msWithin Request
Response streaming40–75 msDuration 35 msWithin Model operation
Tool attempt80–100 msDuration 20 msWithin Request
Retry attempt100–115 msDuration 15 msWithin Request
Request finalization115–120 msDuration 5 msWithin Request
The 120 ms request contains overlapping retrieval, model, streaming, tool, retry, and finalization spans. Model work overlaps retrieval, and streaming overlaps the model operation. The final five milliseconds are recorded as a request-finalization child span. This timeline shows containment and overlap only; it does not encode dependency causation or identify a critical path.
Read the diagram as text
  • Request. End-to-end server observation boundary. 0 to 120 ms; duration 120 ms.
  • Retrieval. Evidence lookup overlaps the model operation. 5 to 35 ms; duration 30 ms. Parent: Request.
  • Model operation. Includes processing before and after response onset. 10 to 80 ms; duration 70 ms. Parent: Request.
  • Response streaming. First response chunk appears at its start. 40 to 75 ms; duration 35 ms. Parent: Model operation.
  • Tool attempt. A recorded tool interval; the timeline does not assert what caused it to begin. 80 to 100 ms; duration 20 ms. Parent: Request.
  • Retry attempt. A separate recorded attempt; temporal adjacency alone does not prove dependency. 100 to 115 ms; duration 15 ms. Parent: Request.
  • Request finalization. A recorded five-millisecond finalization span inside the parent request. 115 to 120 ms; duration 5 ms. Parent: Request.

Distributions matter. Engaged users issue repeated requests and accumulate exposure to slow tails; voice conversations can be disrupted by a single long pause. Histograms can be aggregated before estimating fleet percentiles, but averaging instance percentiles does not produce a fleet percentile. Component p95 values also cannot be added to obtain end-to-end p95 because the slow observations need not occur on the same requests. Slice latency by operation, dependency, version, workload shape, and outcome disposition before attributing a regression.

Build an attributable cost ledger

Cost attribution starts with execution identity. For each directly metered item, preserve quantity, unit, currency, applicable price reference, request and attempt, dependency, customer or project attribution, and whether the value is provider-reported, independently measured, or estimated. Model records may include input, output, and cached tokens when the provider supplies them; other modalities and reasoning usage require their own supported units. Token counts are not monetary cost until a price rule is applied.

Count attempts, not only successful logical calls. A retry may produce multiple provider calls for one workflow step. Failed attempts, paid tools, discarded branches, and human review can incur cost even when the workflow fails. Avoid double counting when both an application instrumenter and a gateway report the same model request, or when a parent record already contains its children's cumulative expense. Interrupted streams may never deliver a final provider-usage record; mark that quantity unavailable rather than zero and keep separate estimates visibly labeled.

Accounting boundaries

These totals answer different accounting questions.
ViewIncludesInterpretation
Attempt costObserved calls and tools for one attemptExpense incurred by this execution, including failed work when recorded.
Workflow costDeduplicated attempt and dependency costs across one workflowDirect cost of the workflow under the stated inclusion rules.
Provider cost exportProvider-accounted expense for its service period and groupingReconciliation boundary; it may not join to every application request.
Allocated shared costA stated rule distributing platform, network, storage, or support expenseOrganizational allocation, not proof of physical consumption by one request.
Cost per useful outcomeCost divided by independently established useful outcomesRequires outcome assessment and explicit unresolved handling.

Keep cost per attempted request, completed workflow, and useful outcome separate. A growing token total may reflect more use, longer contexts, retries, or waste. It does not establish delivered value. Deeper optimization and workload economics belong in AI Cost and Performance Engineering.

Connect operations to outcomes

Monitoring repeatedly measures a defined operational question over an eligible population and period. A service-level indicator specifies what counts as a good event and which events are eligible; a service-level objective gives that indicator a target and period. For AI systems, transport availability can be one indicator, while task completion, tool success, escalation, correction, review duration, latency, cost, safety events, and recovery describe different properties. Do not collapse them into one unexplained health score.

Stable task and attempt identities let later evidence join the execution: a reviewer override, customer correction, escalation, abandonment, dispute, or authoritative downstream result. Define the numerator, denominator, aggregation unit, follow-up window, unresolved disposition, slice, and sampling policy. In a human-reviewed workflow, rising override frequency or review duration can be useful investigation signals, but they are indirect: a long review may reflect confusing output, difficult cases, or a changed reviewer population.

Production outcomes are often delayed or selectively observed. Recent requests have had less time to receive a result. Only escalated cases may receive expert review. Users may report visible failures and remain silent about others. The observed success rate therefore estimates success among observed outcomes, not automatically among all eligible cases. Report successes, failures, and unresolved cases separately; absence of a recorded failure is not success. Incomplete production feedback explains the evaluation consequences.

Aggregate changes also require slicing. A deployment can look worse because it received a harder task mix even when within-slice behavior stayed constant; the reverse can hide regressions. Compare relevant workload, version, provider, model, risk, and disposition slices while preserving denominators and uncertainty. When a causal product decision is needed, use an appropriate live experiment; telemetry supplies exposure and outcome records, but randomization and analysis supply the intervention claim.

Part VI — Evidence under constraints

Sample for the question

Full-fidelity tracing is often impractical, so sampling selects executions for recording or retention. Head sampling decides before the complete trace is known and is efficient, but cannot preferentially retain failures discovered later. Tail sampling buffers most or all of a trace before deciding and can retain executions with errors, high latency, or selected attributes, but it adds state, routing, and operational complexity. Nothing downstream can recover a trace discarded by an earlier sampler.

Sampling changes the supported claim

Example

Uniform and outcome-enriched policies retain different evidence from the same workload, so their retained percentages answer different questions.

1 / 4 · Underlying workload

All execution types exist before telemetry retention is applied.

New this stepRoutine successA common technically successful execution.
New this stepCommon failureAn execution with a recorded error.
New this stepSlow successA successful execution in the latency tail.
New this stepRare severe failureA low-frequency, high-consequence execution.
New this stepIncomplete traceLate or dropped spans leave uncertain disposition.
The workload contains routine successes, a common failure, a slow success, a rare severe failure, and an incomplete trace. Uniform retention can support population estimation under its design but may miss rare cases; error-enriched retention supports diagnosis but not an unadjusted failure rate; latency-tail retention selects slow executions. The underlying workload does not change.
Read the diagram as text
  • Routine success. A common technically successful execution.
  • Common failure. An execution with a recorded error.
  • Slow success. A successful execution in the latency tail.
  • Rare severe failure. A low-frequency, high-consequence execution.
  • Incomplete trace. Late or dropped spans leave uncertain disposition.
  • Uniform retained set. Known-probability selection intended for population estimation.
  • Failure-enriched set. Retains diagnostic failures but overrepresents them.
  • Latency-tail set. Retains slow traces and requires complete routing and buffering.
  • Routine success → Uniform retained set: may be selected.
  • Common failure → Uniform retained set: may be selected.
  • Slow success → Uniform retained set: may be selected.
  • Common failure → Failure-enriched set: selected by error.
  • Rare severe failure → Failure-enriched set: selected by error.
  • Slow success → Latency-tail set: selected by latency.
  • Incomplete trace → Latency-tail set: may evade decision.
  1. Underlying workload. All execution types exist before telemetry retention is applied. Active: Routine success, Common failure, Slow success, Rare severe failure, Incomplete trace. New: Routine success, Common failure, Slow success, Rare severe failure, Incomplete trace.
  2. Uniform selection. A known-probability sample can estimate population properties, subject to sample size and coverage. Active: Routine success, Common failure, Slow success, Rare severe failure, Incomplete trace, Uniform retained set. New: Uniform retained set.
  3. Failure enrichment. Error-based retention improves diagnostic availability while changing the represented population. Active: Routine success, Common failure, Slow success, Rare severe failure, Incomplete trace, Uniform retained set, Failure-enriched set. New: Failure-enriched set.
  4. Tail selection. Tail sampling can retain slow traces but remains vulnerable to incomplete routing, late spans, and buffer limits. Active: Routine success, Common failure, Slow success, Rare severe failure, Incomplete trace, Uniform retained set, Failure-enriched set, Latency-tail set. New: Latency-tail set.

Sampling choices

Selection changes the population represented by retained traces.
PolicyUseful forClaim it cannot support alone
Uniform probabilisticEstimating population properties when inclusion behavior is known and the sample is large enoughReliable capture of every rare severe failure.
Parent-based distributed decisionKeeping child spans aligned with the upstream sampled flag when trace context is propagated and samplers are configured compatiblyRecording or export completeness; propagation can fail and a sampled flag is not a delivery guarantee.
Error- or latency-enriched tailDiagnosing known failures and slow executionsPopulation error or latency frequency without accounting for deliberate selection.
Attribute- or status-selected tailRetaining executions matching a defined operational conditionThe prevalence of that condition in all traffic.
Rate-limited retentionBounding retained telemetry volumeA representative population sample without a separate probability design.
Adaptive instrumentationAllocating capture to changing diagnostic value under a budgetReconstruction of instrumentation that was disabled at the time.
ExemplarNavigating from an aggregate observation to a selected traceEvidence that the selected trace is typical.

Keep separate streams when one policy cannot answer both monitoring and diagnosis. A representative sample can estimate rates; an error-enriched sample can preserve diagnostic detail; bounded escalation can capture sensitive payload evidence for selected incidents. Record the sampling policy and version with retained data. Tail sampling still does not guarantee completeness under buffer eviction, late spans, overload, or upstream head sampling.

Govern telemetry as sensitive data

Telemetry is a governed copy of application data. Prompts, system instructions, retrieved passages, headers, tool arguments and results, model outputs, attachments, conversation identifiers, and user feedback can contain credentials, personal information, confidential material, or regulated records. OpenTelemetry cannot infer what is sensitive in a particular application. Give every captured field a diagnostic purpose, classify it, and decide who may read it and for how long before enabling collection.

Prefer bounded metadata and stable references when raw content is unnecessary. When investigation requires content, store it separately with narrower access and retention where feasible. Redaction before capture prevents the application telemetry path from ever receiving a field. Redaction in a self-deployed collector can prevent it from reaching an external backend, but the field already existed in the application and transport to that collector. Redaction after export leaves another retained copy. These boundaries create materially different exposure.

Synthetic architecture comparison. Moving redaction later changes which systems ever receive raw content and which application buffers, transports, collector queues, external stores, derived views and backups require separate access and deletion controls. The illustration does not establish compliance or successful deletion.

Redaction must understand structure: headers, nested JSON, tool schemas, streaming fragments, and attachments require different handling. Test likely secrets, partial identifiers, multilingual personal data, encoding changes, and false positives that destroy diagnostic value. Hashing predictable identifiers does not automatically anonymize them because candidate values may be enumerable. Minimization is therefore a field, precision, recipient, and lifetime decision—not a one-time text substitution.

Inventory all copies through data-flow governance: application buffers, collectors, observability backends, external content stores, derived metrics, evaluator records, support tools, exports, and backups. Give metadata and payload evidence separate access rules where appropriate. Set artifact-specific retention with a clock-start event, expiry or review condition, disposal action, and accountable owner. Auditability is not permission to keep everything indefinitely.

Keep observability from failing the service

Observability consumes resources: instrumentation CPU and memory, serialization, network egress, collector queues, ingestion, indexing, storage, queries, evaluator records, and engineer review. Agent traces can contain large semi-structured text payloads and need both immediate per-trace inspection and aggregate analysis. Reported unusually large examples show the possible systems problem, not a typical trace size. Measure your workload rather than inheriting an anecdotal budget.

Bound metric cardinality, attribute and payload size, trace depth, queue capacity, batch size, and retention. Truncation can preserve pipeline availability while sacrificing evidence; expose truncation counters and diagnostic status. Batch export reduces per-record overhead but bounded queues can still drop data, and a flush request can fail or time out. Instrumentation callbacks should avoid blocking the application path; exporter failure policy must be explicit rather than accidentally determined by a full queue.

Instrumentation decision record

Choose a signal by the decision it enables and the burden it creates.
Decision fieldQuestion to answer
Operational purposeWhich investigation, alert, attribution, or governance decision uses this signal?
Fidelity and coverageWhat is captured, sampled, truncated, derived, or unavailable?
Volume and cardinalityHow do traffic, branches, labels, and payload size change cost?
SensitivityDoes it contain identity, prompts, retrieved content, arguments, results, or secrets?
Retention and accessWho may inspect it, for what purpose, and until which event or date?
Failure behaviorCan collection block user work? Which losses, delays, retries, and truncations become visible?
OwnerWho maintains semantics, budgets, alerts, access, and deletion behavior?

A useful observability system can say both “this is what the evidence establishes” and “this evidence is incomplete.” It preserves enough identity and structure to reconstruct consequential executions, enough aggregation to detect population changes, and enough integrity telemetry to reveal when those conclusions are no longer justified. More records are valuable only when they improve a decision enough to justify their runtime, financial, and privacy costs.

Open questions

  1. How can teams verify end-to-end instrumentation coverage when providers, queues, tool servers, and client applications expose different semantics? Progress would include coverage manifests, conformance tests, and explicit unavailable regions rather than inferred completeness.

  2. What common cost record could reconcile model modalities, paid tools, retries, discarded branches, shared infrastructure, mutable prices, currencies, and human review without double counting? Progress would preserve each accounting boundary and measured-versus-estimated status instead of forcing one unsupported total.

  3. How can rare severe failures be retained for diagnosis while telemetry still supports unbiased population estimates? Progress would combine known-probability monitoring samples with bounded diagnostic escalation and report selection policies alongside every quantitative claim.

  4. How can content-rich agent traces remain diagnostically useful under strict minimization and deletion requirements? Progress would provide structure-aware redaction tests, separately governed payload evidence, resolvable metadata references, and verified deletion across derived stores and exports.

  5. How can an investigation confirm arbitrary external business effects across systems that do not share one transaction or acknowledgment contract? Progress must come from operation-specific authoritative state, idempotency, and reconciliation interfaces; trace completion alone cannot supply it.

Follow the curated reading path through the speakers and demonstrations behind this entry.

9 min

AI Engineer Summit 2025 · 2025

OpenLLMetry is all you need

Nir Gazit

Cited in this entry

A concise introduction to logs, metrics, traces, automatic instrumentation, collectors, and the application of OpenTelemetry to generative-AI systems.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

353 matching talks

TalkSpeakerEventYear
Dat NgoAI Engineer Europe 20262026
Danny Gollapalli, Ben Hylak, Zubin KotichaAI Engineer Europe 20262026
Rustem FeyzkhanovAI Engineer World's Fair 20262026
Phil HetzelAI Engineer Europe 20262026
Daniel ChalefAI Engineer World's Fair 20262026
Vinoo GaneshAI Engineer World's Fair 20262026
Building security around ML

Transcript reviewed

Dr. Andrew DavisAI Engineer World's Fair 20242024
AI Engineering 101

Transcript reviewed

Noah HeinAI Engineer Summit 20232023
RAG for VPs of AI

Transcript reviewed

Jerry LiuAI Engineer World's Fair 20242024
Roy DerksAI Engineer Summit 20252025
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Laurie VossAI Engineer Europe 20262026
Ofer MendelevitchAI Engineer World's Fair 20252025
Mohak SharmaAI Engineer Summit 20252025
Phil HetzelAI Engineer Europe 20262026
Pierluca D'OroAI Engineer World's Fair 20262026
Nick Ung, Akshay SharmaAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Anju KambadurAI Engineer Summit 20252025
Daniel WhitenackAI Engineer World's Fair 20242024
Rene BrandelAI Engineer World's Fair 20252025
Security Firewall for Agents

Transcript reviewed

Ryan DahlAI Engineer World's Fair 20262026
Gaurav MishraAI Engineer World's Fair 20262026
Will BrownAI Engineer World's Fair 20262026
Nick HeinerAI Engineer World's Fair 20262026
Mark BissellAI Engineer World's Fair 20252025
Dat Ngo, Aman KhanAI Engineer World's Fair 20252025
Paul HenryAI Engineer World's Fair 20242024
Hamel Husain, Emil SedghAI Engineer World's Fair 20242024
Uday Kiran Medisetty, Adam HudaAI Engineer World's Fair 20262026
Thierry Moreau, Pedro TorruellaAI Engineer World's Fair 20242024
Jonathan MortensenAI Engineer World's Fair 20252025
Judging LLMs

Transcript reviewed

Alex VolkovAI Engineer World's Fair 20242024
Lukas PeterssonAI Engineer World's Fair 20262026
Anna Marie BenzonAI Engineer World's Fair 20262026
Eugene YanAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20252025
Chintan Agrawal, Daniel WirjoAI Engineer World's Fair 20262026
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Eugene Yan, Hamel Husain, Jason Liu, Dr Bryan Bischof, Charles Frye, Shreya ShankarAI Engineer World's Fair 20242024
Soumya Gupta, Jai ChopraAI Engineer World's Fair 20262026
Drasko ProfirovicAI Engineer World's Fair 20262026
Nishant GuptaAI Engineer World's Fair 20262026
Git push, get an AI API.

Transcript reviewed

Ryan Fox-TylerAI Engineer World's Fair 20242024
Xiaofeng WangAI Engineer Summit 20252025
Juan PeredoAI Engineer Summit 20252025
Diamond BishopAI Engineer Summit 20252025
Lawrence JonesAI Engineer Europe 20262026
Anish Agarwal, Matthew SchoenbauerAI Engineer World's Fair 20252025
Ritvik PandyaAI Engineer World's Fair 20262026
Matthias LoiblAI Engineer World's Fair 20252025
Agents Need Feature Flags

Transcript reviewed

Sachin GuptaAI Engineer World's Fair 20262026
Peter WielanderAI Engineer Code 20252025
Hamza TahirAI Engineer World's Fair 20262026
Eric AllamAI Engineer World's Fair 20252025
Rishabh BhargavaAI Engineer Europe 20262026
Nishant GuptaAI Engineer World's Fair 20262026
Sohail Shaikh, Ankush RastogiAI Engineer World's Fair 20262026
Jia WuAI Engineer World's Fair 20262026
Apoorva JoshiAI Engineer World's Fair 20262026
Angel Ortmann LeeAI Engineer World's Fair 20262026
Christopher Lovejoy, Saul HowardAI Engineer World's Fair 20262026
Jim BennettAI Engineer World's Fair 20252025
Zach BlumenfeldAI Engineer Europe 20262026
Merve NoyanAI Engineer Europe 20262026
John DickersonAI Engineer World's Fair 20252025
Harrison ChaseAI Engineer World's Fair 20252025
Agents Building Agents

Metadata candidate

Alfonso GrazianoAI Engineer World's Fair 20262026
Gabe De MesaAI Engineer World's Fair 20262026
Anita KirkovskaAI Engineer Summit 20252025
Olivier Leplus, Yohan LasorsaAI Engineer Europe 20262026
Gagan Bhat, Isabella Kai HeAI Engineer World's Fair 20262026
Richmond AlakeAI Engineer World's Fair 20252025
Henry MaoAI Engineer World's Fair 20252025
Stephen BatifolAI Engineer Europe 20262026
Aparna DhinakaranAI Engineer World's Fair 20252025
Samuel DentonAI Engineer World's Fair 20262026
SallyAnn DeLucia, Fuad AliAI Engineer Code 20252025
Will BrykAI Engineer World's Fair 20252025
Michael HablichAI Engineer Europe 20262026
Mahesh MuragAI Engineer Summit 20252025
Building AI For All

Metadata candidate

Amjad Masad, Michele CatastaAI Engineer Summit 20232023
Bennet FennerAI Engineer Europe 20262026
Michael AlbadaAI Engineer World's Fair 20252025
Harrison ChaseAI Engineer Summit 20232023
Anoop Kotha, Toki SherbakovAI Engineer World's Fair 20252025
Shaan DesaiAI Engineer Summit 20252025
Adam TerlsonAI Engineer Summit 20252025
Michael FesterAI Engineer World's Fair 20252025
Eric ZakariassonAI Engineer Europe 20262026
Abed MatiniAI Engineer World's Fair 20262026
Atul RamachandranAI Engineer World's Fair 20262026
Boris ChernyAI Engineer World's Fair 20252025
Cat Wu, Thariq Shihipar, Simon WillisonAI Engineer World's Fair 20262026
Sunil PaiAI Engineer Europe 20262026
Jacob KahnAI Engineer Code 20252025
Naman JainAI Engineer Code 20252025
Šimon PodhajskýAI Engineer Europe 20262026
Jedrick Kosinski, ComfyAnonymousAI Engineer World's Fair 20252025
Yusuf OlokobaAI Engineer Code 20252025
Conquering Agent Chaos

Metadata candidate

Rick BlalockAI Engineer World's Fair 20252025
Hanchi WangAI Engineer World's Fair 20242024
Mahesh SathiamoorthyAI Engineer World's Fair 20262026
Ben HylakAI Engineer World's Fair 20262026
Phil HetzelAI Engineer Europe 20262026
Kevin MaduraAI Engineer Code 20252025
Aparna Dhinkaran, Aparna DhinakaranAI Engineer Summit 20252025
Sayash KapoorAI Engineer Summit 20252025
Julia Neagu, Deanna Emery, Maitar AsherAI Engineer World's Fair 20252025
Mehedi HassanAI Engineer Europe 20262026
fighting slop with slop

Metadata candidate

Vaibhav GuptaAI Engineer World's Fair 20262026
Chaitanya AsawaAI Engineer World's Fair 20262026
Jason LopateckiAI Engineer World's Fair 20262026
Romain HuetAI Engineer World's Fair 20242024
Alex AtallahAI Engineer World's Fair 20252025
Ilan BigioAI Engineer Summit 20252025
Gateways are All You Need

Metadata candidate

Karan SampathAI Engineer Europe 20262026
Emil EifremAI Engineer World's Fair 20242024
KP Sawhney, Ian BallantyneAI Engineer Europe 20262026
Ash Prabaker, Andrew WilsonAI Engineer Europe 20262026
Chau TranAI Engineer World's Fair 20252025
Sarah Sachs, Carlos Esteban, Doug GuthrieAI Engineer World's Fair 20252025
Jeff Huber, Jason LiuAI Engineer World's Fair 20252025
Isaac RobinsonAI Engineer Europe 20262026
Sally-Ann DeLuciaAI Engineer Europe 20262026
Patricija ŽemaitytėAI Engineer World's Fair 20262026
Ankur Goyal, Olmo MaldonadoAI Engineer World's Fair 20242024
Samuel ColvinAI Engineer World's Fair 20252025
Hypermode Launch

Metadata candidate

Kevin Van GundyAI Engineer World's Fair 20242024
Vivek TrivedyAI Engineer World's Fair 20262026
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
Mahmoud MabroukAI Engineer Europe 20262026
Raymond FengAI Engineer World's Fair 20262026
Shlok KhemaniAI Engineer World's Fair 20262026
Daniel HanAI Engineer World's Fair 20242024
Ronan McGovernAI Engineer World's Fair 20252025
Pietro ZulloAI Engineer World's Fair 20262026
MCP is all you need

Metadata candidate

Samuel ColvinAI Engineer World's Fair 20252025
Theodora ChuAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer World's Fair 20262026
Alvaro MoralesAI Engineer World's Fair 20252025
Notion's Token Town

Metadata candidate

Sarah SachsAI Engineer World's Fair 20262026
On AI and Knowledge

Metadata candidate

Pablo CastroAI Engineer World's Fair 20262026
Frank CoyleAI Engineer World's Fair 20262026
Simon WillisonAI Engineer Summit 20232023
Kwindla Hultman KramerAI Engineer World's Fair 20252025
Juan Herreros ElorzaAI Engineer Europe 20262026
Samuel ColvinAI Engineer Europe 20262026
Steven MoonAI Engineer Summit 20252025
Lukas BiewaldAI Engineer World's Fair 20242024
Nick NisiAI Engineer Europe 20262026
Benoit SchillingsAI Engineer World's Fair 20262026
Rayan GargAI Engineer World's Fair 20262026
Patrick DeboisAI Engineer Summit 20252025
Aakanksha ChowdheryAI Engineer World's Fair 20252025
Scaling to Long Horizons

Metadata candidate

Ross Taylor, Chengxi TaylorAI Engineer World's Fair 20262026
Peter BarAI Engineer World's Fair 20252025
Giran Moodley, Mayan Soni, Oussama Hafferssas, Mayank SoniAI Engineer Europe 20262026
Marc KlingenAI Engineer Europe 20262026
Ishan AnandAI Engineer World's Fair 20242024
Charles PackerAI Engineer Summit 20252025
Manish SanwalAI Engineer Summit 20252025
David BrumleyAI Engineer World's Fair 20262026
Nuno CamposAI Engineer Europe 20262026
Michele CatastaAI Engineer Code 20252025
The AI Evolution

Metadata candidate

Mario RodriguezAI Engineer Summit 20232023
Brook RiggioAI Engineer World's Fair 20252025
Natalie MeurerAI Engineer World's Fair 20262026
Beyang LiuAI Engineer World's Fair 20252025
The End of Apps

Metadata candidate

KitzeAI Engineer Europe 20262026
Addy OsmaniAI Engineer World's Fair 20262026
Aparna DhinakaranAI Engineer World's Fair 20262026
Phil HetzelAI Engineer Europe 20262026
Raphael KalandadzeAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Ted JohnsonAI Engineer World's Fair 20262026
Jonathan FernandesAI Engineer World's Fair 20252025
Gorkem YurtsevenAI Engineer World's Fair 20252025
Alex VolkovAI Engineer World's Fair 20262026
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
Ayush BhardwajAI Engineer World's Fair 20262026
Rafal Wilinski, Vitor BaloccoAI Engineer World's Fair 20252025
Sonam PankajAI Engineer World's Fair 20262026
Victor DibiaAI Engineer World's Fair 20252025
Philipp KrennAI Engineer World's Fair 20252025
Vibes won't cut it

Metadata candidate

Chris KellyAI Engineer World's Fair 20252025
Peter GostevAI Engineer Europe 20262026
Sam JulienAI Engineer World's Fair 20252025
Philipp SchmidAI Engineer Europe 20262026
Manu GoyalAI Engineer World's Fair 20252025
Dan FarrellyAI Engineer World's Fair 20262026
Your agent is blindfolded

Metadata candidate

Johan LajiliAI Engineer Europe 20262026
Veronica HylakAI Engineer World's Fair 20262026
Dex HorthyAI Engineer World's Fair 20252025
Justin MullerAI Engineer World's Fair 20252025
Kevin HouAI Engineer Summit 20252025
Rajat ShahAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Justin SmithAI Engineer World's Fair 20262026
Greg BensonAI Engineer World's Fair 20252025
Shawn ChanAI Engineer World's Fair 20262026
Raj NavakotiAI Engineer Europe 20262026
Building a Chess Coach

Metadata candidate

Anant Dole, Asbjørn SteinskogAI Engineer Europe 20262026
Cornelia DavisAI Engineer Code 20252025
Damien MurphyAI Engineer World's Fair 20242024
Dylan PatelAI Engineer World's Fair 20242024
Val Bercovici, Callan FoxAI Engineer Code 20252025
Sylendran ArunagiriAI Engineer World's Fair 20252025
Rhythm Garg, Linden LiAI Engineer Code 20252025
Joseph Wang, Sid PatllolluAI Engineer World's Fair 20262026
Ending AI Slop

Metadata candidate

Thais Castello BrancoAI Engineer World's Fair 20262026
Rafael LeviAI Engineer Europe 20262026
Fuzzing in the GenAI Era

Metadata candidate

Leonard TangAI Engineer World's Fair 20252025
Brian JohnAI Engineer Code 20252025
Evan BoyleAI Engineer World's Fair 20252025
Ben KunkleAI Engineer Europe 20262026
AI Engineer Summit 20252025
Michael RichmanAI Engineer Europe 20262026
Kelvin MaAI Engineer World's Fair 20252025
Lin Qiao, Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Shirsha ChaudhuriAI Engineer Summit 20252025
Ryan MartenAI Engineer World's Fair 20252025
Dmitry KuchinAI Engineer World's Fair 20252025
Shawn JanseparAI Engineer World's Fair 20242024
Kyle Penfound, Jeremy Adams - CasañasAI Engineer World's Fair 20252025
Eno ReyesAI Engineer World's Fair 20252025
Louis Knight-WebbAI Engineer Europe 20262026
Alejandro VidalAI Engineer World's Fair 20262026
Barr YaronAI Engineer World's Fair 20252025
Barr YaronAI Engineer World's Fair 20262026
Zack Reneau-WedeenAI Engineer Summit 20252025
The Agentic AI Engineer

Metadata candidate

Benedikt Sanftl, Burak Cemil ÖzafşarAI Engineer World's Fair 20262026
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
Trust, but Verify

Metadata candidate

Shreya RajpalAI Engineer Summit 20232023
Mike ConoverAI Engineer Summit 20252025
Vision: Zero Bugs

Metadata candidate

Johann Schleier-SmithAI Engineer Code 20252025
Bilge YücelAI Engineer Europe 20262026
Remy GuercioAI Engineer Europe 20262026
Lei ZhangAI Engineer Code 20252025
Why MLX

Metadata candidate

AI Engineer Europe 20262026
Ameya BhatawdekarAI Engineer World's Fair 20262026
Erik HanchettAI Engineer World's Fair 20262026
Sachin KumarAI Engineer World's Fair 20262026
Melanie WarrickAI Engineer World's Fair 20262026
Evan KormosEvent unknownUnknown
Remy GuercioAI Engineer World's Fair 20262026
Anekha SokhalEvent unknownUnknown
Matt BrockmanAI Engineer World's Fair 20262026
Roberto Milev, Uday KanagalaAI Engineer World's Fair 20262026
Dmitry BuykinAI Engineer World's Fair 20262026
Noyan TokgozogluEvent unknownUnknown
Vayum Arora, Zhengyao Jiang, Dixing Xu, Dhruv SrikanthAI Engineer World's Fair 20262026
Marina PetzelEvent unknownUnknown
Giselle van DongenEvent unknownUnknown
Vishal Srivastava, Tanmay SahEvent unknownUnknown
Xander SongEvent unknownUnknown
Vlad LuzinAI Engineer World's Fair 20262026
Gareth YoderEvent unknownUnknown
Laurie VossAI Engineer World's Fair 20262026
Matt GibiecAI Engineer World's Fair 20262026
Sachin MalhotraAI Engineer World's Fair 20262026
Fuad AliAI Engineer World's Fair 20262026
Zubin AysolaAI Engineer World's Fair 20262026
Jetashree RaviAI Engineer World's Fair 20262026
Michele GrantAI Engineer NYC 2026 Online Talks2026
Bogdan GazaAI Engineer Code 2025Unknown
Hossein NiazmandiAI Engineer World's Fair 20262026
Fuad AliEvent unknownUnknown
Du’An Lightfoot, Khaja OmerAI Engineer World's Fair 20262026
Jose PalafoxAI Engineer World's Fair 20262026
Ankur DuggalAI Engineer World's Fair 20262026
Aparna DhinakaranEvent unknownUnknown
Asaf Gardin, Yuval BelferAI Engineer World's Fair 20262026
Ali-Reza Adl-TabatabaiAI Engineer World's Fair 20262026
Laurie VossAI Engineer World's Fair 2026Unknown
Jess WangEvent unknownUnknown
Shy RuparelAI Engineer New York 20262026
Tisha Chawla, Susheem KoulAI Engineer World's Fair 20262026
Dan BălăceanuAI Engineer World's Fair 20262026
Yohan RajuAI Engineer World's Fair 20262026
Janie Gu, Suyeol YunEvent unknownUnknown
Jakub HojsanAI Engineer Code 2025Unknown
Felipe BlanesAI Engineer World's Fair 20262026
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Kent C. DoddsEvent unknownUnknown
Wolfram RavenwolfAI Engineer World's Fair 20262026
SallyAnn DeLuciaEvent unknownUnknown
Fuad AliEvent unknownUnknown
Marc KlingenAI Engineer World's Fair 20262026
Morgan SuoEvent unknownUnknown
Jake BroekhuizenAI Engineer World's Fair 20262026
Connor Guerrero, Young Jeong, Nikhil GuptaAI Engineer Code 2025Unknown
Sarah SimionescuAI Engineer Code 2025Unknown
Rémi LoufAI Engineer World's Fair 20262026
Chris VelaEvent unknownUnknown
David AbitbolEvent unknownUnknown
Laurie VossAI Engineer World's Fair 20262026
Eric SchwartzAI Engineer Code 2025Unknown
Varun KrovvidiEvent unknownUnknown
Joshua MoAI Engineer World's Fair 20262026
Oliver KennonEvent unknownUnknown
Nina Žižakić, Eleonora DrykinaAI Engineer NYC 2026 Online Talks2026
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Aidan SochowskiEvent unknownUnknown
Sandhya SubramaniEvent unknownUnknown
Kevin OrellanaAI Engineer World's Fair 20262026
Harish GaggarAI Engineer NYC 2026 Online Talks2026
Ang LiAI Engineer World's Fair 20262026
Sandhya Subramani, Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Al ChenEvent unknownUnknown
CJ Avilla, Harrison StallEvent unknownUnknown
Jason MaAI Engineer World's Fair 20262026
Yunmo KooAI Engineer World's Fair 20262026
Nick Nisi, Zack ProserEvent unknownUnknown
Vinoo GaneshEvent unknownUnknown
Branislav BrzakAI Engineer NYC 2026 Online Talks2026
Viren BaraiyaAI Engineer World's Fair 20262026
Tim TylerEvent unknownUnknown
Sofia JakovcevicEvent unknownUnknown
Charles FryeAI Engineer World's Fair 20262026
Annabell Schäfer, Doneyli De JesusEvent unknownUnknown
Andrey RybkaEvent unknownUnknown
Jesse HallEvent unknownUnknown
Allie Howe, Ian Livingstone, Michael Davis, Dick HardtEvent unknownUnknown
Ben DickenAI Engineer World's Fair 20262026
Shubhankar SrivastavaAI Engineer World's Fair 20262026
Michael PattersonEvent unknownUnknown
Jiachen JiangAI Engineer NYC 2026 Online Talks2026
Jacob Lauritzen, Simon EskildsenEvent unknownUnknown
Vlad LuzinEvent unknownUnknown
Evan ConradEvent unknownUnknown
Alvin PaneAI Engineer NYC 2026 Online Talks2026
Lotte Verheyden, Annabell SchäferAI Engineer World's Fair 20262026
Paden GayleAI Engineer NYC 2026 Online Talks2026
Eduardo Simon, Javier CasalAI Engineer NYC 2026 Online Talks2026
Fuad AliEvent unknownUnknown
Tim SweeneyAI Engineer World's Fair 20262026
Kanish ManujaAI Engineer World's Fair 20262026
Gabe Cemaj, Priyanka PhatakAI Engineer World's Fair 20262026
Doug GuthrieAI Engineer World's Fair 20262026
Derek MeeganAI Engineer Code 2025Unknown
Yoni Michael, Brandon CallenderAI Engineer World's Fair 20262026
Stephanie JarmakEvent unknownUnknown
Justin ReockEvent unknownUnknown
Laurie VossEvent unknownUnknown
Michael Forrester, Whitney LeeAI Engineer World's Fair 20262026
Suchet BargotiAI Engineer World's Fair 20262026
Abhi AryaEvent unknownUnknown
Karthik Ranganathan, Heather DowningAI Engineer World's Fair 20262026
Salman MunafAI Engineer World's Fair 20262026
Pedro LopezAI Engineer World's Fair 20262026
Sam ProdgerAI Engineer World's Fair 20262026
Hrishi OlickelAI Engineer NYC 2026 Online Talks2026

References

Coverage and source review
Processed transcripts
67 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
291 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. OpenTelemetry: Observability primer

    https://opentelemetry.io/docs/concepts/observability-primer/

  2. OpenTelemetry: Traces

    https://opentelemetry.io/docs/concepts/signals/traces/

  3. Google SRE: Effective Troubleshooting

    https://sre.google/sre-book/effective-troubleshooting/

  4. OpenTelemetry GenAI model and tool spans

    https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-spans.md

  5. RabbitMQ: Consumer Acknowledgements and Publisher Confirms

    https://www.rabbitmq.com/docs/confirms

  6. Metrics design — vLLM

    https://docs.vllm.ai/en/stable/design/metrics/

  7. OpenAI Python SDK: ChatCompletionStreamOptionsParam

    https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_stream_options_param.py

  8. LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize AI

    https://www.youtube.com/watch?v=JsCCrBF7F1g&t=198

  9. Everything You Need To Know About Agent Observability

    https://www.youtube.com/watch?v=-aM2EDTiaMs&t=2702

  10. How agent o11y differs from traditional o11y

    https://www.youtube.com/watch?v=XBaznoTRDFI&t=394

  11. The NetLogger Methodology for High Performance Distributed Systems Performance Analysis

    https://www.es.net/assets/pubs_presos/NetLogger.HPDC.paper.ieee.pdf

  12. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

    https://code.garrettmills.dev/Archives/papers-we-love_papers-we-love/raw/commit/5d30a30605e226c6736f7d81e5caf5131aa2b08e/distributed_systems/dapper-a-large-scale-distributed-tracing-infrastructure.pdf

  13. Canopy: An End-to-End Performance Tracing And Analysis System

    https://cs.brown.edu/people/jcmace/papers/kaldor2017canopy.pdf

  14. X-Trace: A Pervasive Network Tracing Framework

    https://static.usenix.org/event/nsdi07/tech/full_papers/fonseca/fonseca_html/index.html

  15. A brief history of OpenTelemetry (So Far)

    https://www.cncf.io/blog/2019/05/21/a-brief-history-of-opentelemetry-so-far/

  16. W3C Trace Context Recommendation

    https://www.w3.org/TR/trace-context/

  17. Sunsetting OpenCensus

    https://opentelemetry.io/blog/2023/sunsetting-opencensus/

  18. The LinkedIn Generative AI Application Tech Stack: Extending to Build AI agents

    https://news.linkedin.com/2025/linkedin-s-ai-stack--expanding-to-build-tomorrow-s-agents

  19. OpenTelemetry GenAI agent spans

    https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-agent-spans.md

  20. Traces — OpenTelemetry

    https://opentelemetry.io/docs/concepts/signals/traces/

  21. OpenTelemetry: Semantic conventions for messaging spans

    https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/

  22. OpenTelemetry: Context propagation

    https://opentelemetry.io/docs/concepts/context-propagation/

  23. Temporal: Workflow Id and Run Id

    https://docs.temporal.io/workflow-execution/workflowid-runid

  24. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

    https://research.google.com/archive/papers/dapper-2010-1.pdf

  25. OpenLLMetry is all you need

    https://www.youtube.com/watch?v=KVgbERRPU4M&t=86

  26. OpenTelemetry Logging Specification

    https://opentelemetry.io/docs/specs/otel/logs/

  27. OpenTelemetry Metrics API: Gauge

    https://opentelemetry.io/docs/specs/otel/metrics/api/#gauge

  28. OpenTelemetry Metrics Data Model

    https://opentelemetry.io/docs/specs/otel/metrics/data-model/

  29. Histograms and summaries — Prometheus

    https://prometheus.io/docs/practices/histograms/

  30. Prometheus: metric-label cardinality and resource costs

    https://prometheus.io/docs/practices/instrumentation/

  31. OpenTelemetry: Handling sensitive data

    https://opentelemetry.io/docs/security/handling-sensitive-data/

  32. OpenTelemetry: Sampling

    https://opentelemetry.io/docs/concepts/sampling/

  33. Transformers: chat templates serialize conversation history

    https://huggingface.co/docs/transformers/chat_templating

  34. Transformers GenerationConfig: stopping and output budgets

    https://huggingface.co/docs/transformers/main_classes/text_generation

  35. OpenTelemetry GenAI semantic conventions: spans

    https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

  36. Your Agent Failed in Prod. Good Luck Reproducing It.

    https://www.youtube.com/watch?v=Lc8zRh9muoY&t=385

  37. Your Agent Failed in Prod. Good Luck Reproducing It.

    https://www.youtube.com/watch?v=Lc8zRh9muoY&t=520

  38. Node.js HTTP: server response completion and termination

    https://nodejs.org/api/http.html

  39. Making Retries Safe with Idempotent APIs

    https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

  40. What We Learned From A Year of Building With LLMs

    https://www.youtube.com/watch?v=qBHfQT3YtyY&t=1798

  41. OpenTelemetry Telemetry Schemas

    https://opentelemetry.io/docs/specs/otel/schemas/

  42. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    https://www.youtube.com/watch?v=Lcqat4iP_lE&t=419

  43. The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

    https://www.youtube.com/watch?v=Lcqat4iP_lE&t=534

  44. OpenTelemetry Baggage API

    https://opentelemetry.io/docs/specs/otel/baggage/api/

  45. OpenTelemetry: Tracing SDK

    https://opentelemetry.io/docs/specs/otel/trace/sdk/

  46. Resiliency — OpenTelemetry Collector

    https://opentelemetry.io/docs/collector/resiliency/

  47. OpenTelemetry Collector Contrib: Tail Sampling Processor

    https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md

  48. OpenTelemetry: OTLP Specification 1.11.0

    https://opentelemetry.io/docs/specs/otlp/

  49. LangGraph: checkpoint state and execution identity

    https://docs.langchain.com/oss/python/langgraph/checkpointers

  50. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

    https://arxiv.org/abs/2305.04388

  51. Anthropic: Reasoning models don't always say what they think

    https://www.anthropic.com/research/reasoning-models-dont-say-think

  52. LangGraph: Use time-travel

    https://docs.langchain.com/oss/python/langgraph/use-time-travel

  53. LangGraph: replaying from an earlier checkpoint

    https://docs.langchain.com/oss/python/langgraph/use-time-travel

  54. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    https://www.youtube.com/watch?v=svCnShDvgQg&t=475

  55. Mind the Gap (In your Agent Observability)

    https://www.youtube.com/watch?v=iOXM3zE-2dk&t=2084

  56. Lawrence Jones - Fighting AI with AI

    https://www.youtube.com/watch?v=L2r6vLlLgs8&t=567

  57. VCR.py: Usage and record modes

    https://vcrpy.readthedocs.io/en/latest/usage.html

  58. OpenTelemetry GenAI semantic conventions: metrics

    https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md

  59. CRISP: Critical Path Analysis of Large-Scale Microservice Architectures

    https://www.usenix.org/system/files/atc22-zhang-zhizhou.pdf

  60. AI Engineering 201: The Rest of the Owl

    https://www.youtube.com/watch?v=zl4EdALzktU&t=2661

  61. Designing Voice Agents for Real Conversations

    https://www.youtube.com/watch?v=hMlLw1LeIK8&t=1135

  62. LLM cost management: how to track, attribute, and control spend — Langfuse

    https://langfuse.com/resources/engineering/llm-cost-management

  63. Agentic SDLC at Uber - Building Blocks for Uber’s Software Factory

    https://www.youtube.com/watch?v=17-YSUHo6Lk&t=135

  64. OpenTelemetry Metrics — gRPC

    https://grpc.io/docs/guides/opentelemetry-metrics/

  65. How do I export monthly usage details from the API Usage Dashboard?

    https://help.openai.com/en/articles/20001072

  66. Allocation — FinOps Framework

    https://www.finops.org/framework/capabilities/allocation/

  67. How Forward Deployed Engineering is done at Cognition

    https://www.youtube.com/watch?v=RVxym6mmIns&t=303

  68. Implementing SLOs — Google SRE Workbook

    https://sre.google/workbook/implementing-slos/

  69. Google SRE: Monitoring Distributed Systems

    https://sre.google/sre-book/monitoring-distributed-systems/

  70. Production Evals For Agentic AI Systems

    https://www.youtube.com/watch?v=vljxQZfJ9wY&t=163

  71. AI System Design: From Idea to Production

    https://www.youtube.com/watch?v=T0HhO4YtTfE&t=1440

  72. LangSmith: Feedback data format

    https://docs.langchain.com/langsmith/feedback-data-format

  73. National Academies: inference with missing outcomes

    https://www.ncbi.nlm.nih.gov/books/NBK209900/

  74. National Academies: informative censoring and sensitivity analysis

    https://www.ncbi.nlm.nih.gov/books/NBK209895/

  75. Operationalizing Machine Learning: An Interview Study

    https://arxiv.org/html/2209.09125

  76. Understanding Simpson’s Paradox — Judea Pearl

    https://www.cs.ucla.edu/~kaoru/r385.pdf

  77. Practical Guide to Controlled Experiments on the Web

    https://ai.stanford.edu/~ronnyk/GuideControlledExperiments.pdf

  78. OpenTelemetry Tracing SDK: sampling

    https://opentelemetry.io/docs/specs/otel/trace/sdk/

  79. OpenTelemetry Collector Contrib: Tail Sampling Processor

    https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md

  80. An Online Probabilistic Distributed Tracing System

    https://arxiv.org/abs/2405.15645

  81. OpenTelemetry: Handling sensitive data

    https://opentelemetry.io/docs/security/handling-sensitive-data/

  82. OWASP: Logging Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html

  83. OpenLLMetry is all you need

    https://www.youtube.com/watch?v=KVgbERRPU4M&t=326

  84. OpenTelemetry GenAI client spans and content capture

    https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

  85. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.01162020.pdf

  86. How agent o11y differs from traditional o11y

    https://www.youtube.com/watch?v=XBaznoTRDFI&t=442

  87. Why building eval platforms is hard

    https://www.youtube.com/watch?v=_fQ7Z_Wfouk&t=1047

  88. Billable Units — Langfuse

    https://langfuse.com/docs/administration/billable-units

  89. Configure Tempo — attribute limits and compression

    https://grafana.com/docs/tempo/latest/configuration/

  90. Your Agent Failed in Prod. Good Luck Reproducing It.

    https://www.youtube.com/watch?v=Lc8zRh9muoY&t=471

  91. Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

    https://www.youtube.com/watch?v=LrGCT7G_rU8&t=135

  92. AWS Builders' Library: timeouts, retries, backoff and jitter

    https://d1.awsstatic.com/builderslibrary/pdfs/timeouts-retries-and-backoff-with-jitter.pdf