AI Engineer Summit 2025

OpenLLMetry is all you need

Nir Gazit9:12

Loading slides…

Read the talk

OpenLLMetry is all you need

Logs, metrics and traces already describe cloud applications. OpenLLMetry extends that machinery to model calls, vector databases and AI frameworks.

From a talk by Nir Gazit

Before you start: Basic familiarity with application code and client libraries is helpful; no prior OpenTelemetry experience is required.

How do you observe what an application is doing?

How do you see what is happening inside a cloud application—and keep that visibility in the observability platform you already use? OpenLLMetry, introduced by Nir Gazit, CEO of Traceloop, starts with an existing foundation: OpenTelemetry. Before adding LLMs to the picture, it helps to understand what that foundation provides.

OpenTelemetry is a CNCF-maintained project that standardizes cloud observability. Gazit describes it as one of CNCF’s largest projects after Kubernetes; the relevant comparison is project activity, rather than adoption. CNCF’s mid-year 2024 analysis ranked OpenTelemetry second in project velocity for July 2023–July 2024. Its practical appeal is broad backend support: Splunk, Datadog, Dynatrace, New Relic, Grafana and Honeycomb are among the platforms Gazit names. A common telemetry format lets application instrumentation work with those existing destinations.

0:010:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Events, aggregates and multistep requests

Gazit introduces OpenTelemetry through its protocol, which standardizes logs, metrics and traces. More fully, OpenTelemetry is a framework and toolkit containing protocols, APIs, SDKs and other components. Start with the simplest signal: a log records an event at some point during an application’s lifetime, optionally with metadata. If you have written a Python print statement, you have already used the basic idea. For example, this records a request event and its identifier:

python

import json

request_id = "request-123"
print(json.dumps({
    "event": "request_received",
    "request_id": request_id,
}))

The event says something happened. By itself, it does not describe an aggregate trend or the request’s complete journey.

Metrics describe behavior in aggregate: across days, across users or across another grouping. Traditional cloud metrics include CPU usage, memory usage and latency. For a GenAI application, the corresponding questions include how many tokens it consumes, how long calls take and how often they fail.

Application contextExample metrics
Cloud infrastructureCPU usage, memory usage, latency
GenAI applicationToken usage, latency, error rate

Latency appears in both rows because adding a model does not remove the need to understand response time.

Metrics slide listing token usage, latency, and error rate beside dashboard charts, with the speaker inset at lower right.Open full source frame
Metrics include token usage, latency, and error rate.

Tracing follows a multistep process. Gazit identifies tracing as OpenTelemetry’s first defined signal. In a microservice architecture, a trace connects the work performed as a user request passes through several services. The same structure fits GenAI applications: chains, workflows and agents can make several calls and execute tools before producing a result. A trace preserves the relationship between those steps, giving the individual operations a shared request context.

1:041:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04 · section reference included

From a standard to working telemetry

A format alone cannot collect data from an application. OpenTelemetry supplies an ecosystem for producing and moving that data, beginning with SDKs for explicit instrumentation. At the time of the talk, Gazit reports SDK support for eleven languages, including Python, TypeScript, Go and C++.

The components divide the work into three roles:

ComponentRole
SDKExplicitly emit logs, metrics and trace spans from application code
InstrumentationAutomatically capture telemetry from supported libraries
CollectorProcess telemetry before exporting it to a backend

With an SDK, you decide where to emit a signal and write the call yourself. Instrumentation supplies that behavior for supported dependencies. A Collector handles the subsequent processing stage before the data reaches an observability platform.

3:203:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:20 · section reference included

Automatic instrumentation wraps the client library

Consider an application that accesses Postgres. Rather than manually adding telemetry around every database operation, you can use instrumentation for the database client. In the mechanism Gazit describes, the instrumentation monkey-patches the client library: it wraps the library’s behavior inside the application and emits telemetry when the application uses it. This places capture on the application side, where the outgoing operations are visible.

That is the source of the automatic visibility: the instrumented dependency produces telemetry as existing application code calls it. Gazit describes the latency impact as almost negligible, but supplies no workload, baseline or measurement conditions. The talk therefore establishes the capture mechanism, while leaving its overhead for a particular application unmeasured.

4:284:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:28 · section reference included

Control what leaves the application environment

Collectors are self-deployable components that can run inside your own cloud environment, including on Kubernetes. They provide a processing point between telemetry generation and delivery to an external platform.

Their useful operations include:

  • Filtering: remove telemetry that is not useful for your application’s observability needs.
  • Sensitive-data handling: obscure personally identifiable information or other sensitive values before export.
  • Multiple destinations: send telemetry to more than one provider.

Open-source Collector components provide these processing capabilities, but sensitive-data handling requires configuration; deploying a Collector does not automatically establish a privacy policy. This layer gives you a place to apply that policy before forwarding the data.

Collectors slide with bullets for processing telemetry, filtering, obscuring PII and sensitive data, and sending to multiple sinks, beside a colorful illustration.Open full source frame
Collectors filter telemetry, obscure sensitive data, and send to multiple sinks.
5:265:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:26 · section reference included

Extend the same machinery to GenAI dependencies

OpenLLMetry applies this instrumentation approach to foundation models, vector databases and GenAI frameworks. The resulting telemetry uses OpenTelemetry, so the destination can remain the platform the application team already uses. Gazit names Datadog, Sentry, Grafana Tempo and Dynatrace as trace destinations. Automatic collection still needs a correctly configured export destination.

At the time of the talk, Gazit reports more than forty providers supported through community-built instrumentations. The integrations span several layers of an AI application:

  • Model services: OpenAI, Anthropic, Cohere, Gemini and Bedrock.
  • Vector databases: Pinecone and Chroma.
  • Application frameworks: LangChain, LlamaIndex, CrewAI and Haystack.

The aim is automatic visibility into these dependencies through the same logs, metrics and traces model. The precise signals and fields available depend on the integration; the list does not establish identical coverage for every library. Once the appropriate instrumentation and export configuration are in place, supported operations can emit telemetry without manual emission around every call.

6:086:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:08 · section reference included

What Pinecone instrumentation exposes

The Pinecone integration makes the idea concrete. Gazit describes visibility into outgoing queries and indexing activity, followed by the ability to investigate returned vectors. That connects the operation sent by the application to the retrieval results it receives.

The demonstrated scope includes returned vector data, vector distances, scores and latencies in standard OpenTelemetry format. These provide different views of an operation: the query shows what was requested, the returned data and scores help explain the retrieval result, and latency describes how long the operation took. This is the useful extension beyond observing only whether a database call succeeded.

Pinecone Instrumentation slide listing queries and indexes, returned vector data, distances and scores, and score and latency metrics beside the Pinecone logo.Open full source frame
Pinecone instrumentation exposes queries, returned vectors, scores, and latency.
7:538:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:53 · section reference included

Keep instrumentation independent of the backend

A shared telemetry standard separates application instrumentation from the choice of observability platform. OpenLLMetry connects LLM applications to an existing backend through OpenTelemetry. Gazit characterizes switching platforms as a configuration change because compatible platforms accept the same telemetry format. That portability applies to telemetry export: storage, visualization and backend-specific functionality remain the responsibility of the receiving tools. The application can retain its instrumentation while changing where its telemetry goes.

8:288:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:28 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…