AI Engineer World's Fair 2025

Maximize GPU Efficiency with Continuous Profiling for GPUs

Loading slides…

Read the talk

Finding Where GPU Time Goes with Continuous Profiling

Matthias Loibl walks from sampled CPU stacks to GPU telemetry and GPU-time profiles, showing how to investigate idle accelerators and trace device work back to its callers.

From a talk by Matthias Loibl

Before you start: Familiarity with function call stacks, CPU–GPU workloads, and basic Linux deployment concepts will help you follow the profiling examples.

Where is the machine spending its time?

Where does a program spend its time—and which work could you remove or make cheaper? Profiling supplies the measurements needed to answer that question. Matthias Loibl tentatively traces its history to IBM work in the 1970s, but the practical questions remain familiar: how much memory does a program use, where does CPU or GPU time go, which instructions execute, and how frequently and for how long do functions run?

Slide titled “What?” defines profiling and lists space, time complexity, instruction usage, and frequency and duration of function calls.Open full source frame
Profiling measures memory, CPU and GPU time, instruction usage, and function calls.

Those measurements connect performance engineering to infrastructure cost. Loibl offers a hypothetical: a 10% software performance improvement might let an operator turn off 10% of its servers. The point is the possibility of reducing the capacity needed for a workload, not a guaranteed conversion from faster software to a smaller bill.

0:170:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Trade event completeness for continuous observation

A profiler can record every event, or periodically observe what is happening. Recording everything provides detail, but collection cost and data volume make that approach difficult to sustain continuously.

ApproachWhat it recordsTradeoff
TracingEvery eventDetailed history; high cost and data volume
SamplingObservations at selected intervalsLower collection volume; individual events can be missed

Loibl's sampling example uses a ten-second window with an adjustable rate, such as 20 or 100 samples per second. The rate controls how often the profiler looks, rather than how often the application performs work.

Loibl associates 100 samples per second with less than 1% CPU overhead and cites approximately 4 MB of memory overhead. He does not supply the workload, hardware, or measurement conditions for those figures. The broader tradeoff is clear: sampling will miss events, but collecting continuously gives recurring work more opportunities to appear. A stack that executes only once matters less to this view than work that repeatedly consumes resources.

Each observation captures a call stack: the chain of functions calling one another at that moment. For CPU profiling, that means recording the stack executing on the CPU; for allocation profiling, it can mean recording the stack responsible for allocating memory. Repeated observations build a picture of where the application spends its resources without requiring a complete event history.

1:321:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:32 · section reference included

Observe the production environment

The useful profile is the one from the environment running the workload. A developer's machine does not reproduce production, so low overhead matters partly because it makes production observation practical. Loibl's collector uses Linux eBPF, drawing on kernel facilities to gather profiles. He describes starting one collector to profile applications without changing or individually instrumenting them.

Loibl introduces himself as the Berlin-based director of Polar Signals Cloud and a maintainer of Prometheus, Prometheus Operator, and Parca. He presents Parca as the open-source counterpart to the profiling work, before turning from established CPU and memory collection to the GPU preview.

3:243:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:24 · section reference included

Use GPU metrics to choose the investigation

After three or four years working on CPU and memory profiling, the team began a GPU profiling preview earlier in the year of the talk. The first view uses the NVIDIA Management Library, NVML, to collect GPU metrics. In the demonstration, the blue utilization line represents the overall node, while the orange line represents one GPU process, identified by its process ID. Memory utilization and clock speed appear alongside those signals. A utilization drop identifies a time interval worth investigating: what stopped the GPU from staying busy?

The next metrics help distinguish possible constraints:

  • Power: Compare measured power usage with the dashed power-limit line.
  • Temperature: Look for heat that could lead to throttling. Loibl warns about sustained temperatures around 80°C; that is an approximate warning, not a threshold shared by every GPU.
  • PCIe throughput: Investigate whether transfers between CPU and GPU are limiting the workload. In the displayed convention, negative values represent receiving and positive values represent sending; Loibl points to an example of sending 10 MB/s.

These signals narrow the question from “Why is utilization low?” to a more specific investigation of power, cooling, or data movement.

4:264:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Find what the CPU does while the GPU waits

GPU telemetry becomes more useful when it shares a time axis with stored CPU profiles. The collector already captures CPU stacks with eBPF, so an interval of interest in the GPU metrics can lead directly into host-side execution.

  1. Find a time interval worth investigating in the metrics.
  2. Inspect the corresponding CPU activity. Loibl selects activity on CPU zero near the end of the example.
  3. Drag over that time range to open a flame chart.
  4. Follow the call stack to see what the CPU was executing while the GPU was not fully utilized.

In the selected profile, Python eventually calls into CUDA functions. The chart supplies the calling context that the utilization curve alone cannot show.

Slide titled “Visualize with flame charts” shows a vertical colored call stack with a function-details tooltip.Open full source frame
A CPU flame chart helps investigate what happens while the GPU is not fully utilized.

A common explanation is that the CPU is busy loading data and fails to keep the GPU supplied with work. The host can therefore be active while the accelerator remains underutilized. This is why looking only at GPU utilization leaves an important part of the workload unexplained.

The same investigation is not limited to Python. Loibl names Rust calling CUDA, compiled-language stacks generally, and runtime support including Python, Ruby, and the JVM. The underlying profiling approach also applies beyond accelerator workloads—to web servers, databases, and vector databases whose performance depends on where execution time goes.

6:246:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:24 · section reference included

Attribute GPU execution time to its callers

CPU profiles explain host activity, but they do not directly answer how much GPU time a calling function causes. Loibl next introduces GPU-time profiling, a new preview demonstrated that morning, distinct from the earlier GPU metrics preview. The small example shows branches for matmul_naive and generate_random_matrix: matrix multiplication and random-matrix generation become identifiable work in a profile.

Loibl describes obtaining a start timestamp when a CUDA call stack appears on the CPU and a termination timestamp for the GPU kernel, then using the interval to determine the kernel's GPU duration. Once matching device-execution timestamps are available, the duration calculation is straightforward:

python

def gpu_duration_ns(start_ns: int, end_ns: int) -> int:
    if end_ns < start_ns:
        raise ValueError("Kernel end precedes kernel start")
    return end_ns - start_ns

The difficult part is obtaining and matching the right events. CUDA launches are asynchronous, and the talk does not specify the launch-to-completion correlation mechanism; timing only a host launch call would not establish device execution time.

Slide titled “GPU-time profiling” shows horizontal colored stack bars branching into matmul_naive and generate_random_matrix.Open full source frame
A small GPU-time profile shows branches for matrix multiplication and random-matrix generation.

The resulting view attributes GPU time to individual functions. Loibl then moves from the small CUDA example to a larger profile, following a Python main function down through the calling stack into libcuda. The host call path provides the context for the device work.

8:138:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:13 · section reference included

Read widths, leaves, and colors separately

The larger chart prompts a useful question: does it show CPU time, GPU time, or both? Its calling path comes from the CPU, while its width represents attributed GPU time. Loibl explains that each stack's leaf is the function consuming time on the GPU. The chart connects the host's chain of calls to device execution rather than simply combining two unrelated timing measurements.

A second question concerns the colors. They distinguish binaries, not timing categories. Blue identifies Python in this example; Loibl tentatively identifies one of the lower binaries as CUDA.

Chart featureMeaning
WidthAttributed GPU time
Path to the leafCPU calling stack
LeafFunction consuming GPU time
ColorBinary associated with the frame

Keeping those meanings separate lets the reader follow which application code led to device work without mistaking a color for a measure of expense.

GPU-time profiling slide with a branching chart of purple, blue, green, and cyan horizontal bars.Open full source frame
The larger GPU-time profile contains colored branches of varying widths and depths.
9:379:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:37 · section reference included

Deploy the collector

The deployment described in the talk starts with a Linux host and eBPF. The collector can run as a binary managed by systemd, in Docker, or as a Kubernetes DaemonSet. For Kubernetes, Loibl describes obtaining the manifest YAML, supplying a token, and deploying it. This is the talk's collector setup: the later CUDA profiling release additionally documents a GPU-process injection library and agent configuration, so the preview's setup should not be treated as a complete current CUDA installation procedure.

Existing CPU and memory profiling customers were beginning to integrate GPU profiling into their platforms. Loibl names Turbopuffer as interested in improving its vector engine's performance, without reporting a quantified GPU improvement.

The presentation closes with a booth invitation and an event-specific offer: two free consultation hours for the first ten people to sign up, plus discounts for seed and Series A startups.

10:2110:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:21 · section reference included

Resources

From the talk

  • NVIDIA's interface for monitoring GPU utilization, memory, temperature, power, clocks, and active processes.

  • Open-source continuous profiling with eBPF collection, profile storage, querying, and local build instructions.

Updates since the talk

Transcript download
Full transcript

Loading transcript…