AI Engineer World's Fair 2025
Maximize GPU Efficiency with Continuous Profiling for GPUs
Loading slides…
Read the talk
Finding Where GPU Time Goes with Continuous Profiling
Matthias Loibl walks from sampled CPU stacks to GPU telemetry and GPU-time profiles, showing how to investigate idle accelerators and trace device work back to its callers.
From a talk by Matthias Loibl
Before you start: Familiarity with function call stacks, CPU–GPU workloads, and basic Linux deployment concepts will help you follow the profiling examples.
Where is the machine spending its time?
Where does a program spend its time—and which work could you remove or make cheaper? Profiling supplies the measurements needed to answer that question. Matthias Loibl tentatively traces its history to IBM work in the 1970s, but the practical questions remain familiar: how much memory does a program use, where does CPU or GPU time go, which instructions execute, and how frequently and for how long do functions run?
Those measurements connect performance engineering to infrastructure cost. Loibl offers a hypothetical: a 10% software performance improvement might let an operator turn off 10% of its servers. The point is the possibility of reducing the capacity needed for a workload, not a guaranteed conversion from faster software to a smaller bill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trade event completeness for continuous observation
A profiler can record every event, or periodically observe what is happening. Recording everything provides detail, but collection cost and data volume make that approach difficult to sustain continuously.
| Approach | What it records | Tradeoff |
|---|---|---|
| Tracing | Every event | Detailed history; high cost and data volume |
| Sampling | Observations at selected intervals | Lower collection volume; individual events can be missed |
Loibl's sampling example uses a ten-second window with an adjustable rate, such as 20 or 100 samples per second. The rate controls how often the profiler looks, rather than how often the application performs work.
Loibl associates 100 samples per second with less than 1% CPU overhead and cites approximately 4 MB of memory overhead. He does not supply the workload, hardware, or measurement conditions for those figures. The broader tradeoff is clear: sampling will miss events, but collecting continuously gives recurring work more opportunities to appear. A stack that executes only once matters less to this view than work that repeatedly consumes resources.
Each observation captures a call stack: the chain of functions calling one another at that moment. For CPU profiling, that means recording the stack executing on the CPU; for allocation profiling, it can mean recording the stack responsible for allocating memory. Repeated observations build a picture of where the application spends its resources without requiring a complete event history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Observe the production environment
The useful profile is the one from the environment running the workload. A developer's machine does not reproduce production, so low overhead matters partly because it makes production observation practical. Loibl's collector uses Linux eBPF, drawing on kernel facilities to gather profiles. He describes starting one collector to profile applications without changing or individually instrumenting them.
Loibl introduces himself as the Berlin-based director of Polar Signals Cloud and a maintainer of Prometheus, Prometheus Operator, and Parca. He presents Parca as the open-source counterpart to the profiling work, before turning from established CPU and memory collection to the GPU preview.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use GPU metrics to choose the investigation
After three or four years working on CPU and memory profiling, the team began a GPU profiling preview earlier in the year of the talk. The first view uses the NVIDIA Management Library, NVML, to collect GPU metrics. In the demonstration, the blue utilization line represents the overall node, while the orange line represents one GPU process, identified by its process ID. Memory utilization and clock speed appear alongside those signals. A utilization drop identifies a time interval worth investigating: what stopped the GPU from staying busy?
The next metrics help distinguish possible constraints:
- Power: Compare measured power usage with the dashed power-limit line.
- Temperature: Look for heat that could lead to throttling. Loibl warns about sustained temperatures around 80°C; that is an approximate warning, not a threshold shared by every GPU.
- PCIe throughput: Investigate whether transfers between CPU and GPU are limiting the workload. In the displayed convention, negative values represent receiving and positive values represent sending; Loibl points to an example of sending 10 MB/s.
These signals narrow the question from “Why is utilization low?” to a more specific investigation of power, cooling, or data movement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find what the CPU does while the GPU waits
GPU telemetry becomes more useful when it shares a time axis with stored CPU profiles. The collector already captures CPU stacks with eBPF, so an interval of interest in the GPU metrics can lead directly into host-side execution.
- Find a time interval worth investigating in the metrics.
- Inspect the corresponding CPU activity. Loibl selects activity on CPU zero near the end of the example.
- Drag over that time range to open a flame chart.
- Follow the call stack to see what the CPU was executing while the GPU was not fully utilized.
In the selected profile, Python eventually calls into CUDA functions. The chart supplies the calling context that the utilization curve alone cannot show.
A common explanation is that the CPU is busy loading data and fails to keep the GPU supplied with work. The host can therefore be active while the accelerator remains underutilized. This is why looking only at GPU utilization leaves an important part of the workload unexplained.
The same investigation is not limited to Python. Loibl names Rust calling CUDA, compiled-language stacks generally, and runtime support including Python, Ruby, and the JVM. The underlying profiling approach also applies beyond accelerator workloads—to web servers, databases, and vector databases whose performance depends on where execution time goes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attribute GPU execution time to its callers
CPU profiles explain host activity, but they do not directly answer how much GPU time a calling function causes. Loibl next introduces GPU-time profiling, a new preview demonstrated that morning, distinct from the earlier GPU metrics preview. The small example shows branches for matmul_naive and generate_random_matrix: matrix multiplication and random-matrix generation become identifiable work in a profile.
Loibl describes obtaining a start timestamp when a CUDA call stack appears on the CPU and a termination timestamp for the GPU kernel, then using the interval to determine the kernel's GPU duration. Once matching device-execution timestamps are available, the duration calculation is straightforward:
python
def gpu_duration_ns(start_ns: int, end_ns: int) -> int:
if end_ns < start_ns:
raise ValueError("Kernel end precedes kernel start")
return end_ns - start_ns
The difficult part is obtaining and matching the right events. CUDA launches are asynchronous, and the talk does not specify the launch-to-completion correlation mechanism; timing only a host launch call would not establish device execution time.
The resulting view attributes GPU time to individual functions. Loibl then moves from the small CUDA example to a larger profile, following a Python main function down through the calling stack into libcuda. The host call path provides the context for the device work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read widths, leaves, and colors separately
The larger chart prompts a useful question: does it show CPU time, GPU time, or both? Its calling path comes from the CPU, while its width represents attributed GPU time. Loibl explains that each stack's leaf is the function consuming time on the GPU. The chart connects the host's chain of calls to device execution rather than simply combining two unrelated timing measurements.
A second question concerns the colors. They distinguish binaries, not timing categories. Blue identifies Python in this example; Loibl tentatively identifies one of the lower binaries as CUDA.
| Chart feature | Meaning |
|---|---|
| Width | Attributed GPU time |
| Path to the leaf | CPU calling stack |
| Leaf | Function consuming GPU time |
| Color | Binary associated with the frame |
Keeping those meanings separate lets the reader follow which application code led to device work without mistaking a color for a measure of expense.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy the collector
The deployment described in the talk starts with a Linux host and eBPF. The collector can run as a binary managed by systemd, in Docker, or as a Kubernetes DaemonSet. For Kubernetes, Loibl describes obtaining the manifest YAML, supplying a token, and deploying it. This is the talk's collector setup: the later CUDA profiling release additionally documents a GPU-process injection library and agent configuration, so the preview's setup should not be treated as a complete current CUDA installation procedure.
Existing CPU and memory profiling customers were beginning to integrate GPU profiling into their platforms. Loibl names Turbopuffer as interested in improving its vector engine's performance, without reporting a quantified GPU improvement.
The presentation closes with a booth invitation and an event-specific offer: two free consultation hours for the first ten people to sign up, plus discounts for seed and Series A startups.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
NVIDIA's interface for monitoring GPU utilization, memory, temperature, power, clocks, and active processes.
Open-source continuous profiling with eBPF collection, profile storage, querying, and local build instructions.
Further reading
Loibl and Vincent's conference companion covers GPU metrics, per-process utilization, and a preview of GPU-time flame graphs.
Versioned guidance for inference performance measurement, including GPU clocks, cooling, and thermal throttling.
Updates since the talk
The later CUDA profiler release explains how CUPTI execution records connect GPU durations to CPU launch stacks through USDT probes and eBPF.
Full transcript
Loading transcript…