Contents
  1. Numerical work and GPU execution
    1. What a kernel implements
    2. Assigning work to the device
    3. Turning points in GPU programming
  2. Data placement and cooperation
    1. Where operands live
    2. Indices become memory transactions
    3. Safe sharing and reuse
  3. Correctness before speed
    1. Specify the numerical result
    2. Test the supported domain
  4. Measurement and diagnosis
    1. Measure completed work
      1. Unchanged arithmetic, different overhead
    2. Bound arithmetic and traffic
    3. Turn observations into hypotheses
  5. Matrix computation
    1. Reuse operands through tiling
    2. Match tiles to instructions and shapes
    3. Overlap movement with computation
  6. Fusion and collective operations
    1. Keep useful intermediates close
    2. Combine partial results
    3. Normalize rows without excess traffic
      1. Stable softmax
      2. RMSNorm
  7. Attention as a kernel algorithm
    1. Carry normalization across score tiles
    2. Masks and work partitioning
      1. After reducing traffic
  8. Workload variation and implementation
    1. When indices choose the addresses
    2. Choose the level of control
      1. Specialization and tuning
    3. Establish the useful gain
  9. Check understanding
  10. Open questions
  11. Selected talks
  12. References
  13. Talk library
← All topics

GPU programming & kernels

GPU programming organizes numerical computations so many workers can execute them together. A kernel is a function launched on the GPU; optimizing it can reduce execution time or temporary memory without changing the intended computation. The challenge is to supply enough independent work while limiting data movement and coordinating shared results. A useful optimization must also preserve supported inputs, arithmetic requirements, and acceptable numerical differences. These principles help you choose libraries and compilers, decide when custom code is justified, and understand why an improvement for one workload can hurt another.

Numerical work and GPU execution

What a kernel implements

A graphics processing unit, or GPU, executes parallel numerical work. The host is the CPU-side program submitting that work; the GPU is the device. A kernel is a function launched for execution on the device.

A tensor is a multidimensional numerical array; its shape lists the dimension sizes. In a model, some arrays contain learned weights, numerical parameters established during training. Inference applies those parameters to inputs. Kernel programming concerns the execution of the resulting array operations.

A∈RM×K,B∈RK×N,Cij=∑k=0K−1AikBkj.A\in\mathbb{R}^{M\times K},\quad B\in\mathbb{R}^{K\times N},\quad C_{ij}=\sum_{k=0}^{K-1}A_{ik}B_{kj}. Matrix multiplication produces an MM-row, NN-column output. Each entry combines one row of AA with one column of BB. For example, [1,2,3][1,2,3] and [4,5,6][4,5,6] contribute 1⋅4+2⋅5+3⋅6=321\cdot4+2\cdot5+3\cdot6=32. Different output positions can be computed independently.

The operation specifies the result, not the launch count. A compiler can combine several operations into one kernel. Conversely, a large sum can use one kernel to produce partial sums and another to combine them. Identical mathematics can therefore involve different launches, temporary arrays, and memory transfers.

Assigning work to the device

A thread is one logical execution of kernel code. CUDA, NVIDIA's GPU programming platform, groups threads into cooperating thread blocks and blocks into a launch grid. Each block runs on one streaming multiprocessor (SM). Blocks with allocated execution resources are resident; others wait. Ordinary blocks cannot assume scheduling order.

CUDA groups threads into 32-thread warps. Single-instruction, multiple-thread (SIMT) execution applies shared instructions across participating lanes, the thread positions within a warp. When threads branch differently, lanes outside the executing path become inactive. Warp membership alone does not synchronize accesses to shared data.

For a simple output array, a logical thread might own one position. More elaborate kernels assign several values to each thread or an entire output region to a block. The mapping creates available work; it does not determine how much progresses at once. Latency hiding means issuing eligible work while other work waits for operands or earlier instructions. Charles Frye's GPU introduction explains why concurrency helps sustain throughput without making each individual operation complete sooner.

Execution-group vocabulary varies. AMD HIP calls its hardware group a wavefront and exposes its width through warpSize; portable code must not assume 32. OpenCL organizes work-items into workgroups and implementation-controlled subgroups. Their collective operations and progress guarantees require their own contracts.

Turning points in GPU programming

General computation on early programmable graphics hardware required expressing ordinary work through graphics interfaces, textures, and drawing operations. Stanford's Brook project addressed this mismatch with collections of records, kernels applied across them, and reductions. Its streams were data collections, distinct from the command queues called streams in CUDA.

DevelopmentDateContribution
BrookSIGGRAPH 2004Expressed stream computations without requiring graphics-oriented programming.
CUDAReleased 2007Exposed independently schedulable blocks containing cooperating threads.
RooflineApril 2009 publicationRelated attainable arithmetic throughput to data movement.
TritonJune 2019 paperMade multidimensional tiles a programming unit.
FlashAttention2022Reorganized exact attention around memory transfers.

CUDA's decomposition separated logical work from processor count: a runtime could distribute independent blocks across devices with different available parallelism. Nickolls, Buck, Garland, and Skadron's 2008 account describes shared memory and barriers as tools for cooperation inside those blocks. Later performance models, compilers, and specialized algorithms addressed different remaining problems. They continue to complement numerical libraries and explicit GPU programming.

Data placement and cooperation

Where operands live

Registers hold thread-private values. Shared memory is a programmer-managed, on-chip scratchpad accessible to a block. Global memory holds device allocations that persist across launches. Hardware-managed caches retain accessed data automatically. Register spilling places private values in device-backed local memory, adding traffic.

Capacity is how much fits; latency is how long an access takes; bandwidth is bytes transferred per second. These constraints are independent enough that fitting a model does not establish responsive execution. High-bandwidth memory, HBM, is one device-memory technology, not a name for every GPU storage space.

Keep reusable inputs in device allocations across calls. Host-to-device transfers belong in the complete-call cost when inputs arrive from host memory; retaining those inputs avoids repeating that transfer.

Indices become memory transactions

Memory layout maps logical indices to addresses. A stride is the element step along one dimension. Storage offset selects the starting element; element size converts these steps to bytes. Distinct tensor objects can share storage.

address⁡(i,j)=b+e (o+is0+js1).\operatorname{address}(i,j)=b+e\,(o+i s_0+j s_1). Here bb is the storage base address, ee the bytes per element, oo the storage offset, and s0,s1s_0,s_1 the strides. A contiguous row-major matrix with NN columns has strides (N,1)(N,1).

A transposed view changes this interpretation without moving values. The view is cheap to create, but its consumers encounter different address patterns. Materializing a contiguous representation copies a non-contiguous input. That conversion earns its cost only if subsequent use saves enough work; shared-storage mutations must also retain their intended meaning.

A memory transaction transfers an address segment. Coalescing lets a warp's requests share transactions. In CUDA's documented 32-byte model, aligned consecutive four-byte reads across 32 lanes need four transactions; scattered reads can need 32. Both request 128 useful bytes.

The same reads can touch more segments

32 lanes each read four bytes. Addresses are relative to a 32-byte-aligned base.

4 segments · 128 transaction bytes
128 useful bytes in every case.

Lane 0Byte 0Segment 0
Lane 1Byte 4Segment 0
Lane 2Byte 8Segment 0
Lane 3Byte 12Segment 0
Lane 4Byte 16Segment 0
Lane 5Byte 20Segment 0
Lane 6Byte 24Segment 0
Lane 7Byte 28Segment 0
Lane 8Byte 32Segment 1
Lane 9Byte 36Segment 1
Lane 10Byte 40Segment 1
Lane 11Byte 44Segment 1
Lane 12Byte 48Segment 1
Lane 13Byte 52Segment 1
Lane 14Byte 56Segment 1
Lane 15Byte 60Segment 1
Lane 16Byte 64Segment 2
Lane 17Byte 68Segment 2
Lane 18Byte 72Segment 2
Lane 19Byte 76Segment 2
Lane 20Byte 80Segment 2
Lane 21Byte 84Segment 2
Lane 22Byte 88Segment 2
Lane 23Byte 92Segment 2
Lane 24Byte 96Segment 3
Lane 25Byte 100Segment 3
Lane 26Byte 104Segment 3
Lane 27Byte 108Segment 3
Lane 28Byte 112Segment 3
Lane 29Byte 116Segment 3
Lane 30Byte 120Segment 3
Lane 31Byte 124Segment 3

Aligned stride 1: 4 segments. Offset 1: 5 segments. Stride 8: 32 segments.

Each of 32 lanes reads four bytes from a 32-byte-aligned base: address = base + 4 × (offset + lane × stride). Useful reads stay at 128 bytes. Aligned stride 1 touches four segments; offset 1 touches five; stride 8 touches 32. Controls change addresses. Modeled transaction bytes are not measured cache or HBM traffic, or latency.

Shared memory's banks serve accesses in parallel. Bank conflicts serialize distinct-address requests to one bank; same-address reads can broadcast. In CUDA's float-transpose example, successive words map to successive banks: a 32-word row stride sends column accesses to one bank, while padding to 33 advances the bank between rows. Architecture and access width affect this mapping.

Safe sharing and reuse

Unique output ownership makes parallel writes straightforward: each worker writes its own result. Cooperation adds dependencies. If workers fill a shared operand buffer, consumers must wait for those writes. Before overwriting that buffer, producers must also wait for the previous readers. Readiness for consumption and readiness for reuse are different conditions.

A barrier coordinates participating workers at a specified scope. Its participation rules matter: a block barrier reached by only some required threads is invalid. Warp-level sharing can need an earlier synchronization point even when a block barrier occurs later; a later barrier cannot repair a preceding race.

An atomic operation protects a read-modify-write update to its target. It does not automatically publish unrelated writes or establish that every producer has finished. CUDA's legacy atomics use relaxed ordering and specified thread scopes; newer interfaces allow explicit ordering. Protecting a counter and making its associated operand buffer ready are separate obligations.

Across launches, a CUDA stream orders commands. Different streams need explicit dependencies. An event recorded after a producer can make another stream wait; combining results from several streams requires completion dependencies covering every producer.

Correctness before speed

Specify the numerical result

A numerical contract states the operation, supported inputs, arithmetic settings, and acceptable differences. Floating-point formats represent a finite set of numbers, so arithmetic rounds. Consequently, changing addition order can change a result. Cancellation, subtracting nearly equal quantities, can expose information already lost to rounding. A fused multiply-add evaluates a product plus an addend with one rounding, rather than rounding both stages separately.

For a small arithmetic illustration, use binary FP32 with round-to-nearest, ties-to-even, and let a = 2²⁴.
EvaluationIntermediate resultFinal result
(a + 1) − aa + 1 rounds to a0
a + (1 − a)1 − a is exactly representable1

FP32 is 32-bit single-precision floating point; FP16 is 16-bit half precision. BF16, Brain Floating Point, is also 16-bit but allocates more bits to exponent range and fewer to fraction precision than FP16. Overflow exceeds a format's finite range; infinity can subsequently produce NaN, meaning not-a-number. Storage and intermediate arithmetic formats can differ.

Matching input dtype therefore does not fully specify a comparison. PyTorch documents matrix implementations that sometimes reduce intermediate accumulation precision, and an attention math backend that upcasts FP16/BF16 inputs for intermediate computation. Record these settings alongside the input and output types. Higher intermediate precision can require more memory and execution time.

∣a−r∣≤atol+rtol ∣r∣.|a-r|\leq \mathrm{atol}+\mathrm{rtol}\,|r|. For actual value aa and reference rr, absolute tolerance governs agreement near zero; relative tolerance scales with reference magnitude. Choose tolerances from the operation's requirements. In PyTorch's assert_close, infinities must match and matching NaNs requires explicit permission. Numerical closeness differs from bitwise determinism.

Quantization represents values using restricted numerical levels, with rounding and potentially clipping. It can change storage and executable arithmetic; its calibration methods and quality tradeoffs belong in Quantization.

Test the supported domain

Apply behavioral testing to the declared kernel contract. Compare against an independently understandable reference, using higher precision when it helps distinguish arithmetic error from implementation disagreement. For a model port, comparing corresponding intermediate outputs can locate the first divergence instead of leaving only a final-output mismatch.

Derive cases from the promises the implementation makes. A mask selects positions a memory operation may access: masked loads substitute a specified value at invalid positions, while masked stores skip those positions.
Contract conditionUseful check
Dimensions need not divide tile sizesExercise partial tiles; verify masked loads and valid stores.
Several layouts are supportedCompare each supported stride pattern with the reference.
Inputs or outputs may share storageExercise permitted aliases and verify required input preservation.
A bounded numerical domain is supportedInclude cancellation, near-zero outputs, and large or small permitted values.
Exceptional values are acceptedCheck their explicit comparison and output policy.

Compute Sanitizer separately checks memory access, initialization, shared-memory races, and synchronization. These tools complement output comparison: a numerically correct run can still contain unsafe accesses. Shared-memory race checking does not establish the absence of every global-memory race. Repeated execution can expose intermittent faults without proving their absence.

Validate the configuration that will actually be timed. Changing tile size, arithmetic settings, or generated code after validation changes the candidate. Compilation and successful execution are preliminary gates; a speed comparison is useful only after both implementations satisfy the intended result.

Measurement and diagnosis

Measure completed work

Kernel launches return before device work finishes. A host timer around submission can therefore miss execution. Device events timestamp progress through a stream; wait for the stop event before reading elapsed time. A complete-call host measurement must likewise include completion, while recognizing that extra synchronization changes execution.

Name the measured boundary: kernel execution, a sequence of launches, or the complete call including transfers and allocation. Warmup moves lazy initialization outside a steady-state measurement. Repeat measurements and retain variation, rather than relying on one timing. PyTorch's benchmark utilities provide warmup and accelerator synchronization, but the benchmark statement still determines which costs are included.

Submission ends before execution

Example timings

A submission timer can exclude most device work.

Completed call0–12 microsecondsDuration 12 microseconds
Host submission0–2 microsecondsDuration 2 microsecondsWithin Completed call
Device execution1–11 microsecondsDuration 10 microsecondsWithin Completed call
Host wait and completion2–12 microsecondsDuration 10 microsecondsWithin Completed call
The completed-call interval contains submission, device execution, and host waiting. Device-event endpoints bracket execution. Waiting overlaps device work; summing these spans does not give elapsed time.
Read the diagram as text
  • Completed call. Host measurement includes completion. 0 to 12 microseconds; duration 12 microseconds.
  • Host submission. Launch returns before device completion. 0 to 2 microseconds; duration 2 microseconds. Parent: Completed call.
  • Device execution. Device-event interval: 1 to 11 microseconds. 1 to 11 microseconds; duration 10 microseconds. Parent: Completed call.
  • Host wait and completion. 2 to 12 microseconds; duration 10 microseconds. Parent: Completed call.

Repeatedly reading one small buffer can create a different cache regime from the intended workload. Baseline and candidate order can also affect cache state. Record working-set reuse, competing device work, setup exclusions, and synchronization boundaries as part of the evaluation protocol. A warm measurement and a cold-start measurement answer different questions.

CUDA Graphs prepare an operation graph for repeated submission. Capture records work, instantiation prepares an executable graph, and replay launches it. This reduces recurring host setup without removing the device computations; capture restrictions and setup costs still matter.

Unchanged arithmetic, different overhead

Alan Gray's September 2019 experiment kept a 500,000-element scaling kernel unchanged on Tesla V100, CUDA 10.1, with 512 threads per block. Across 1,000 groups of 20 launches, submission and synchronization changed the cost:
Measured boundary or execution methodMicroseconds per kernel
Device execution alone2.9
Synchronize after every kernel9.6
Synchronize after each group3.8
Graph replay, including amortized setup3.4

The graph's roughly 400-microsecond setup was paid once. These historical repeated-buffer results illustrate submission costs, not universal launch latency or cold-cache performance. The arithmetic stayed fixed; the way the host coordinated it changed.

Bound arithmetic and traffic

Arithmetic intensity is operations per byte crossing a named memory boundary. A Roofline bound combines that intensity with bandwidth and arithmetic capacity. Williams, Waterman, and Patterson's 2009 account counted bytes reaching main memory after cache filtering, making locality part of the performance model.

I=FD,t≥max⁡ ⁣(FPcompute,DB),P≤min⁡(Pcompute,BI).I=\frac{F}{D},\qquad t\geq\max\!\left(\frac{F}{P_{\mathrm{compute}}},\frac{D}{B}\right),\qquad P\leq\min(P_{\mathrm{compute}},BI). Here FF is the operation count, DD transferred bytes, BB bytes per second, and PcomputeP_{\mathrm{compute}} arithmetic operations per second at the chosen precision. Dividing FF by each minimum execution time gives the throughput bound.

For example, scaling FP32 values performs one multiplication per four-byte read and four-byte write: an ideal 0.125 operations per byte. Matrix multiplication conventionally counts a multiply-add as two operations. For ordinary multiplication (β=0\beta=0), reading each FP16 operand once and writing each FP16 output once, without reading an existing output matrix, gives ideal intensity MNK/(MK+KN+MN)MNK/(MK+KN+MN). These are traffic estimates, excluding extra accesses and cache effects.

A bandwidth ceiling does not establish bandwidth saturation. Too little parallel work, dispatch delays, and inefficient accesses can leave execution below both bounds. Use the model to narrow possibilities, then investigate actual execution.

Reuse changes the bandwidth bound

Example

Execution can remain below both resource ceilings.

Fixed precision and memory boundary

The bandwidth slope meets the arithmetic ceiling at intensity 8.

03691202.557.510Arithmetic intensity (FLOP/byte)Throughput (TFLOP/s)Roofline boundStarting pointImproved executionHigher intensityBandwidth boundArithmetic ceiling
  • 1. Roofline bound
  • 2. Starting point
  • 3. Improved execution
  • 4. Higher intensity
Read coordinates and regions as data

X: 0–12 FLOP/byte; Y: 0–10 TFLOP/s, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Roofline bound (polyline)

(0, 0); (8, 8); (12, 8)

Starting point (points)

(2, 1)

Improved execution (polyline)

(2, 1); (2, 2)

Higher intensity (polyline)

(2, 1); (6, 1)

Bandwidth bound: (4.4, 5.4)

Arithmetic ceiling: (9.5, 8.6)

Illustrative FP32 limits: 8 TFLOP/s and 1 TB/s at device memory. Horizontal movement changes intensity; vertical movement changes achieved throughput. Arrows are hypothetical changes, not measurements.

Turn observations into hypotheses

Start with the complete execution timeline. Nsight Systems separates CPU-side CUDA calls from GPU kernels and memory operations. Locate consequential kernels, transfers, and gaps before inspecting individual instructions. A gap identifies missing activity, not its cause.

Occupancy is resident active warps per SM divided by the supported maximum. Registers, shared memory, and block size constrain residency. An eligible warp is ready to issue; useful arithmetic utilization concerns productive execution. Higher occupancy need not improve throughput.

Use counters to propose a discriminating change, following the same logic as testing explanations of a failure.
ObservationPossible explanationControlled change
Many transactions per useful loadLane mapping scatters addressesRemap accesses while preserving arithmetic.
Register-spill trafficToo much live private stateReduce tile size or fusion scope.
Few output blocksToo little independent workTry smaller output tiles or split the reduction.

Nsight Compute's short-scoreboard stalls can involve shared memory, special math, or branching; they do not uniquely diagnose bank conflicts. A not-selected warp was eligible while another issued. Profiling can replay kernels, serialize launches, change clocks, and flush caches. Confirm a promising change under ordinary timing conditions as well as the profiler.

Matrix computation

Reuse operands through tiling

Tiling, or blocking, partitions computation into reusable pieces. The March 1990 Level 3 BLAS paper organized Basic Linear Algebra Subprograms around matrix-matrix operations. Its motivation was locality: vector-at-a-time algorithms moved too much data, while blocks allowed repeated calculation on resident operands. Blocking predates this paper; the contribution included reusable numerical interfaces suited to hierarchical memory.

GEMM, general matrix multiplication, computes C=αAB+βCC=\alpha AB+\beta C; ordinary multiplication uses α=1,β=0\alpha=1,\beta=0. A block retains one output tile's partial sums while traversing KK chunks. Shared operand tiles let each loaded AA value serve multiple columns and each BB value multiple rows.

For an interior bM×bNb_M\times b_N output tile and reduction chunk of length bKb_K: Fchunk=2bMbNbK,Eloads=bMbK+bKbN.F_{\mathrm{chunk}}=2b_Mb_Nb_K,\qquad E_{\mathrm{loads}}=b_Mb_K+b_Kb_N. These count arithmetic operations and operand elements loaded once, respectively; output traffic, boundary waste, and inter-block cache reuse are excluded.

Illustrative pseudocode

Python-like pseudocode
# Conceptual tile program; not executable CUDA.
acc = zeros(output_tile_shape, dtype=float32)
for k_chunk in reduction_chunks:
    a = load_a_tile(k_chunk, invalid_value=0)
    b = load_b_tile(k_chunk, invalid_value=0)
    acc += tile_matmul(a, b)
store_valid_output_positions(acc)

The Triton matrix tutorial uses FP32 accumulators, zero-filled reduction-edge loads, and masked output stores. In an explicitly shared-memory implementation, loading must finish before consumption and consumption before buffer reuse. Larger tiles improve potential reuse but demand more live storage and leave fewer independent output blocks.

One shared-memory implementation: operand buffers change with each K chunk while private partial outputs persist. The drawing shows storage roles rather than an exact tile size or lane assignment.

Match tiles to instructions and shapes

A matrix multiply-accumulate instruction computes a small matrix product plus an accumulator. NVIDIA announced Volta Tensor Cores on May 10, 2017. Its documented hardware operation used 4×4 matrices with FP16 multiplicands and FP16 or FP32 accumulation, while CUDA exposed larger warp-level matrix operations. Application matrices, programmer-visible fragments, and hardware operations are distinct sizes.

WMMA, warp matrix multiply-accumulate, is one collective interface: all warp threads participate with matching parameters. Its fragments distribute elements using an unspecified mapping. Supported shapes, types, layouts, and alignment are constrained; CUDA 13.0 documents 256-bit load alignment. These requirements do not describe every matrix-instruction API.

Tile quantization means work falls into discrete output tiles; wave quantization means the final scheduling wave is partly occupied. In a fixed 256×128-tile A100 example, M=2304 and K=4096, changing N from 1536 to 1544 raises tile count from 108 to 117. With one block per SM on 108 SMs, nine blocks occupy an extra wave. This illustrates a fixed decomposition, not every library's selection.

Resource limits create another tradeoff. Larger tiles can require enough registers to spill or enough shared memory to prevent launch. A matrix-vector or narrow-output product also exposes less output parallelism than a broad matrix product. Select a decomposition for the actual shape; neither the largest tile nor maximum occupancy is an objective by itself.

Overlap movement with computation

Software pipelining overlaps movement for a future tile with computation on a ready tile. Double buffering supplies separate storage for those roles. An asynchronous copy followed immediately by a wait creates no overlap. The first load primes the pipeline; the final computation drains it.

A stage is a buffer slot. Acquire it, submit and commit its copy, wait before reading, then release after reading. Occupied stages block acquisition; more stages consume shared memory.

A buffer must be ready to read—and free to refill

Example timings

Two buffers overlap a future load with computation while preserving both consumption and reuse dependencies.

One buffer0–15 time unitsDuration 15 time units
Load 0 · buffer 00–2 time unitsDuration 2 time unitsWithin One buffer
Compute 02–5 time unitsDuration 3 time unitsWithin One buffer
Load 1 · buffer 05–7 time unitsDuration 2 time unitsWithin One buffer
Compute 17–10 time unitsDuration 3 time unitsWithin One buffer
Load 2 · buffer 010–12 time unitsDuration 2 time unitsWithin One buffer
Compute 212–15 time unitsDuration 3 time unitsWithin One buffer
Two buffers0–11 time unitsDuration 11 time units
Load 0 · buffer 00–2 time unitsDuration 2 time unitsWithin Two buffers
Compute 02–5 time unitsDuration 3 time unitsWithin Two buffers
Load 1 · buffer 12–4 time unitsDuration 2 time unitsWithin Two buffers
Compute 15–8 time unitsDuration 3 time unitsWithin Two buffers
Load 2 · buffer 05–7 time unitsDuration 2 time unitsWithin Two buffers
Compute 28–11 time unitsDuration 3 time unitsWithin Two buffers
Loads take 2 time units and computations take 3 in this example. Load completion permits reading; consumer completion permits reuse. Two buffers shorten the span from 15 to 11. Tile 2 waits for both its load and the preceding accumulator update.
Read the diagram as text
  • One buffer. Each load waits until the preceding computation releases buffer 0. 0 to 15 time units; duration 15 time units.
  • Load 0 · buffer 0. 0 to 2 time units; duration 2 time units. Parent: One buffer.
  • Compute 0. 2 to 5 time units; duration 3 time units. Parent: One buffer.
  • Load 1 · buffer 0. 5 to 7 time units; duration 2 time units. Parent: One buffer.
  • Compute 1. 7 to 10 time units; duration 3 time units. Parent: One buffer.
  • Load 2 · buffer 0. 10 to 12 time units; duration 2 time units. Parent: One buffer.
  • Compute 2. 12 to 15 time units; duration 3 time units. Parent: One buffer.
  • Two buffers. Prime with tile 0's load, overlap later loads with computation, then drain with tile 2's computation. 0 to 11 time units; duration 11 time units.
  • Load 0 · buffer 0. Priming: the first computation needs this completed load. 0 to 2 time units; duration 2 time units. Parent: Two buffers.
  • Compute 0. Releases buffer 0 at time 5. 2 to 5 time units; duration 3 time units. Parent: Two buffers.
  • Load 1 · buffer 1. Overlaps computation on buffer 0. 2 to 4 time units; duration 2 time units. Parent: Two buffers.
  • Compute 1. Its operands are ready at 4; the preceding accumulator update finishes at 5. 5 to 8 time units; duration 3 time units. Parent: Two buffers.
  • Load 2 · buffer 0. Acquires buffer 0 only after compute 0 releases it. 5 to 7 time units; duration 2 time units. Parent: Two buffers.
  • Compute 2. Draining: the final computation has no subsequent load to overlap. 8 to 11 time units; duration 3 time units. Parent: Two buffers.

Fusion and collective operations

Keep useful intermediates close

Kernel fusion implements several operations in one kernel execution. Consider y=2sin⁡(x)y=2^{\sin(x)}. Separate kernels write the sine result to device memory and then read it for exponentiation. A fused kernel can retain that intermediate and store only the final result. For equally sized arrays, the simple element-transfer count falls from four per output to two, before considering caching or spills. This is the mechanism illustrated in Joe Fioti's Luminal talk.

A matrix epilogue applies postprocessing to accumulated outputs before their final storage. For example, cuBLASLt offers bias followed by ReLU, the elementwise function max⁡(x,0)\max(x,0). Its documented bias vector matches output rows and broadcasts across columns: for our matrix convention, Yij=max⁡(Cij+bi,0)Y_{ij}=\max(C_{ij}+b_i,0). Other conventions require reconciling that orientation. Auxiliary modes can retain extra activation outputs when callers need them.

Fusion must preserve both dependencies and efficient work assignment. Thread-local intermediates can remain in registers; block-level sharing may require shared memory and coordination. Incompatible layouts, different preferred parallelizations, or excessive live state can make the fused kernel slower. FusionStitching evaluates saved launches and traffic against execution and resource costs, sometimes generating and timing a candidate because a simple traffic model is insufficient.

Required consumers also constrain removal: an intermediate needed elsewhere must remain available or be recomputed. Fusion can change rounding boundaries; applying an activation before converting an accumulator to FP16 differs from applying it after FP16 storage. Validate that change against the numerical contract.

Fusion differs from buffer reuse. Reuse assigns storage to a later value after the earlier value's last use; the values remain distinct. Whole-graph lifetime analysis can establish that opportunity even when the operations remain separate kernels.

Combine partial results

A reduction combines many values into fewer values using an operation such as sum or maximum. Its neutral element leaves a result unchanged: zero for addition, negative infinity for a maximum over finite scores. Invalid lanes can contribute these identities instead of accessing invalid input positions.

A hierarchical reduction first combines several elements within each thread. Warp-level exchange combines thread partials; one partial per warp can then enter shared memory for block-wide combination. Larger arrays can produce one partial per block, followed by a combining kernel. The reduction tree determines communication and floating-point addition order.

A warp shuffle exchanges register values; it is not a memory barrier. Cross-block combining needs launch ordering or explicit producer dependencies, not assumptions about block scheduling.

Split-K assigns different portions of an output tile's dot products to separate blocks. CUTLASS combines their stored partials in a second kernel, gaining parallel work at the cost of workspace and combination.

Atomic accumulation offers another combination path, but competing updates can contend and their floating-point order can vary. Atomicity protects updates without fixing summation order. Choose between staged and atomic approaches using the destination pattern, workspace allowance, launch costs, and reproducibility requirements.

Warp aggregation combines a group's counter increments before one elected thread updates the counter. Andy Adinets's 2014 filtering article illustrates reduced contention; its 2017 update explains that CUDA 9 automated this transformation in many cases. The example concerns one shared counter, not arbitrary scatter destinations, and its output order differs from an order-preserving prefix-sum filter.

Normalize rows without excess traffic

Stable softmax

Softmax turns scores into normalized exponential weights. For a nonempty finite row, subtracting its maximum keeps exponential inputs nonpositive without changing the mathematical result. The maximum and denominator are reductions; outputs depend on both.

m=max⁡jxj,zj=exj−m,ℓ=∑jzj,pj=zj/ℓ.m=\max_j x_j,\qquad z_j=e^{x_j-m},\qquad \ell=\sum_j z_j,\qquad p_j=z_j/\ell. Subtracting mm multiplies every original exponential by the same factor, which cancels during normalization.

The Triton fused-softmax tutorial retains row intermediates on chip, reading and writing each element once. Computational padding uses negative-infinity loads and masked stores. Longer rows can exceed this resource budget, requiring rereading or partitioning; row count also determines how much independent work exists. The finite-row formula alone does not define fully masked or arbitrary non-finite inputs.

RMSNorm

Root mean square normalization, or RMSNorm, uses a different statistic. Zhang and Sennrich's 2019 RMSNorm paper investigated reducing normalization overhead by omitting mean subtraction. Unlike LayerNorm, RMSNorm does not center the vector. Replacing one with the other changes the operation, rather than merely optimizing its implementation.

yi=xiγiϵ+1n∑j=0n−1xj2.y_i=\frac{x_i\gamma_i}{\sqrt{\epsilon+\frac{1}{n}\sum_{j=0}^{n-1}x_j^2}}. For a row of length nn, γi\gamma_i is a learned elementwise gain and ϵ\epsilon is a stabilizing constant inside the square root. The reduction axes and epsilon are part of the contract; PyTorch's RMSNorm API specifies trailing normalized dimensions.

Computing the normalized output is the forward pass. Training also needs a backward pass, which computes how the loss depends on inputs and learned gains. Differentiating through the reductions creates its own storage and combination requirements. How fitting changes predictions explains how these derivatives guide learning.

Attention as a kernel algorithm

Carry normalization across score tiles

Attention compares queries with keys to produce scores, then uses normalized weights to combine associated values. These are numerical representations; queries, keys, and values and value mixtures explain their model role. Here the implementation problem is avoiding complete score and probability arrays between those operations.

Milakov and Gimelshein's 2018 Online Normalizer Calculation for Softmax maintains a running maximum and exponential sum. A later score can increase that maximum. Earlier contributions must then be rescaled into the new normalization frame; independently normalized pieces cannot simply be averaged.

Given previous maximum mm, sum ℓ\ell, and a new finite score tile ss: m′=max⁡(m,max⁡jsj),a=em−m′,wj=esj−m′,ℓ′=aℓ+∑jwj.m'=\max(m,\max_j s_j),\quad a=e^{m-m'},\quad w_j=e^{s_j-m'},\quad \ell'=a\ell+\sum_j w_j. Multiplying the old sum by aa changes its reference maximum without discarding earlier scores.

For two single-score tiles, 0 then ln(2), the first state is m=0, ℓ=1. The second raises m to ln(2), so a=1/2 and ℓ becomes 1/2+1=3/2. The final weights are 1/3 and 2/3. Online normalization saves an input pass; standalone softmax still needs to produce every output.

FlashAttention, by Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré in 2022, combines tiled attention with normalization state to avoid full pairwise intermediates.

u′=au+∑jwjvj,o=u′/ℓ′.u'=a u+\sum_j w_jv_j,\qquad o=u'/\ell'. Original FlashAttention uses normalized output state; uu instead accumulates unnormalized weighted value vectors vjv_j. Hold a query tile and traverse key/value tiles: compute each score tile, update m,ℓ,um,\ell,u, then discard the scores. Rescale both sums together; normalize after the final contributing tile. State need not stay on chip.
Discard each score tile after updating the running maximum, denominator and weighted-value sum. This uses the chapter’s unnormalized-u formulation; state can move between memory levels. Normalize after the last contributing tile.

Masks and work partitioning

A causal mask excludes positions a query may not see; attention visibility explains the model semantics. Unequal query and key lengths require an alignment rule. FlashAttention 2.1 documents bottom-right alignment, differing from 2.0. With five queries and two keys, its permitted-key rows are:

QueryKey 0Key 1
0MaskedMasked
1MaskedMasked
2MaskedMasked
3AllowedMasked
4AllowedAllowed

That API specifies zero outputs for the first three rows. An ordinary maximum-subtraction formula would encounter negative infinity minus negative infinity. The implementation guards this case and zero-denominator normalization. A streaming implementation must likewise preserve an empty state until valid contributions arrive. These guards do not define arbitrary NaN or positive-infinity behavior.

For sequence length NN and vector width dd per attention head, dense attention retains O(N2d)O(N^2d) arithmetic. Original FlashAttention needs linear additional memory at fixed dd, but can write running state between iterations. Exact attention permits floating-point differences.

After reducing traffic

FlashAttention-2, introduced by Tri Dao in July 2023, addressed remaining work-partitioning costs. Independent query-row tiles provide more blocks when batch size and head count are small. Assigning different query rows to warps avoids combining their partial outputs through shared memory, and delaying normalization reduces non-matrix arithmetic. The paper credits earlier Triton work for related loop reordering and sequence parallelization.

FlashAttention-3 in 2024 assigned loading to producer warps and computation to consumer warps, a division called warp specialization. It also overlapped matrix operations with softmax. To measure each technique's contribution, the paper removed them separately in an FP16 experiment using H100 80GB SXM5, non-causal attention, batch 4, sequence length 8448, 16 heads, and head dimension 128:

Attention implementationReported time
Both overlap techniques3.538 ms
Without matrix–softmax pipelining4.021 ms
Without warp specialization4.105 ms

A three-stage variant performed worse than two stages: compiler scheduling failed to create the intended overlap, while additional live intermediates increased register requirements and forced smaller tiles. These attention measurements show why adding pipeline machinery does not establish useful concurrency.

Workload variation and implementation

When indices choose the addresses

A gather, or lookup, retrieves values selected by integer indices. An embedding lookup selects rows of a representation table; embedding representations explains their meaning. Unlike dense matrix traversal, row addresses depend on input values. Distant row selections can still have coalesced accesses within each row.

Memory-level parallelism is the number of memory operations in flight. cuEmbed uses multiple row loads, vectorized accesses, and prefetched indices to increase it. Repeated popular rows can also hit caches. Requested-byte throughput may therefore exceed physical HBM bandwidth without transferring that many HBM bytes. Index distribution, row width, batch size, and rows selected per sample all belong in the workload description.

A scatter assignment writes to indexed destinations. It does not necessarily combine duplicate destinations: PyTorch's scatter contract permits nondeterministic selection of one value when indices repeat. Scatter-add instead accumulates every contribution, although floating-point accumulation order can still vary.

Repeated destinations can create contention even when dimensions remain unchanged. In Adinets's historical filtering experiment on K80 with CUDA 8.0.61 and 100×2²⁰ integers, raising the passing fraction from 5% to 50% reduced ordinary atomic-filter bandwidth from about 55 to 8 GiB/s. The bandwidth counted input/output bytes, excluding atomics. This isolates a counter-heavy pattern, not a universal scatter performance law.

Reordering can improve grouping only if identities survive. cuEmbed's backward transformation sorts table indices while carrying sample identifiers and optional weights; compressed results retain an inverse mapping to table rows. Its hotness is rows selected per sample, so variable hotness can also create uneven work. Include sorting, workspace, and restoration in the complete cost. There is no general sorting crossover independent of reuse and index distribution.

Choose the level of control

Choose an implementation layer according to the decision that needs changing. A tuned library can supply an excellent matrix primitive while a compiler improves the surrounding operation sequence. Custom code becomes useful when a consequential workload requirement or execution opportunity remains unserved.

LayerProgrammer specifiesImplementation supplies
Numerical libraryOperation, shapes, layout, arithmetic requirementsTuned implementations behind an interface
Graph compilerOperations and dependenciesLayout decisions, fusion, and generated execution
Tile-oriented kernelTile operations and work per program instanceIntra-tile mapping and compilation
Thread-oriented kernelThread ownership and explicit cooperationCompilation to supported device instructions

Tillet, Kung, and Cox's 2019 Triton paper addressed operations underserved by vendor libraries. It placed multidimensional tiles between high-level numerical expressions and explicit thread coordination. Programmers still chose tile work; compiler passes handled intra-tile parallelization and details such as coalescing and synchronization. This abstraction reduces some implementation obligations without removing shape and resource decisions.

Sometimes the useful change stays above the kernel language. Natalie Serrino's kernel-generation talk describes expressing an average-pooling workload through a better-optimized convolution primitive on Metal. Its correctness depends on the intended pooling configuration. The same talk reports a custom matrix multiplication that lost to its baseline. More low-level code does not itself establish a better execution plan.

Specialization and tuning

Specialization selects an implementation under assumptions about inputs or hardware. A guard checks whether those assumptions hold. PyTorch documents recompilation when shapes or guarded scalar values change; dynamic-shape handling can reduce some recompilations, and cached variants remain reusable when their guards pass. Splitting a graph can reduce recompilation scope while imposing execution costs. A guard establishes applicability, not custom-kernel correctness.

Autotuning measures configurations to select one for designated keys. Triton's autotuner supports pruning and cached timings, but evaluates candidates through repeated execution. In-place updates therefore need reset or restoration during tuning. Search covers supplied configurations, not every possible implementation. Treat correctness as a separate gate and count tuning time before claiming a gain.

Compiler search can explore implementation structure as well as numerical parameters. Luminal represents alternative expressions compactly in an e-graph, a structure grouping equivalent expressions, then profiles candidate implementations. Rewrites need not improve performance individually. This makes measured search useful, but leaves search cost, numerical validation, and the target workload central to the decision.

Functional portability means an implementation runs correctly on another target; performance portability means it also runs efficiently there. Compiler backends, dependent libraries, and device-specific assumptions affect both. Generated candidates still need independent validation and hardware measurements. Distribution must identify supported hardware and software versions so consumers can reproduce those conditions.

Establish the useful gain

Compare against a credible library or compiled baseline using identical shapes, layouts, arithmetic requirements, and timing boundaries. Sweep the important workload range rather than selecting its most favorable point. Sequence lengths and batch sizes can change which implementations work well; a single shape does not represent an entire model workload.

Gene Amdahl's 1967 argument emphasized work left unchanged by parallel acceleration, including sequential data management. The same issue limits kernel improvements: accelerating one component leaves the remaining workload to execute.

S=1(1−f)+f/s.S=\frac{1}{(1-f)+f/s}. For fixed, nonoverlapping work, ff is the original time fraction accelerated and ss its speedup. The remaining costs stay unchanged. If f=0.4f=0.4 and s=2s=2, total time becomes 0.8 of the baseline, giving S=1.25S=1.25. Overlapping execution requires critical-path analysis instead of adding durations.

Setup must also be repaid. If specialization adds setup cost CC and saves Δt>0\Delta t>0 per call, it produces a net saving after more than C/ΔtC/\Delta t comparable calls, assuming those savings persist. Include compilation, tuning, conversion, workspace management, and additional synchronization wherever they occur in the real execution boundary.

Report timing variation and consequential regressions alongside the gain. Then remeasure the calling sequence: another kernel, a transfer, or host coordination may now dominate. Request-level responsiveness and throughput require a representative serving experiment. Adopt the implementation whose supported workload coverage and useful time saved justify its setup and maintenance obligations.

Open questions

  1. Predicting profitable fusion remains difficult because saved traffic competes with layout conflicts, longer live ranges, and compiler scheduling. Better models would reject unpromising candidates cheaply while retaining gains verified on complete workloads.

  2. Specialization must balance recurring execution savings against tuning cost and changing inputs. Progress would mean reliable selection across shape distributions with bounded setup costs, rather than a growing collection of individually impressive special cases.

  3. Reordering indexed accesses has no workload-independent payoff. Sorting and identity restoration cost time and workspace, while benefits depend on locality and repeated destinations. Useful progress would establish complete-call crossover measurements under fixed dimensions and varied index distributions.

  4. Automated kernel generation needs validation strong enough to survive unfamiliar shapes, arithmetic settings, and execution schedules. Independent references and hardware checks matter because a generator can reproduce its own mistaken assumptions. Progress would combine broader correctness coverage with measured gains against strong, versioned baselines.

Follow the curated reading path through the speakers and demonstrations behind this entry.

172 min

AI Engineer World's Fair 2024 · 2024

Low Level Technicals of LLMs

Daniel Han

Cited in this entry

Explore numerical debugging through intermediate-output comparisons, and the extra derivative work hidden behind compact forward normalization formulas.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

18 matching talks

TalkSpeakerEventYear
Ben BurtenshawAI Engineer Europe 20262026
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Daniel HanAI Engineer World's Fair 20262026
Sayash KapoorAI Engineer Summit 20252025
Ishan AnandAI Engineer World's Fair 20242024
Daniel SzokeAI Engineer Europe 20262026
Max RyabininAI Engineer Europe 20262026
Philip Kiely, Pankaj GuptaAI Engineer World's Fair 20242024
Philipp KrennAI Engineer World's Fair 20252025
Olive SongEvent unknownUnknown
Tejas BhaktaAI Engineer World's Fair 20262026
Jetashree RaviAI Engineer World's Fair 20262026
Abdul DakkakAI Engineer World's Fair 20262026
Simran AroraAI Engineer World's Fair 20262026
John OusterhoutEvent unknownUnknown
Daniel KimAI Engineer World's Fair 20262026
Filip MakraduliAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
13 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
9 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Roofline Performance Model — NERSC

    https://docs.nersc.gov/tools/performance/roofline/

  2. Triton: Matrix Multiplication

    https://triton-lang.org/main/getting-started/tutorials/03-matrix-multiplication.html

  3. A Set of Level 3 Basic Linear Algebra Subprograms

    https://hammarling.com/sven/pubs/Level3BLAS-1-TOMS16-90.pdf

  4. CUDA Programming Guide: Programming Model

    https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/programming-model.html

  5. PyTorch: torch.Storage

    https://docs.pytorch.org/docs/main/storage.html

  6. Matrix Multiplication Background User's Guide

    https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html

  7. OpenXLA: GPU Architecture Overview

    https://openxla.org/xla/gpu_architecture

  8. Faster Parallel Reductions on Kepler

    https://developer.nvidia.com/blog/faster-parallel-reductions-kepler/

  9. CUDA C++ Best Practices Guide

    https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

  10. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=527

  11. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=291

  12. HIP C++ Language Extensions

    https://rocm.docs.amd.com/projects/HIP/en/develop/how-to/hip_cpp_language_extensions.html

  13. Khronos: cl_khr_subgroups

    https://registry.khronos.org/OpenCL/specs/unified/refpages/man/html/cl_khr_subgroups.html

  14. Brook for GPUs: Stream Computing on Graphics Hardware

    https://graphics.stanford.edu/papers/brookgpu/brookgpu.pdf

  15. Scalable Parallel Programming with CUDA

    https://saulius-grazulis.lt/~saulius/paskaitos/VU/kompiuteri%C5%B3-architekt%C5%ABra/skaidr%C4%97s/bibliography/PDF/2008_Nickolls_40.pdf

  16. Roofline: An Insightful Visual Performance Model for Multicore Architectures

    https://teaching.wence.uk/comp52315/lecture-slides/2020-21/williams2009-roofline.pdf

  17. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations

    https://www.eecs.harvard.edu/~htk/publication/2019-mapl-tillet-kung-cox.pdf

  18. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    https://arxiv.org/abs/2205.14135

  19. CUDA Programming Guide: Writing SIMT Kernels

    https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html

  20. GPU Performance Background User's Guide — NVIDIA

    https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html

  21. Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

    https://www.youtube.com/watch?v=c5-kx2bwoCk&t=513

  22. Nsight Systems User Guide

    https://docs.nvidia.com/nsight-systems/UserGuide/index.html

  23. PyTorch: Tensor Views

    https://docs.pytorch.org/docs/main/tensor_view.html

  24. CUDA Programming Guide: Register Spilling and Shared-Memory Banks

    https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html

  25. CUDA C++ Programming Guide 13.0.3: Multi-Stage Asynchronous Data Copies

    https://docs.nvidia.com/cuda/archive/13.0.3/cuda-c-programming-guide/index.html

  26. NVIDIA Compute Sanitizer

    https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html

  27. CUDA Programming Guide: C/C++ Language Extensions

    https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html

  28. CUDA C++ Programming Guide 13.0: Streams, Synchronization, and Events

    https://docs.nvidia.com/cuda/archive/13.0.0/cuda-c-programming-guide/index.html

  29. NVIDIA: Floating Point and IEEE 754

    https://docs.nvidia.com/cuda/floating-point/index.html

  30. NVIDIA TensorRT: Accuracy Considerations

    https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html

  31. PyTorch: Numerical Accuracy

    https://docs.pytorch.org/docs/main/notes/numerical_accuracy.html

  32. PyTorch: torch.testing

    https://docs.pytorch.org/docs/main/testing.html

  33. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    https://arxiv.org/html/1712.05877

  34. Low Level Technicals of LLMs

    https://www.youtube.com/watch?v=pRM_P6UfdIc&t=7547

  35. AI Kernel Generation: What's Working, What's Not, What's Next

    https://www.youtube.com/watch?v=6guQG_tGt0o&t=276

  36. PyTorch: Benchmark Utils

    https://docs.pytorch.org/docs/main/benchmark_utils.html

  37. AI Kernel Generation: What's Working, What's Not, What's Next

    https://www.youtube.com/watch?v=6guQG_tGt0o&t=341

  38. CUDA Graphs — NVIDIA CUDA Programming Guide

    https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html

  39. Getting Started with CUDA Graphs

    https://developer.nvidia.com/blog/cuda-graphs/

  40. Nsight Compute Profiling Guide

    https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html

  41. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    https://tridao.me/publications/flash2/flash2.pdf

  42. CUTLASS: Efficient GEMM in CUDA

    https://docs.nvidia.com/cutlass/latest/media/docs/cpp/efficient_gemm.html

  43. NVIDIA Launches Revolutionary Volta GPU Platform, Fueling Next Era of AI and High Performance Computing

    https://nvidianews.nvidia.com/news/nvidia-launches-revolutionary-volta-gpu-platform-fueling-next-era-of-ai-and-high-performance-computing

  44. NVIDIA Tesla V100 GPU Architecture

    https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/volta-architecture-whitepaper.pdf

  45. CUDA C++ Programming Guide 13.0: Warp Matrix and Shuffle Functions

    https://docs.nvidia.com/cuda/archive/13.0.0/cuda-c-programming-guide/index.html

  46. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=867

  47. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    https://www.youtube.com/watch?v=0uj9lMI-sIo&t=892

  48. cuBLAS: cublasLtEpilogue_t

    https://docs.nvidia.com/cuda/cublas/index.html

  49. FusionStitching: Boosting Execution Efficiency of Memory Intensive Computations for DL Workloads

    https://arxiv.org/pdf/1911.11576

  50. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    https://www.youtube.com/watch?v=0uj9lMI-sIo&t=1106

  51. Triton: Fused Softmax

    https://triton-lang.org/main/getting-started/tutorials/02-fused-softmax.html

  52. PyTorch: torch.Tensor.scatter_add_

    https://docs.pytorch.org/docs/2.9/generated/torch.Tensor.scatter_add_.html

  53. CUDA Pro Tip: Optimized Filtering with Warp-Aggregated Atomics

    https://developer.nvidia.com/blog/cuda-pro-tip-optimized-filtering-warp-aggregated-atomics/

  54. Root Mean Square Layer Normalization

    https://papers.nips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf

  55. PyTorch: RMSNorm

    https://docs.pytorch.org/docs/main/generated/torch.nn.modules.normalization.RMSNorm.html

  56. Low Level Technicals of LLMs

    https://www.youtube.com/watch?v=pRM_P6UfdIc&t=4073

  57. Online Normalizer Calculation for Softmax

    https://arxiv.org/pdf/1805.02867

  58. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    https://arxiv.org/pdf/2205.14135

  59. FlashAttention README: Causal-Mask and Small-Query Behavior

    https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/README.md

  60. FlashAttention CUDA Softmax Implementation

    https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/csrc/flash_attn/src/softmax.h

  61. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

    https://proceedings.neurips.cc/paper_files/paper/2024/file/7ede97c3e082c6df10a8d6103a2eebd2-Paper-Conference.pdf

  62. Accelerating Embedding Lookups with cuEmbed

    https://developer.nvidia.com/blog/accelerating-embedding-lookups-with-cuembed/

  63. PyTorch: torch.Tensor.scatter_

    https://docs.pytorch.org/docs/2.13/generated/torch.Tensor.scatter_.html

  64. cuEmbed: Embedding Operations and Index Transformations

    https://github.com/NVIDIA/cuEmbed/blob/main/cuembed/README.md

  65. Special topics in Kernels, RL, Reward Hacking in Agents

    https://www.youtube.com/watch?v=uIiA6DquRiE&t=6377

  66. AI Kernel Generation: What's Working, What's Not, What's Next

    https://www.youtube.com/watch?v=6guQG_tGt0o&t=462

  67. AI Kernel Generation: What's Working, What's Not, What's Next

    https://www.youtube.com/watch?v=6guQG_tGt0o&t=580

  68. PyTorch: Dealing with Recompilations

    https://docs.pytorch.org/docs/main/user_guide/torch_compiler/compile/programming_model.recompilation.html

  69. Triton: triton.autotune

    https://triton-lang.org/main/python-api/generated/triton.autotune.html

  70. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    https://www.youtube.com/watch?v=0uj9lMI-sIo&t=798

  71. Low Level Technicals of LLMs

    https://www.youtube.com/watch?v=pRM_P6UfdIc&t=6932

  72. Your Coding Agent Should Do AI System Engineering

    https://www.youtube.com/watch?v=JomVvNDjGb8&t=209

  73. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=496

  74. Vector Search Benchmark[eting]

    https://www.youtube.com/watch?v=YrUBFXa1KUY&t=162

  75. Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities

    https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf

  76. Vector Search Benchmark[eting]

    https://www.youtube.com/watch?v=YrUBFXa1KUY&t=766

  77. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    https://www.youtube.com/watch?v=0uj9lMI-sIo&t=1184

  78. Low Level Technicals of LLMs

    https://www.youtube.com/watch?v=pRM_P6UfdIc&t=9251

  79. Why Rust is the Ideal Language for Vibe-Coding

    https://www.youtube.com/watch?v=ugUeZ8-b-u0&t=351

  80. What every AI engineer needs to know about GPUs

    https://www.youtube.com/watch?v=y-UGrYbJsJk&t=306