Asaf Gardin

Asaf Gardin is a software engineer whom AI21 identified as a senior software engineer for AI Engineer World’s Fair 2026. His work on Jamba models has led to upstream fixes for silent failures in vLLM, the inference engine used in AI21’s reinforcement-learning pipeline. His published investigations explain how a model can generate nonsense with high confidence, without a crash or warning—and how to trace that symptom to the code responsible.

Finding the request behind the tensors

While working on Jamba Reasoning 3B, Gardin investigated occasional gibberish in AI21’s reinforcement-learning pipeline. The same model checkpoints worked correctly in Hugging Face Transformers, suggesting a problem in execution rather than the model’s learned weights. The team built a diagnostic script that compared the engines’ log probabilities for identical token sequences: vLLM generated the tokens, and Transformers evaluated how likely those same tokens should have been. During failures, vLLM reported high confidence for tokens the reference implementation considered extremely unlikely.

His January 2026 debugging account preserves the investigation’s wrong turns. Constraining GPU memory made the failure reproducible, but CUDA memory checks found no out-of-bounds accesses. Forcing every request through the prefill path eliminated the gibberish, making the decode kernels look suspicious; inspecting their mathematics revealed no fault either.

The decisive step was carrying request IDs through vLLM’s forward pass. The scheduler knew which request it was processing, but the model layers received batches of tensors without that identity. Adding the IDs let Gardin stop execution when the failing request appeared and inspect how it had been classified.

A new request had received just one token of scheduler budget. The classification logic treated that single-token allocation as a continuing decode, even though the request had never been processed. Mamba’s decode path then loaded recurrent state left by a previous request in the reused cache slot. Because that state summarizes a sequence’s history and is updated recursively, the incorrect starting state contaminated subsequent generation. The kernels were computing correctly from an invalid starting point.

Gardin’s fix, merged on January 12, 2026, classified requests with no previously computed tokens as prefills regardless of their scheduled token count. This ensured they entered the initialization path. The contribution also added regression tests covering single-token new requests, including cases with multiple new requests.

Isolating an overflow inside the training stack

Gardin and Amir Koblyansky also contributed to an investigation of a separate 32-bit overflow in a Mamba CUDA kernel. During Jamba 3B training with GRPO, the log probabilities assigned during rollout generation diverged from those recomputed by the training model before any weight update. With the same weights and token sequences, those values should have been nearly identical. The mismatch left several possible causes: inference, training, or synchronization between them.

Changing the number of completions per prompt helped narrow the search. Increasing that count shifted the failures earlier, eventually producing a mismatch on the first rollout. That removed accumulated training history from the problem and allowed a standalone inference reproduction. Further experiments showed that reducing vLLM’s GPU memory allocation avoided the failure, while larger allocations exposed it. An attention-only model did not show the issue, directing the investigation toward Mamba’s state cache.

The cause was pointer arithmetic in the selective-scan CUDA kernel. Multiplying a cache index by a state stride used 32-bit arithmetic, which silently overflowed at sufficiently large cache allocations. In the reported setup, a stride of 89,600 elements caused offsets to wrap above roughly 47,935 cache slots. The kernel wrote state to incorrect locations while the intended slots remained zeroed.

Gardin’s upstream patch, merged on February 26, 2026, changed the index type from uint32_t to size_t, allowing the stride multiplication to use 64-bit arithmetic in that kernel.

Debugging methods made concrete

The two investigations show how Gardin and his collaborators turn intermittent failures into problems small enough to inspect:

• Compare confidence on the same outputs. Evaluating identical token sequences with a reference implementation exposed incorrect inference even when vLLM reported high confidence. Generating a different answer with the reference model would have made the comparison less direct.

• Vary conditions and watch the failure pattern. Low memory exposed stale-state reuse in the first investigation; large cache allocations exposed overflow in the second. Changes in failure timing with rollout count helped separate inference behavior from training dynamics.

• Inspect the assumptions around correct computation. Neither failure required incorrect model weights or faulty recurrence mathematics. One passed a new request into a path that assumed existing state; the other used an integer type too small to represent the required memory offset. Request tracking and controlled experiments made those assumptions visible.

AI21 announced Gardin and Yuval Belfer’s joint World’s Fair 2026 session, “Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story,” about these investigations. In his public announcement, Gardin described the symptom that connected them: confident gibberish without an exception, followed by weeks of investigation and fixes contributed back to vLLM.