Yuval Belfer

Yuval Belfer is a developer advocate whom AI21 identified as its Senior Developer Advocate in its World’s Fair 2026 announcement. His public technical presentations connect retrieval design with the reliability of model execution. In “Two Bugs That Hid in Plain Sight,” co-presented with Asaf Gardin, he frames a useful diagnostic distinction: an incorrect answer can reflect an engineering failure rather than a limitation to address through better training. High-confidence gibberish without a crash or warning warrants investigation of the system executing the model.

Belfer makes that distinction concrete through AI21’s Jamba models, which combine attention and Mamba state-space layers. He introduces a failure that was rare, appeared only after substantial workload, and was observed in vLLM rather than the other inference frameworks the team tried. Those conditions made an isolated prompt an inadequate reproduction: request history, scheduling and cache reuse were part of the problem. Gardin then explains the detailed investigation, including generating a continuation with vLLM and scoring that exact prompt-plus-continuation sequence with Hugging Face Transformers. Comparing corresponding token log probabilities holds the sequence fixed, exposing execution discrepancies without confusing them with independently sampled answers. Joint presentation, Gardin’s technical account.

The investigation also illustrates why architecture matters to debugging. Under constrained memory, a fresh request could receive only one scheduled token and be mistaken for a continuing decode. Mamba’s decode path reads recurrent state before updating it, so a reused cache slot could introduce another request’s history into the new generation. In the attention path described, current keys and values are written before reading, and sequence-length masking bounds valid positions. Carrying request IDs through vLLM’s forward context enabled a targeted breakpoint that exposed the misclassification. The team’s correction treated zero previously computed tokens as the signal for prefill—the path that initializes a new request—regardless of its scheduled token count. These are findings Belfer helped present; the supplied sources attribute the detailed debugging and linked contribution to Gardin. Technical explanation, merged correction and regression tests.

A second case gives developers a complementary lesson: changing resources can conceal a defect as well as expose one. Rollout-generation probabilities diverged from those recomputed by the FSDP training model before any weight update. Increasing completions per prompt moved periodic failures into the first rollout, allowing the investigation to shed accumulated training history and isolate inference. Larger state-cache allocations exposed unsigned 32-bit cache-index-by-stride multiplication wrapping around; smaller allocations avoided the offending offsets. Changing the index type to size_t supplied 64-bit arithmetic in the reported kernel. AI21 credits Gardin and Amir Koblyansky with this investigation. Belfer’s co-presentation makes its diagnostic value accessible: watch how a parameter changes the timing and reachability of failure, then identify which subsystem could produce that pattern. Joint presentation, credited overflow investigation.

Belfer also addresses retrieval choices. AI21 announced his 2026 session “Stop Chunking Like It’s 2022” around the premise that no single fixed chunk size suits every query. Its description proposed a simple multi-index approach and reported potential recall improvements of up to 37%, without retraining or additional LLM overhead. The supplied announcement establishes that proposed approach and company-reported result, rather than a complete experimental account. Together with his inference-reliability presentation, it shows his emphasis on examining implementation choices that quietly shape an AI application’s results. AI21’s session announcement.