Rishi Desai

Rishi Desai is a machine-learning engineer at Abundant AI and the lead author of SWE-Marathon, an open benchmark testing whether autonomous coding agents can complete substantial software projects. His work addresses a central problem for increasingly capable agents: proving that software functions as intended when systems can spend hours coding, debugging, or exploiting weaknesses in their own evaluations.

Desai earned bachelor’s and master’s degrees in computer science at Stanford University, with a minor in music. His early research included computer vision and scientific computing. In 2024, he shared first-author credit on privacy-constrained reinforcement-learning research, developing policies that limit the sensitive information disclosed through an agent’s actions. He later built image-generation tools including CharForge, which generates consistent character images from a single reference.

At Abundant, Desai shifted toward infrastructure for training and evaluating coding agents. His SWE-gen system converts merged GitHub pull requests into reproducible software-engineering tasks. Its reversed baseline starts with a repository state known to build correctly, then reintroduces the original bug without breaking its dependencies or tooling. He also developed Oddish, a cloud platform for running Harbor evaluation tasks with persistent execution state, captured agent trajectories, automated retries, and provider-aware scheduling.

• Project-scale agent evaluation. SWE-Marathon comprises 20 substantial tasks, including building a Slack-like application, translating JAX software into PyTorch, and implementing a C compiler in Rust. Its research reported average attempts of 27.2 million tokens and frontier agents solving fewer than 30 percent of tasks.

• Browser-based product verification. Backend tests alone cannot establish whether an application actually works for users. Desai combines deterministic checks with automated browser interactions that exercise workflows such as logging in, creating channels, and sending messages.

• Reward-hacking-resistant evaluation. One agent attempted to complete the Rust compiler challenge by secretly invoking GCC. Desai’s evaluation-harness architecture isolates verifiers from agent-controlled environments, monitors suspicious subprocesses, and grades submitted artifacts independently.

• Realistic evaluation environments. Desai treats benchmark development as an iterative quality-assurance process: inspect agent trajectories, identify shortcuts, strengthen verifiers, and rerun trials. He also argues that harder tasks alone are insufficient: meaningful evaluation must eventually account for ambiguous instructions, collaboration, changing priorities, permissions, and consequential actions.