Nachiket Paranjape develops machine-learning and AI evaluation infrastructure that gives product teams more control over model development and quality assessment. At AI Engineer World’s Fair 2026, he represented DoorDash’s AI Platform team as a software engineer contributing to its AI Evals initiative. With Swaroop Chitlur Haridas, he described how shared APIs, operator-built annotation tools and reviewable automated judges let product managers and operations teams turn their domain knowledge into improvements to AI products.
An early shopping prototype
In fall 2016, Paranjape was an electrical-engineering graduate student at Columbia University. With three teammates, he built an E Shopping Assistant that compared prices across stores, sent shopping lists to a selected seller and connected the shopping application to a Raspberry Pi checkout kiosk. RFID tags let the kiosk retrieve purchase information. The prototype brought software and physical interfaces together around a complete shopping workflow.
Building usable machine-learning platforms
At DoorDash, he contributed to the growth of Fabricator, a declarative feature-engineering framework. Practitioners describe features through configuration, while shared infrastructure handles their production, reducing the need to assemble each data-processing workflow manually.
By November 2023, Paranjape was leading ML Workbench on DoorDash’s Machine Learning Platform team. He co-authored an account of its development with product designer Archit Jha. The initiative began with a browser interface for testing models because manual testing had become difficult to scale. As adoption grew, users wanted to inspect models, predictions and features in the same workspace.
Paranjape and his collaborators used interviews with teams including Search, Ads and ETA to decide which workflows to build. Although their ambition covered the machine-learning lifecycle, user research led them to narrow the initial scope. The work combined infrastructure engineering with interface design: understanding how practitioners developed models helped the team choose which capabilities would make the platform useful.
In his public writing about AI quality, Paranjape argues that product managers and Strategy & Operations teams should be able to improve agent performance directly. His joint presentation with Haridas explains how DoorDash put cross-functional evaluation into practice. Strategy and operations teams establish priorities and the quality bar; product teams translate requirements into rubrics and workflows; operations teams annotate examples; engineers provide telemetry, datasets, APIs and automated judges.
Those responsibilities connect through a repeated evaluation loop. Teams capture agent traces and sessions, sample examples for review, annotate them with domain expertise and review the resulting judgments. The reviewed examples become golden datasets: reference cases used to measure performance and calibrate automated judges. Monitoring supplies further examples for the next round. This makes human review an ongoing input to quality assessment rather than a one-time labeling task.
Giving operators control over annotation and judges
Task-specific annotation tools: Different products need different annotation experiences, so a central platform team cannot efficiently build every interface itself. DoorDash’s stable APIs let operations partners use coding agents to build task-specific annotation tools while relying on common dataset and scoring capabilities. A restaurant-menu annotation interface illustrates the division of work: operators shape the review experience around the task they understand, and the platform team maintains the infrastructure underneath it.
Self-service judge calibration: The platform also makes self-service judge calibration available through a user interface. Product managers and operators can start with a judge prompt, establish baseline scores and run an optimization loop against golden datasets without repeated engineering handoffs. They can inspect the original and calibrated prompts before adopting the revised judge. That visibility helps them understand what changed, while flexible prompt ownership lets engineering, product or operations take responsibility according to the team’s needs.
Paranjape and Haridas reported lower per-annotation costs and faster iteration from these workflows, without supplying numerical results in the provided recording. The practical connection to ML Workbench is clear: both efforts make shared infrastructure useful by giving practitioners accessible ways to perform their own work. In evaluation, that means letting the people who understand a product’s quality requirements help define, test and improve the systems that judge it.
DoorDash’s GenAI platform moved evaluation beyond an engineering harness by giving domain experts stable APIs, task-specific annotation workflows, golden datasets, and a reviewable way to calibrate LLM judges.
Evaluation is cross-functional because engineering can provide traces, datasets, and judges, but domain experts must define and apply the product’s quality criteria.
Stable APIs let the platform team maintain shared capabilities while operators use coding agents to build annotation UIs suited to menus, images, manual tests, and other tasks.
Prompt ownership can remain flexible while teams learn; DoorDash has seen strategy and operations, product, and engineering each own it in different groups.
DoorDash reports lower per-annotation costs and faster iteration for thousands of rows each week, but the talk does not quantify the savings or speedup.
▶ Watch ↗