Aman Khan is an AI product leader at Google Cloud, working on the Gemini Enterprise Agent Platform and the evaluation, governance, and risk-management systems enterprises need to trust autonomous software. Previously head of product at Arize AI, he has built evaluation systems for self-driving cars, machine-learning products, and AI agents.
Khan began his career in engineering at Cruise before becoming a product manager for autonomous-vehicle evaluation systems. He then joined Spotify’s machine-learning platform organization, working on recommender systems, embeddings, and the Jukebox feature store. His product background also includes Apple and Zipline.
At Arize AI, Khan spent more than four years developing products for machine-learning observability, generative-AI evaluation, and agent development. He coauthored the launch of Alyx 2.0, a planning agent for AI development workflows, and contributed to work on hierarchical agent memory. He subsequently joined Google Cloud AI to work on its enterprise agent platform.
Evaluations as executable product requirements. Product managers should specify desired AI behavior through representative datasets, explicit grading criteria, and acceptance thresholds. Khan treats prompts, evaluation data, and user experience as shared responsibilities between product and engineering.
Evaluate the evaluator. Automated judges need their own quality controls: human-labeled examples, clearer grading instructions, lower-temperature model settings, repeated runs, and continued monitoring as production data changes. Direct code-based checks can replace model calls when requirements are straightforward.
Agent observability before optimization. A convincing answer can mask faulty routing, tool selection, or intermediate agent behavior. Khan’s multi-agent travel-planning demonstration shows how traces reveal interactions among research, budgeting, local-experience, and itinerary agents, enabling prompt changes to be tested against application data.
Evaluation datasets should evolve with failure modes. At Cruise, increasingly demanding scenarios progressed from straight roads to left turns and pedestrians. Khan applies that principle to AI agents: collect difficult production examples, revisit ambiguous cases, and continuously expand datasets as new weaknesses emerge.
Khan publishes AI Product Playbook and maintains personal-os, an open-source framework for an AI-powered personal operating system.
Scalable evaluation combines traces, human labels, model judgments and code checks, then uses that signal to improve applications, diagnose agent paths and refine the evaluators themselves.
▶ Watch ↗