David Karam is an AI product leader at Microsoft and the co-founder and former chief executive of Pi Labs, an AI evaluation and optimization company acquired by Microsoft. He previously spent a decade at Google, becoming a product director working on Google Search’s artificial-intelligence and natural-language-understanding systems.
Karam founded Pi Labs with fellow Google Search veteran Achint Srivastava to bring sophisticated search-quality techniques into everyday AI development. The company built tools for application-specific scoring, synthetic data, prompt optimization, customized retrieval, and feedback-driven improvement. After the acquisition, Karam and the Pi Labs team joined Microsoft.
Complexity-adjusted impact. Karam approaches AI quality empirically: assemble representative queries, establish a baseline, investigate failures, and introduce only improvements whose benefits justify their cost. His framework for layered retrieval progresses from documents placed directly in context to BM25, vector search, cross-encoder reranking, and specialized embeddings as specific failures demand.
Application-specific scoring systems. Broad questions about whether an answer is useful become more reliable when divided into inspectable signals. A meeting-summary evaluator might separately assess document structure, action items, and factual coverage, then calibrate their combined score against human judgment and user feedback. Karam treats evaluation as an expanding development loop that helps teams compare models, refine prompts, and catch regressions; his evaluation workshop demonstrates the approach through Python, spreadsheets, synthetic examples, and real feedback data.
Measuring subjective AI behavior. Even qualities such as an assistant’s personality can be decomposed into narrower, more interpretable dimensions and calibrated against preference data. Karam describes this approach as a way to improve precision and diagnostic clarity while reducing reward hacking.
Ranking beyond semantic relevance. Useful results also depend on price, merchant characteristics, popularity, document relationships, and individual preferences. Ambiguous requests may require explicitly orchestrated searches across multiple backends because language models cannot reliably infer every retrieval system’s constraints.
Graceful degradation. When engineering cannot eliminate uncertainty, Karam adapts the product experience: show a decisive interface when understanding is strong, offer broader choices when confidence weakens, and involve a person when automation cannot resolve the situation reliably.
Turn application requirements into inspectable scoring signals, calibrate them against user feedback, and reuse the resulting scorer for model comparisons, prompt changes, and response selection.
A CRM agent’s launch bar provides a practical way to choose RAG techniques: inspect failing queries, repair the relevant layer, and add complexity only when the product needs it.
▶ Watch ↗