Justin Muller is an applied AI architect who helps organizations turn generative AI prototypes into production systems they can evaluate, debug, and operate efficiently. At Amazon Web Services, where he was a principal applied AI architect in 2025, he developed a practical approach to generative AI evaluations that identifies specific failures instead of merely scoring model outputs.
Trained in physics, mathematics, law, and business, Muller worked in natural language processing before specializing in generative AI. His AWS customer work encompassed document processing, weather summaries, and structured business data. He also co-created an open-source Amazon Bedrock image-generation application with Sindhu Jambunathan.
His writing maps an expanding approach to production AI: prompt decomposition in 2024; the human design challenges of text-to-SQL in early 2025; and rubric-based evaluation without conventional gold-standard answers later that year.
- Evaluation as failure diagnosis. An overall accuracy score cannot explain which component failed or how to repair it. Muller favors explicit grading criteria, category-level error analysis, and rapid feedback, while examining both a model’s stated reasoning and the explanations generated by its evaluator.
- Prompt decomposition. Breaking large prompts into independently testable stages reveals where failures originate and allows engineers to choose the appropriate tool. For weather summaries, numerical comparisons belong in deterministic Python, not in a language model.
- Human-reviewed evaluation data. Synthetic answers are provisional until people verify them; otherwise, a model’s mistakes become embedded in the benchmark. Where reference answers are unnecessary, clearly specified rubrics can provide an alternative.
- Semantic routing and right-sized models. Directing simpler requests to smaller models and harder ones to more capable systems helps control accuracy, cost, and latency. Muller also applies established measures such as precision, recall, and F1 when they answer the evaluation question more reliably.
▶ Watch ↗