Hamel Husain
Conference affiliation: Parlance Labs · 2025
Related work and projects
Hamel Husain is a machine-learning engineer, independent consultant, and educator who leads Parlance Labs. His defining work is domain-specific AI evaluation: turning failures in real products and the judgment of domain experts into tests that help teams decide what to fix. His current focus is bringing data science back to AI—helping teams debug, analyze, and measure language-model applications.
On this page
From applied models to developer infrastructure
Hamel began working on credit-risk models at Washington Mutual in 2003, followed by telecommunications analytics at Accenture. From 2011 to 2015, he was vice president of applied analytics at AlixPartners, leading data-science work across consulting engagements. He joined DataRobot in 2015, combining modeling with product design and data workflows, then moved to Airbnb in 2016. There, he built customer-lifetime-value models used across marketing, customer service, risk, and finance, as well as image-classification models for listing photos. He also developed data pipelines and reusable infrastructure that helped other data scientists iterate. His career history spans both applied modeling and the systems that make it useful to others.
At GitHub, where he worked as a staff machine-learning engineer from October 2017 to January 2022, Hamel led the launch of CodeSearchNet. The team was developing models for finding code through natural-language queries but lacked a shared way to measure progress. GitHub, Microsoft Research contributors, and partners at Weights & Biases collaborated on a corpus, baseline models, and an evaluation environment.
The corpus paired functions with documentation across six programming languages. For evaluation, programmers and other technical reviewers rated how well candidate results answered search queries. Code paired with documentation supplied training material; human judgments of relevance provided a separate test of search quality. Hamel led the launch within a collaborative research effort that included Marc Brockschmidt, Miltos Allamanis, Ho-Hsiang Wu, and Tiferet Gazit.
Alongside his GitHub work, Hamel was a core contributor and maintainer in the fast.ai ecosystem from 2019 to 2022. His contributions included nbdev, fastcore, ghapi, and continuous-integration, testing, and build tools. He also helped develop fastpages, which used GitHub Actions to publish notebooks, Markdown, and Word documents as blog posts. Notebook authors could share explanations, code outputs, and interactive visualizations without reconstructing their work in another publishing format. The project was later deprecated in favor of Quarto.
In 2022, Hamel headed machine learning and data science at Outerbounds, working on tooling around Metaflow. He left to explore commercial opportunities with fast.ai and nbdev, a notebook-enabled software development environment he helped maintain, becoming an entrepreneur in residence from August 2022 to May 2023. Commercializing it raised questions that enthusiasm for the software could not settle. Adoption friction, doubts about a hosted business, and competing priorities complicated the opportunity, while clients increasingly wanted help with language models. Hamel set aside nbdev commercialization and shifted toward that consulting work. His work at Parlance Labs began in May 2023.
Making evaluation part of product development
Hamel’s work with Rechat on Lucy, its assistant for real-estate agents, shows what his evaluation approach changes. Rechat could improve a prototype through prompt changes and informal checks, but a change that helped one use case could break another. A handful of successful interactions did not reveal the application’s success or failure rate. Hamel joined as an outside consultant to help the team move beyond those checks.
Several complementary decisions shaped the evaluation system:
- Turn observed failures into inexpensive tests. Rechat wrote assertions for mistakes such as invalid placeholders and repeated details. Existing continuous-integration workflows ran the tests, and Metabase tracked results. Synthetic requests representing real-estate agents expanded coverage across features and scenarios. The tests made regressions visible without requiring an LLM judge for every failure.
- Put the context needed for review in one place. Rechat used LangSmith to log traces, but reviewing them still involved too much friction. A custom interface brought interactions, domain-specific metadata, filtering, and labeling controls together. Reviewers could inspect a case without hunting for information elsewhere. The same workflow helped curate successful examples and work through failures for fine-tuning data.
- Calibrate automated judges against an expert. Hamel’s work on an evaluator for Honeycomb shows why a plausible automated critique is insufficient. A domain expert identified a missing trace-grouping requirement in an example answer. The expert’s critiques helped shape the judge, while examining concrete cases also clarified the expert’s own criteria. The work involved discovering what a good answer required and teaching the evaluator to recognize it. Hamel recommends iterating against expert-labeled examples before using a judge’s scores to guide product decisions.
His advice about tools preserves a practical tradeoff. He recommends adopting trace-logging tools early, using existing infrastructure for tests and reporting, and building a small review interface when generic tools make inspection cumbersome. Teams may outgrow continuous integration as an evaluation runner; understanding their process gives them a basis for choosing the next tool. Generic scores should not replace criteria tied to the application’s actual failures.
Hamel also treats evaluation difficulty as a product-design signal. Generating an entire lesson plan, for example, can leave a teacher with a large, unfamiliar document to check. His alternative design starts with vetted lesson plans and uses AI to adapt them, making the changes visible. That proposal reduces the amount of new material a user must inspect and makes review more focused. Evaluation can therefore change what the product produces, rather than merely score its output afterward.
Engineering skills and domain participation
Hamel treats evaluation and data literacy as core engineering skills, including for developers who access models through APIs. Rapid prototyping can give way to stagnation when teams cannot measure whether their changes help. He and Jason Liu advocate deliberate practice in evaluation and data analysis so application builders can develop those missing skills.
His hiring advice follows product maturity: begin with application development, add platform and data engineering to capture usage data, and introduce specialized machine-learning expertise once there is meaningful data to work with. He challenges catch-all job descriptions that make one AI engineer responsible for every problem, comparing them with the earlier expectation that a data scientist would combine software engineering, statistics, mathematics, and domain expertise in one role.
Participation matters beyond the engineering team. Hamel describes a mental-health consulting engagement where calling prompt work agent development made domain experts unsure how to contribute. His joint critique of AI strategy with Greg Ceccarelli connects jargon with inaccessible data, generic metrics, and tool purchases made before a problem is understood. The consulting example gives that criticism a concrete consequence: technical language can exclude the people whose knowledge is needed to evaluate the system.
Teaching teams to improve independently
With Shreya Shankar, Hamel co-teaches AI Evals for Engineers & PMs. He is also a co-author of Evals for AI Engineers, covering error analysis, LLM-based judges, synthetic data, production monitoring, and feedback loops.
Parlance Labs makes client independence an explicit consulting goal. Its engineers work alongside teams to transfer evaluation methods and establish improvement processes they can continue after the consultants leave. The firm rejects long-term maintenance dependency as an incentive that can work against that goal.
Read the topics behind these talks
3 conference talks
▶ Watch ↗AI Engineer World's Fair 202418:45
How to construct domain-specific LLM evaluation systems.
Explore the talk →
▶ Watch ↗AI Engineer World's Fair 202435:21
What We Learned From A Year of Building With LLMs
Explore the talk →Loading more talks and highlights…
References
- ai.engineer/talks/how-to-build-an-ai-strategy-that-fails ↗
- ai.engineer/talks/how-to-construct-domain-specific-llm-evaluation-systems ↗
- ai.engineer/talks/what-we-learned-from-a-year-of-building-with-llms ↗
- huggingface.co/hamel ↗
- hamel.dev/blog/posts/evals-faq/ ↗
- hamel.dev ↗
- arxiv.org/abs/1909.09436 ↗
- ai.engineer/summit/2025 ↗
- ai.engineer/worldsfair/2024 ↗
