▶ Watch ↗19:50
Conference affiliation: Director, Data Science & AI · Air Canada · 2026
Nathanael Weill is Director, Data Science & AI at Air Canada, leading enterprise data science across the airline’s operations, corporate, commercial and loyalty businesses. His work addresses a practical obstacle in business AI: how engineers can improve a system when they cannot grade its answers themselves and the specialists who can have little time to help. His approach starts with a known answer, then constructs a problem that tests whether the system can recover it.
Nathanael holds a doctorate in pharmaceutical science. He worked as a senior IoT data scientist at Mnubo, directed data science and research at LANDR, and led data science at B-Yond. His work has spanned audio processing, recommendation systems and telecommunications diagnostics. Alongside his industry work, he is a program advisor and lecturer at McGill’s School of Continuing Studies, teaching predictive modeling and time series analysis.
In a 2020 account of his data science process, Nathanael drew lessons from unsuccessful projects: identify the actual user early, understand the existing workflow, and define success around the decision the system should improve. His maintenance example makes that distinction concrete. If a crew can inspect only a limited number of machines each day, a model should help choose those inspections. A better statistical score matters only insofar as it improves that constrained decision.
At B-Yond, Nathanael co-authored LLMcap with Łukasz Tulczyjew, Kinan Jarrah, Charles Abondo and Dina Bennett. The research applies masked language modeling to packet-capture representations: learning patterns in successful network exchanges helps the system identify failures without requiring labeled training examples. The work extends his telecommunications experience into learning from network traffic itself.
A telecommunications question-answering project in 2024 sharpened Nathanael’s approach to evaluation. His team was searching a large collection of technical and vendor documents, while access to experts allowed roughly one experimental comparison a week. Adding sophisticated retrieval techniques sometimes broke answers that had previously worked, and the slow review cycle made those regressions difficult to untangle.
The team adopted an existing technique: generate a question from a known page, ask the system to retrieve supporting material, and check with code whether it recovered that page. This gave them a repeatable retrieval test without asking an expert to judge every answer. They moved from one evaluation a week to dozens a day, removed complexity, and concentrated on embeddings, chunking and retrieval settings. Expert assessments improved over time. Nathanael credits the faster feedback loop, while remaining explicit that he cannot establish precisely why their solution succeeded where earlier teams’ attempts had not.
Nathanael joined Air Canada in June 2026. Supplier-contract review presented a related problem: specialists needed to identify troublesome clauses, find supporting guidance and propose amendments, but there was no public benchmark for the task. His team initially hoped to compare original contracts with negotiated versions and test whether an agent found the same issues. The necessary version pairs were unavailable, so they needed another way to construct cases with known answers.
In his account of testing agents without an existing benchmark, Nathanael explains several complementary choices:
Nathanael favors code-based checks where they fit because his team found that an LLM judge introduced substantial noise. Designing and validating the judge became another difficult engineering problem alongside the agent itself. He is willing to change a system’s design to make its behavior testable, even when that means moving away from the original approach.
These tests help developers improve a system; they do not establish that it is ready to ship. Expert feedback still matters. Nathanael uses it to improve the test set—adding cases, refining prompts and finding better ways to introduce defects—so the next round of development benefits from more than a one-time correction.
In his 2025 writing on agent design, Nathanael advocates separating retrieval, reflection and action. A customer-service assistant, for example, should gather the relevant messages, assess the complaint and check a proposed response before sending it. Separating those steps makes the business task explicit and creates opportunities to inspect the work before an action is taken. His evaluation work applies the same practical discipline: choose a decision the system must make, then find a way to tell whether it made it correctly.
▶ Watch ↗19:50
Loading more talks and highlights…