AI Engineer World's Fair 2026
Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AI
Slide chapters
21 slide and demo groups · Timestamps are approximate. 3 additional groups were omitted from this pass.
Read the talk
Small Models, Big Results: Training a Finance Agent for Under $500
Charles Dickens explains how verified financial data and reinforcement learning helped a 4-billion-parameter agent outperform its 235-billion-parameter sibling—and why disciplined tool use mattered more than harder training examples or elaborate rewards.
From a talk by Charles Dickens
At a glance
Ideas worth remembering
On 290 held-out financial questions, the trained 4B model reached almost 60% accuracy versus 51% for its 235B sibling. The advantage is specific to this simulated financial task.
Schema hallucination, context flooding, and repeated failed strategies make tool discipline a concrete training target.
The reported training run cost about $420 for compute and $40 for judging. A binary final-answer reward outperformed the more elaborate reward designs tested.
Single-table training produced the greatest lift among the tested data mixtures and transferred to questions involving two to five tables, while general tool-calling performance held up.
Enterprise evaluations should match the intended environment complexity, autonomy horizon, and output complexity.
The financial task behind the model comparison
A financial agent can know plenty about finance and still fail at the job. It must work with the tables and APIs actually available, finish a workflow without compounding mistakes, and leave behavior that someone can audit. Charles Dickens, a research scientist at Snorkel AI, frames those requirements as the practical meaning of the question, “How well does the model or the agent reason?”
Snorkel and UC Berkeley’s Sky Computing Lab tested whether specialization could meet those requirements more effectively than parameter scale. Their trained 4-billion-parameter Qwen model reached almost 60% on financial questions, compared with 51% for its 235-billion-parameter variant. This is a result in a simulated financial environment, rather than evidence that the smaller model is generally more capable.
The project sits within Snorkel’s open research and benchmarking work, alongside its work on frontier-model data and enterprise deployments. Its starting judgment is deliberately practical: a specialist that knows the forms, tools, and rules may be the right choice for a narrow workflow. As Dickens puts it, you would not call Terence Tao for a tax audit. The experiment asks whether training can give a small model that kind of operational competence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn annual filings into questions with checkable answers
Snorkel’s FinQA data begins with SEC 10-K annual reports pulled from EDGAR. These filings supply the financial material analysts use to identify risks and inform investment decisions. A model converts that material into approximately 6,900 SQL tables; each table then supplies one generated question-answer pair.
Generation follows a question taxonomy developed with financial experts who work with these documents. That taxonomy gives the model context about the kinds of questions to produce. The generated examples also carry metadata, including lineage, table names, and columns. Those details make it possible to check whether a question refers to the data that was actually extracted.
Verification has three distinct jobs:
- Programmatic consistency checks: Use the metadata to catch invented table or column references and check the example’s connection to its source data.
- Independent agent reviews: Add automated review beyond the original generation step.
- Expert manual review: Have humans assess whether the questions and answers are realistic and verified.
The resulting tasks require planning, tool calling, and reasoning, but finish with a single verifiable answer. That choice matters later: training can reward a final outcome without having to assign a separate score to every intermediate action. The reported splits contain 4,000 training examples, 500 validation examples, and 290 held-out benchmark examples. Companies in the benchmark do not overlap with the other splits, so evaluation tests work on different companies rather than another question about a company already represented in training.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A plausible query can still derail the workflow
The benchmark exposed recurring failures even in frontier models. These failures concern how an agent uses its environment:
- Schema hallucination: Assume a table or column exists, potentially carrying a familiar schema from pretraining into an unfamiliar database.
- Context flooding: Issue poorly planned queries that return more information than the agent can use within its context limits.
- Poor recovery: Repeat a failed strategy instead of using the error message to change the next attempt. Dickens reports this behavior in the 235-billion-parameter model.
Consider the talk’s concrete example: SELECT *. The query asks for all columns, so a tool response can fill the agent’s context with material that was never selected for relevance to the question. The causal problem starts before the final calculation: a broad retrieval produces an oversized observation, and that observation consumes the space needed to continue the task. Better query planning would control what comes back. The training results later show an overall improvement in answering questions; they do not isolate a measured before-and-after change for this particular query behavior.
Error recovery creates another opportunity to learn from the environment. An error message can tell the agent that its chosen operation will not work; repeating the same strategy wastes that information. Dickens also reports seeing these kinds of failures in insurance underwriting. That recurrence motivates training for reliable tool use rather than treating every failure as a shortage of financial knowledge.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reward the answer produced by the whole tool-use loop
The team used rLLM, developed by its Sky Computing Lab collaborators, to train the agent. Dickens describes a CLI-first framework that connects to existing agent frameworks through a decorator pattern with very little code change. It supports multiple reinforcement-learning algorithms and training backends; this experiment uses GRPO.
Inside the training environment, the agent runs a ReAct loop: it reasons about what to do, calls a tool to interact with the generated tables, receives the result, and continues toward an answer. The environment contains roughly 7,000 tables. A language-model judge compares the final answer with a reference and supplies a binary correctness reward. The reward therefore evaluates the result of a trajectory, rather than separately paying the agent for accessing a table or completing a query.
Where does feedback enter this system? The diagram separates two kinds of feedback: tool responses guide the next action within an attempt, while the reference-based correctness reward guides training across attempts. That distinction explains how a simple final score can train a workflow containing several decisions.
The base model is Qwen3 4B. Roughly 1,000 concurrent environments generated training trajectories, with training run on eight H100s. The reported cost was about $420 for compute and $40 for the judge API, keeping the run below $500. Those figures describe the training compute and judging costs; they do not establish an all-in cost for building and reviewing the financial dataset.
On the 290 expert-curated held-out examples, the trained model’s accuracy more than doubled relative to its base model. Its almost-60% result exceeded the 235-billion-parameter model’s 51%. The observable change is substantial, but the remaining errors matter too: the successful specialist still answers only about six in ten questions correctly in this evaluation.
Tool feedback within an attempt, reward across attempts
Read the diagram as text
- Qwen3 4B agent. Runs a ReAct loop to choose actions and produce an answer.
- Tools and financial tables. Tools give access to roughly 7,000 generated tables.
- Final answer. The trajectory ends with an answer that can be compared with a reference.
- Reference-based judge. Assigns a binary correctness reward.
- GRPO training. Uses trajectories and binary rewards to train the model.
- Qwen3 4B agent → Tools and financial tables: Tool calls.
- Tools and financial tables → Qwen3 4B agent: Observations and errors inform subsequent actions.
- Qwen3 4B agent → Final answer: Complete the attempt.
- Final answer → Reference-based judge: Compare with the reference.
- Reference-based judge → GRPO training: Binary reward.
- GRPO training → Qwen3 4B agent: Train the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Single-table training helped with multi-table questions
The first benchmark asks one question about one table. A harder extension, FinQA reasoning data, introduces dependencies across two to five tables and requires sequential decisions and planning. Its reported splits contain roughly 1,000 training examples, 120 validation examples, and 80 benchmark examples.
Training on the simpler single-table data improved performance on this multi-table benchmark without explicit multi-table training. The useful interpretation is compositional: a model that becomes more reliable at using a table can apply those skills repeatedly when a question requires several tables. This transfer result makes the original tool-discipline diagnosis more plausible, because the benefit survives an increase in planning demands.
Specialization also raised a different concern: would finance training damage general tool-calling ability? On BFCL, a benchmark for general tool calling, overall accuracy slightly improved, with minor gains in multi-turn and memory categories. For this model and evaluation, reinforcement-learning fine-tuning preserved broader tool-use competence rather than sacrificing it for the financial task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Harder examples and richer rewards did not win
The data ablation compared three ways to train:
- Single-table FinQA only: Concentrate on the simpler tasks.
- A single- and multi-table mixture: Include both kinds of questions during training.
- A curriculum: Start with single-table questions, then move to multi-table questions.
The simpler dataset produced the most lift. Dickens interprets this as evidence that reasoning depth was not the main bottleneck in this experiment. The model first needed to become reliable at the fundamentals of tool use; once those improved, it could compose them into longer solutions. Adding harder examples did not beat addressing the failures already visible in the easier tasks.
Reward design tested a separate idea. Perhaps an agent would learn faster if it received credit for intermediate behavior, such as table access or query completeness. The most elaborate variant used an expert-developed rubric with multiple weighted components. That creates a more detailed definition of a good trajectory, but also requires decisions about which behaviors deserve credit and how much each should count.
Again, the simplest version won: a single binary correctness reward produced the greatest lift on this data. The result supports starting with the task’s verifiable outcome before investing in a complicated reward rubric. It does not make intermediate rewards useless everywhere; here, the additional reward engineering did not outperform pass or fail.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the environment, the workflow length, and the output
The enterprise blueprint starts with data that represents the target tasks and outcomes, then uses reinforcement learning to improve a small model within that setting. Dickens connects this approach to better deployment economics and reports related results in healthcare, law, and insurance. The concrete finance experiment supplies the detailed example: verified questions, an interactive table environment, a checkable final answer, and a training objective that improves completion.
Reliable evaluation must then grow beyond a short question with one final answer. Dickens proposes three axes:
- Environment complexity: Test how realistic, complex, and dynamic the agent’s working environment is. A tidy task does not exercise all the difficulties of an enterprise stack.
- Autonomy horizon: Evaluate the lengths of workflows people actually intend to delegate, including whether the agent makes safe, useful decisions on its own and improves its judgment as a copilot over time.
- Output complexity: Cover the range of outputs produced in day-to-day work. Dickens sees rigorous evaluation beyond the prevailing focus on text artifacts as an underexplored opportunity.
Those axes expose the next challenge for the specialist. Answering a verifiable financial question is a useful training target, but dependable enterprise work also requires handling changing environments, longer sequences of decisions, and richer deliverables. The benchmark should expand along the dimensions that the intended deployment actually demands.
Snorkel’s Open Benchmarks grants funded this project as part of a multimillion-dollar commitment to open benchmarks. The program’s described path runs from submission and selection through development, publication, launch, and promotion. The talk closes by pointing readers toward the released training scripts, synthetic data, and model artifacts—a practical invitation to build on the experiment and to create evaluations that test the next set of real workflows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explores a complementary way to improve agent behavior: use execution traces and error messages to edit prompts and programs, rather than updating model weights from a scalar reward.
Read the complete timestamped transcript
21 slide groups. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 3 additional groups were omitted from this pass.
- 0:12
Everyone, thanks for joining. I'm Charles, and I'm a research scientist at Snorkel AI. And today I'm gonna be talking about how we got a four billion parameter model to outsmart its two hundred and thirty-five billion parameter variant. And this is a specific result in a simulated financial environment, but I think there are general takeaways that I wanna make sure are taken home. And the first is for enterprise AI workloads,
How a 4B Model Outsmarted a 235B Giant
In a simulated financial environment, a 4B-parameter model outsmarted its 235B-parameter variant. Charles Dickens frames the result around reliability, specialization, and task-representative data.
Text in image
Snorkel The Frontier AI Data Lab Towards Reliable Financial Agents: How a 4B Model Outsmarted a 235B Giant Charles Dickens Research Scientist AI Engineer World’s Fair Snorkel AI | Proprietary & Confidential | Not for distribution
- 0:42
reliability and specialization, uh, can actually outweigh raw parameter scale. And this is possible with high-quality data that actually represents the domain and task distribution that you're targeting. And, uh, this can result in smaller models that can achieve parity with frontier models at just a fraction of the cost. So before I get to it, I'm gonna take this opportunity to introduce ourselves a little bit. This is our team. We're a frontier
Snorkel AI Team and Research
Snorkel pairs data development and model evaluation expertise with research contributions: 200+ peer-reviewed papers and work on 30+ benchmarks, including tests of autonomous coding and legal reasoning.
Text in image
We build on a decade of pioneering data-centric AI to help top labs advance frontier research with high-signal data. Leadership Alex Ratner Co-Founder & CEO Stanford PhD, Asst. Prof. (Affiliate status) at University of Washington Chris Re Co-Founder Prof. at Stanford, MacArthur “Genius” Award winner Co-founder, Stanford Center for Research on Foundation Models (CRFM), NeurIPS “Test of Time” Fred Sala Chief Scientist UCLA PhD, Asst. Prof. at University of Wisconsin Team Researchers, FDEs, and Applied AI Engineers from industry and leading institutions working on data centric AI with leading global labs Stanford University Wisconsin Berkeley University of Maryland Massachusetts Institute of Technology Carnegie Mellon University UTA Columbia University Georgia Institute of Technology University of Illinois University of Toronto Yale Expertise and Capabilities • Task design for human-in-the-loop data development, ontology, rubric, and benchmark curation • Weak supervision, programmatic data development for agentic and domain-specific capabilities • Verifier, evaluator, and reward-model development, • Agentic workflow, harness, and tool-use modeling, model failure analysis and benchmarking Publications /OSS Collaborations 200+ Peer-reviewed academic papers published on data-centric AI and foundation models Browse our research library • Automating Benchmark Design • Terminal Bench 2.0 • Shrinking the generation-verification gap with weak verifiers • Skill-It! A data-driven skills framework for understanding and training language models Awards at NeurIPS, ICML, ICLR, UAI, VLDB, and more Benchmarks Co-authored / contributed to 30+ benchmarks • Agentic Coding — Tests autonomous SWE tasks. • TPBench — Tests LLMs on advanced physics problems. • WONDERBREAD — Tests enterprise business-processes. • HELM — Tests real-world scenarios. • LegalBench — Tests legal reasoning tasks. • SlopCodeBench — Tests code erosion as agents extend their own solutions • Continual Learning Bench — Tests continual learning capabilities Snorkel AI | Proprietary & Confidential | Not for distribution 2
- 1:12
AI data lab, and our team, we have founders from labs at University of Washington, Wisconsin, Stanford, and an internal team of really talented FDEs, researchers. And we're in a fortunate position to play, and by that I mean build and deploy datasets and environments, uh, for frontier AI. And that's at the intersection of academia and industry research, but also grounded in enterprise deployments. Um, and this is all to say that we have a team that's
- 1:42
built to focus on building the best data possible to define and advance AI. And the way we do this is threefold. First is open research and benchmarks, so we're constantly contributing to open source benchmarking, and the project I'll be talking about today actually follows under this pillar. Um, but I also wanna acknowledge frontier lab data and enter-- environments as well as enterprise, uh, deployments are the things we work on.
How We Partner with Labs and Enterprises
Snorkel supports open research through grants, co-authoring, and published artifacts. It also builds specialized data and environments for labs and partners on enterprise deployments.
Text in image
How we partner with labs and enterprises to advance frontier AI. Open Research & Benchmarks Build and maintain open, frontier-pushing benchmarks GRANTS CO-AUTHORING OPEN ARTIFACTS Frontier Lab Data / Environments Expert-authored datasets + environments for specialized skills FRONTIER DATA SERIES BESPOKE DATA/ENVS SCALABLE HUMAN FEEDBACK Enterprise Deployments End-to-end partnerships for data, evaluations, and feedback loops DATA / ENVS CUSTOM MODELS / DEPLOYMENTS EXPERT FEEDBACK LOOPS Snorkel AI | Proprietary & Confidential | Not for distribution 3
- 2:12
And, um, under open research and benchmarks, we support research through grants, and I'll talk about opportunities here at the end of this talk. So, um, get ready for that. And then co-authoring with academic, uh, or industry partners, um, and leading independent research and publishing open artifacts. So what I'm talking about today is, uh, a partnership with UC Berkeley Sky Computing Lab, and we built, like I said, a four billion
A 4B Model for Financial Tasks
The 4B Qwen model with RL scores 59.7% accuracy on financial tasks, versus 51.4% for its 235B variant. The slide also lists the collaborators and open project artifacts.
Text in image
We partnered with the UC Berkeley Sky Computing Lab to build a 4B model that outperforms a 235B giant on financial tasks Collaborators Manan Roongta — UC Berkeley Sky Computing Lab Bhavishya Pohani — Snorkel AI Chris Glaze — Snorkel AI Sijun Tan — UC Berkeley Sky Computing Lab Charlie Dickens — Snorkel AI rLLM Snorkel Accuracy score: 4B + RL — 59.7% 235B — 51.4% Pass@1 line (≈59.7%) Artifacts GitHub.com/rllm-org/rllm huggingface.co/rLLM/rLLM-FinQA-4B huggingface.co/datasets/rLLM/rLLM-FinQA-Dataset huggingface.co/datasets/snorkelai/finqa-data Snorkel AI | Proprietary & Confidential | Not for distribution 4
- 2:42
parameter Qwen model to outperform its two hundred and thirty-five billion model variant on financial tasks. We saw the four billion parameter model, spoiler alert, uh, achieve almost a sixty percent pass at run, while the two thirty-five billion model, uh, achieved fifty-one percent. So exciting results, and this is all, uh, in collaboration with-- I've gotta give credit where credit's due, Manan Rungta, Shujun Tan from the Sky Computing Lab, as well as Bhavishya and Chris Glaze and myself, uh, from Snorkel. And you can
- 3:12
find all the assets, the, uh, training scripts, synthetic data, uh, on our GitHub. So check those out. And this is all open source and, um, available to you. So I'll cover today what, uh, we see financial institutions need, um, from AI, and this is from our enterprise deployments, uh, learnings, as well as the FinQA open source benchmark, and this is expert-validated financial question answering data that we trained our, uh-- and evaluated our model on.
What We’ll Cover
Dickens draws on enterprise deployments to frame financial institutions’ AI needs, then introduces FinQA data and the training process for rLLM-FinQA-4B.
Text in image
What we’ll cover 1 What Financial Institutions Need from AI 2 FinQA: Expert Validated Financial QA Data 3 Training rLLM-FinQA-4B (60x more efficient!) Snorkel AI | Proprietary & Confidential | Not for distribution 5
- 3:42
And finally, uh, the training process for the RLLM FinQA 4B model. So let's get started. This is, uh, the one and only slide with the Charles Dickens reference, I promise. But this is a tale of two models, I'd say. First, the large generalist versus small specialist. And the point is here, when financial institutions ask us, "How well does the model or the agent reason?" What they typically mean is, can it operate in my stack of complex schemas, legacy
Financial Institutions Need Customized, Reliable AI
Financial institutions need agents that work with messy schemas and legacy APIs, remain reliable across 20-step workflows, and let compliance officers audit each step.
Text in image
Financial Institutions Need Customized, Reliable AI • The conventional question: "How well does the agent reason?" • The questions financial institutions actually ask: ○ Can it operate in our stack — messy schemas, legacy APIs, tech debt? ○ Can we trust it across 20-step workflows without silently compounding errors? ○ Can a compliance officer audit and defend every step? How can we evaluate and improve agents to make them reliable and trustworthy enough for financial use cases in the real world? Snorkel AI | Proprietary & Confidential | Not for distribution 7
- 4:12
APIs, and tech debt? And can we trust it in realistic, uh, and long workflows without compounding errors? And can we reliably audit the agent? And that's important in financial and legal domains, for example. Which brings us to the question is, uh, how can we evaluate and improve agents to make them reliable in these, uh, real-world environments? So, um, the reflex may be to scale. We've all seen scaling law results. Uh, but today I'm gonna argue that specialized
- 4:42
workflows don't actually need generalists. We don't need polymaths. You wouldn't call Terence Tao for a tax audit, um, if you're lucky enough to have him in your contacts. You'd call the specialist that knows the forms, the tools, and the rules. So that's why we built FinQA. This is expert-validated financial question answering data, and I'll walk through the multi-stage pipeline of how we built this benchmark and training data. So first is schema and data extraction. This is coming from 10-K
- 5:12
reports. Uh, these are annual reports required by the SEC for all public companies, and they're used, uh, by analysts for identifying risks and informing investment decisions. Uh, we pull these from the EDGAR system and use the Qwen three thirty billion model to produce tables, SQL tables, approximately six thousand nine hundred of those. And we use each table to generate a single question-answer pair. And
- 5:42
this is, uh, produced along with a question taxonomy that was developed with financial experts that work with these documents, and these are used as context, uh, for the model to generate a question-answer pair along with metadata that we'll see, uh, what that's used for later. And finally, the third step is verification, and this is a three-layer verification process. First is programmatic consistency checks. That's where we use, um, the metadata, uh, which includes
- 6:11
lineage and table names and columns to make sure we're not hallucinating those. The second is automated reviews from independent agents. And finally, expert manual review, so humans actually going through these and making sure the questions and answers are realistic and verified.
- 6:29
And the outcome is Snorkel's FinQA data, and this is our first pass at this. Uh, realistic queries requiring planning, tool use, uh, tool calling, and reasoning. And, um, answers are single, verifiable, final answers. And from this, um, we get from our, uh, original set of roughly seven thousand tables, four thousand in train, five hundred in val, and two hundred and ninety in the
Snorkel’s FinQA Environment
FinQA equips financial agents with table discovery, schema inspection, SQL and calculation tools. Its split has 4,030 training, 522 validation and 290 benchmark entries; benchmark companies do not overlap with other splits.
Text in image
Snorkel’s FinQA Environment Financial agent question-answering environment with specialized tools • get_table_names — Discover available tables for a specific company • get_table_info — Inspect table schemas, columns, and sample values • sql_query — Execute SQL queries over financial tables • calculator — Evaluate mathematical expressions Dataset | Train | Val | Benchmark | Description FinQA | 4,030 | 522 | 290 | Questions and answers over individual tables Environment built with OpenEv and available on Hugging Face Spaces Snorkel AI | Proprietary & Confidential | Not for distribution 14
- 6:59
held-out benchmark. And the data is split so that no company in the benchmark overlaps, um, in any split.
- 7:08
And on this data, we identified, um, even with frontier models, uh, reoccurring discipline gaps, uh, failure modes. And the first is schema hallucination. So a model will actually assume that tables exist, column names exist, and, um, this can be an artifact of pre-training data, something it's seen, um, in its past. The next is context flooding, so actually flooding its own context, um, with poorly planned qu-queries. For example, just
The Discipline Gap: Context Flooding
Qwen3-235B’s broad SELECT * query illustrates context flooding: if successful, the query could return enough data to overwhelm the model’s context limits.
Text in image
The Discipline Gap Schema Hallucination Context Flooding Qwen3-235B would run SELECT * queries that could overwhelm context limits if successful Question: "What is the year-over-year growth rate of YouTube ads revenue from 2023 to 2024?" Qwen3-235B: # Skips table discovery, guesses a table name sql_query(company_name="youtube", table_name="revenue_breakdown", query="SELECT youtube_ads_2023, youtube_ads_2024 FROM revenue_breakdown LIM[illegible] # ❌ Error: table revenue_breakdown for company youtube could not be found. # Guesses again instead of calling get_table_names sql_query(company_name="youtube", table_name="us_gaap_RevenueTable", query="SELECT * FROM us_gaap_RevenueTable WHERE segment = 'YouTube'") # ❌ Error: table us_gaap_RevenueTable for company youtube could not be found. # Hallucination: Fabricates a growth rate (Actual is ~14.7%) # "YouTube ads revenue grew approximately 20-25% year-over-year in 2024..." Snorkel AI | Proprietary & Confidential | Not for distribution 16- 7:39
calling select star, um, and this could overwhelm its context limits successful. And finally, poor recovery, um, from errors. So Qwen three two thirty-five would actually repeat the same failed strategy, um, instead of considering the error messages and adapting. And these failure modes are not actually just seen in the financial environment that I'm talking about today, but also what we saw in insurance underwriting. And this is, uh, work that I'm calling out that was
The Discipline Gap: Error Recovery
After two guessed tables fail, Qwen3-235B reports approximately 20–25% year-over-year growth in YouTube ad revenue from 2023 to 2024. The slide gives the actual rate as approximately 14.7%.
Text in image
The Discipline Gap Schema Hallucination Context Flooding No Error Recovery Qwen3-235B would repeat the same failed strategy (guessing table names and schemas here) instead of reading error messages and adapting Question: "What is the year-over-year growth rate of YouTube ads revenue from 2023 to 2024?" Qwen3-235B: # Skips table discovery, guesses a table name sql_query(company_name="youtube", table_name="revenue_breakdown", query="SELECT youtube_ads_2023, youtube_ads_2024 FROM revenue_breakdown LIM[illegible] # ❌ Error: table revenue_breakdown for company youtube could not be found. # Guesses again instead of calling get_table_names sql_query(company_name="youtube", table_name="us_gaap_RevenueTable", query="SELECT * FROM us_gaap_RevenueTable WHERE segment = 'YouTube'") # ❌ Error: table us_gaap_RevenueTable for company youtube could not be found. # Hallucination: Fabricates a growth rate (Actual is ~14.7%) # "YouTube ads revenue grew approximately 20-25% year-over-year in 2024..." Snorkel AI | Proprietary & Confidential | Not for distribution 17Agent Failure Modes in Insurance Underwriting
In five underwriting tasks, tool error recovery has the strongest absolute correlation with answer correctness (0.41–0.84). For business classification, uncertain count has the strongest correlation (−0.29).
Text in image
Benchmarking Agents in Insurance Underwriting Environments Amanda Dsouza, Ramya Ramakrishnan, Charles Dickens, Bhavishya Pohani, Christopher M Glaze Snorkel AI, Redwood City, CA, USA Task | Num Steps | Token Counts | Tool Errors | Uncertain Count | Hallucinations | Tool Error Recovery Appetite Check | 0.21 | -0.07 | 0.08 | -0.15 | 0.00 | 0.84 Small Business Eligibility Check | 0.19 | 0.04 | 0.14 | -0.06 | - | 0.41 Deductibles | 0.22 | -0.20 | 0.13 | -0.02 | -0.17 | 0.68 Business Classification | -0.21 | -0.27 | 0.01 | -0.29 | - | - Product Recommendations | 0.30 | -0.14 | 0.26 | -0.16 | -0.38 | 0.54 Policy Limits | 0.17 | -0.05 | 0.08 | 0.02 | -0.11 | 0.68 Table 5: Pearson correlation coefficients of different indicators to answer correctness, grouped by tasks. The largest absolute correlation for each task type is bolded. We see very similar failure modes with agents in other enterprise environments Check out Dsouza et al. CAIS 2026! Partially obscured background text: The Discipli[illegible] Schema Hall[illegible] Context Floo[illegible] No Error Rec[illegible] Qwen3-235B wo[illegible] (guessing table [illegible] reading error me[illegible] [illegible] from 2023 to 2024?" [illegible]kdown", [illegible]revenue_breakdown LIM[illegible] [illegible]ould not be found. [illegible]nueTable", [illegible]t = 'YouTube'") [illegible]e could not be found. [illegible]%) # "YouTube ads revenue grew approximately 20-25% year-over-year in 2024..." Snorkel AI | Proprietary & Confidential | Not for distribution 18
- 8:09
presented at the CAIS conference, um, a few weeks ago. So now I'll get into actually training the RLLM FinQA four billion model. Uh, for this, we used the RLLM, uh, framework, and this was developed by the-- our Sky Computing Lab collaborators. And if you haven't used this yet, um, this is a really user-friendly tool, and I'll-- A few core features, um, I wanna
Training rLLM-FinQA-4B
The next section covers training rLLM-FinQA-4B with the rLLM framework. The slide describes the model as 60x more efficient.
Text in image
3 Training rLLM-FinQA-4B (60x more efficient!) Snorkel AI | Proprietary & Confidential | Not for distribution 19
rLLM Framework Features
rLLM wraps agent code with a decorator to trace LLM calls, and offers command-line evaluation and training across multiple RL algorithms and two backends.
Text in image
rLLM: A Framework for Post-Training Language Agents rLLM Core Features • 🔌 Works with any agent framework: LangGraph SmolAgent, OpenAI Agents SDK, etc. Just swap the client! • 🪄 Near-Zero Code Changes: Add @rllm.rollout to wrap your agent code, and rLLM traces every LLM call automatically! • ⚡ CLI-first workflow: Eval and train from the command line with 50+ built-in benchmarks. rllm eval gsm8k just works. • 📈 Battle-tested results: rLLM-trained agents beat models 50x their size (4B → outperforms 235B on finance, 1.5B → surpasses O1-Preview on math). • 🧠 Multiple RL algorithms: GRPO, REINFORCE, RLOO, rejection sampling, and more. 🧠 • 🔧 Two training backends: verl for distributed multi-GPU training, tinker for single-machine / CPU setups. Same API either way. 👨💻 GitHub.com/rllm-org/rllm Snorkel AI | Proprietary & Confidential | Not for distribution 20
- 8:38
call out is it really works with any agent framework, requires near zero code changes, just using, uh, a decorator pattern, CLI-first workflow, battle-tested results. So, um, we've seen RLLM actually improve not just on this finance, um, case study, but also in math reasoning, for example. Um, multiple RL algorithms are built in, GRPO, Reinforce RL OO, for example, and you can customize those, and then supports multiple, uh, training backends.
Training Environment
The agent uses SQL and calculator tools in a ReAct loop of up to 20 steps. Its environment holds ~6,900 tables in memory, and GPT-5 nano judges correctness against a reference for a binary reward.
Text in image
Training Environment Agent ReAct loop for up to 20 steps. Tools • get_table_names • get_table_info • sql_query • calculator State Reward: Binary correctness LLMaJ (gpt-5-nano) reference based evaluation Action Environment All ~6,900 tables are in in-memory SQLite database to reduce I/O Latency SQL Tables [illegible embedded table text] Snorkel AI | Proprietary & Confidential | Not for distribution 21
- 9:10
All right, now for our training environment. Um, this is the, uh, training environment we use for our model. The agent runs in a ReAct loop and has access to tools, uh, to interact with the tables we generated. The environment includes the roughly seven thousand tables we created, and finally, reward is binary correctness, uh, determined by a language model as judge, in this case, GPT-5 nano using reference-based evaluation.
- 9:41
Training details are listed here. This is, uh, I guess would be interesting to folks. We have the Qwen three four billion model is our base model, GRPO and binary reward, uh, for optimization, eh, the RLLM framework. Um, we ran roughly a thousand concurrent environments to generate trajectories, and this all was a cost under five hundred dollars to get the results that we saw, um, broken down between
- 10:11
compute, roughly four hundred and twenty, and forty dollars for a judge, uh, the judge API. And this was on eight H100s. The first result, um, is the central question: Can a four billion model, uh, armed with the specialized tools and training actually compete with its larger variant? And the answer is yes. So what we saw was actually the base model's accuracy, um, more than doubled, and it
FinQA: Specialist vs. Generalist
On 290 expert-curated held-out samples, rLLM-FinQA-4B scored 59.7% pass@1, more than twice its base model’s 27.9% and above Qwen3-235B-A22B’s 51.4%.
Text in image
FinQA: Specialist vs. Generalist FinQA Pass@1 Results o3-mini: 63.8% GPT-4.1: 62.7% Gemini 2.5 Pro: 60.6% rLLM-FinQA-4B (Ours): 59.7% Qwen3-235B-A22B: 51.4% gpt-5-nano: 50.0% Qwen3-4B-Instruct-2507 (Base): 27.9% X-axis: Pass@1 (%) X-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70 rLLM-FinQA-4B more than doubles the base model’s accuracy and outperforms Qwen3-235B-A22B despite being a fraction of the size! Snorkel AI | Proprietary & Confidential | Not for distribution 23
- 10:41
outperforms its larger variant, the two thirty-five parameter model, and this is, uh, despite being a fraction of the size. And we evaluate all models shown here, um, on the expert-curated held-out two hundred and ninety samples.
FinQA-Reasoning Data
FinQA-Reasoning extends single-table questions to problems requiring 2–5 tables and 10–20 computed metrics. Its questions call for planning and tool use, with open-ended answers.
Text in image
Snorkel’s FinQA-Reasoning Data Questions • Realistic queries requiring planning, tool calling, and reasoning • Queries are dependent on a 2-5 tables table Answers • Open-ended final answer For AT&T, How significant are the company’s postretirement benefit obligations in terms of interest burden, and what does this indicate about the company’s long-term liability management in 2024? I'll help you analyze AT&T's post-retirement benefit obligations and their interest burden to understand the company's long-term liability management in 2024. Let me start by exploring the available data Based on my analysis of AT&T's post-retirement benefit obligations and their interest burden, I can provide you with a comprehensive assessment: Analysis Summary: AT&T's postretirement benefit obligations represent a relatively modest portion of the company's overall interest burden, indicating effective long-term liability management in 2024. The postretirement benefit interest cost of $310 million accounts for approximately 4.6% of AT&T's total interest expense of $6,759 million, while the total postretirement benefit obligation of $6,339 million represents only 4.7% of the company's total long-term debt of approximately $136 billion. The company has successfully reduced its postretirement obligations from $6,693 million to $6,339 million year-over-year, demonstrating proactive liability management, while benefiting from higher discount rates (5.60% in 2024 vs 5.00% in 2023) that help reduce the present value of future obligations, ultimately positioning these liabilities as well-controlled and not a significant financial burden relative to the company's overall debt structure. Dataset | Train | Val | Benchmark | Description FinQA | 4,030 | 522 | 290 | Questions and answers over individual tables FinQA-Reasoning | 991 | 126 | 79 | Requires 2-5 tables, computing 10-20 metrics Snorkel AI | Proprietary & Confidential | Not for distribution 24
- 10:57
So a natural concern is whether this actually transfers to harder problems. So this is just single question-answer pairs on a single table. Um, and to investigate this, we, uh, looked at our FinQA reasoning data. So this is a development on the FinQA, um, data I presented earlier. And the difference here is that now we're having, uh, dependencies on multiple tables, two to five tables, and requires sequential, um, decision-making and planning.
- 11:26
And we evaluated on this dataset. You can see the breakdown, um, as well. We have roughly a thousand in train, a hundred and twenty in validation, eighty in benchmarked. Um, and what we see is the tool use discipline actually, uh, generalized directly from training on the simple data. And no explicit training on multi-table examples was needed actually to see lift in, uh, this multi-table, um, variant of the dataset. And
FinQA-Reasoning: Domain Generalization
On multi-table FinQA-Reasoning, rLLM-FinQA-4B scores 26.6% pass@1 versus 13.9% for its base model and 18.9% for Qwen3-235B-A22B, despite training only on single-table queries.
Text in image
FinQA-Reasoning: Domain Generalization FinQA-Reasoning Pass@1 GPT-4.1: 37.9% Gemini 2.5 Pro: 34.6% o3-mini: 30.4% gpt-5-nano: 26.6% rLLM-FinQA-4B (Ours): 26.6% Qwen3-235B-A22B: 18.9% Qwen3-4B-Instruct-2507 (Base): 13.9% X-axis: Pass@1 (%) X-axis ticks: 0, 10, 20, 30, 40 rLLM-FinQA-4B nearly doubles the base model’s accuracy and again outperforms Qwen3-235B-A22B despite being trained only on single-table queries! Snorkel AI | Proprietary & Confidential | Not for distribution 25
- 11:57
we'll investigate why with ablation studies. Um, but the next concern we had was whether this, um, result would generalize to, um, more general use cases. So for this, we used the BFCL benchmark, and, um, this measures general tool-calling capability. And overall accuracy, uh, was, um, actually slightly improved. And then, um, we had also minor gains on multi-turn and memory. So specialization didn't actually
BFCL: General Tool-Calling Capability
The finance-tuned model scored 35.6% overall versus 35.0% for its base model. It gained on multi-turn and memory tasks, while scoring slightly lower on non-live AST and live accuracy.
Text in image
BFCL: General Tool-Calling Capability Overall Weighted blend of capabilities and tool calling Non-Live AST Can the model produce the expected structured tool call on controlled tasks? Live ACC Accuracy on the “Live” single-turn set Multi Turn Can the model complete tool workflows over several turns? Memory Can the model remember information and use it across sessions? FinQA rLLM BFCL Generalization Y-axis: Accuracy (%) Y-axis ticks: 0, 20, 40, 60, 80 X-axis: Category Legend: rLLM-FinQA-4B; Qwen3-4B-Instruct-2507 (Base) Overall Acc: 35.6%; 35.0% Non-Live AST: 86.7%; 87.9% Live Acc: 75.7%; 76.5% Multi Turn: 21.8%; 20.9% Memory: 23.7%; 19.1% Specialization for financial analysis did not erode the model’s broader tool-use competence! Snorkel AI | Proprietary & Confidential | Not for distribution 26
- 12:26
erode, uh, the model's broader tool use competence when we did RL fine-tuning for this model.
Simple Data Teaches Complex Skills
On validation data, FinQA-only training reached 66.3% internal Pass@1, ahead of the curriculum at 64.8% and mixed training at 61.6%. The slide attributes the bottleneck to tool-use reliability.
Text in image
Ablation Studies: Simple Data Teaches Complex Skills Single Table Only: FinQA training data Single + Multi-Table: FinQA and FinReasoning training data Curriculum (Single-> Multi): FinQA to FinReasoning training data FinQA rLLM Curriculum Ablation Single Table Only: 66.3% Curriculum (Single → Multi): 64.8% Single + Multi-Table: 61.6% X-axis: Internal Pass@1 (%) X-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70 The bottleneck was simply tool-use reliability. Once the model mastered schema discovery, SQL syntax, and error recovery it could compose those skills! *Results on validation data Snorkel AI | Proprietary & Confidential | Not for distribution 27
- 12:35
So to isolate what drove this performance, we ran an ablation study on data mixture, so different ways we can, um, combine the training data we had. The first is just training on FinQA data, and the second single and multi-table, and finally a curriculum where we start with single and then move to multi. And surprisingly, just training on the simpler set, uh, resulted in the most lift. Um, and this partially explains why, um, the generalization result we saw earlier, the bottleneck was never reasoning depth, it was just tool use, meaning the
- 13:05
model where it was actually failing and improving the reliability of that. And once you mastered the fundamentals, it could compose those skills.
Binary Rewards Outperform Partial Rewards
On validation data, binary correctness reached 66.3% Pass@1, versus 61.6% with a multi-table rubric and 54.0% with partial credit for table access.
Text in image
Ablation Studies: Binary Rewards Outperform Partial Rewards Binary Correctness: Baseline binary reward on FinQA training data Partial: Partial credit for successful table access on FinQA Training data (0.2 x table_access + 0.8 correctness) Binary + Multi-Table Rubric: Binary reward with additional rubric covering Data Retrieval, Calculated Metrics, Reasoning, Consistency, Completeness, Structure on FinQA and FinReasoning Data FinQA rLLM Reward Ablation Binary Correctness: 66.3% Binary + Multi-Table Rubric: 61.6% Partial: 54.0% X-axis: Pass@1 (%) X-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70 Despite an investment in reward design, partial reward variants underperformed the simplest possible reward: a single binary signal! *Results on validation data Snorkel AI | Proprietary & Confidential | Not for distribution 28
- 13:14
Um,
- 13:18
yeah. So, oops. Let me go back. Um, the next natural question is whether, um, more informative reward signals, um, for example, rewarding intermediate steps, uh, table access, query completeness, things like that could actually accelerate learning. Uh, so this is different from curriculum, for example. And the most sophisticated variant we came up with was a, a rubric, and this uses fine grain scoring with, uh, multiple weighted components and this is actually coming-- came up with, with experts.
- 13:48
Um, and despite this investment in reward engineering, we actually again saw, um, simpler was better. A single binary reward signal, um, was what got us the most lift on this data, which led us to a blueprint, um, that we've been developing for enterprise agents. Um, and generally, I would say this study showed us that small models, uh, can successfully be trained, uh, with RL, um, when you have quality data to, uh, represent the
The Blueprint for the Enterprise Agent
A 4B model can run on one GPU, while models of 235B and above require multi-node clusters. The speaker says RL with quality task data has produced repeatable results in healthcare, law, and insurance.
Text in image
The Blueprint for the Enterprise Agent ⚡ Efficiency over Scale The performance of rLLM-FinQA-4B shows us that the goal is not always to build a model that can write poetry and solve IMO problems, its to build reliable specialists. 💸 Deployment Economics A 4B model can run on a single GPU, while 235B and larger models require multi-node clusters, enabling lower cost and stronger accuracy 🔁 Repeatable The same approach generalizes to specialized tasks in healthcare, law, insurance, and other domains! Snorkel AI | Proprietary & Confidential | Not for distribution 29
- 14:18
target, um, outcomes and tasks. And this can result in better deployment economics, uh, for specialized tasks and domains. And, um, we've seen this and repeated this result in other areas, including healthcare, law, and insurance, um, which I pointed to earlier. So check that out. Um, and our point of view as far as building, um, axes for evaluating agents is this, is environment
- 14:48
complexity is, um, one of the most important axes. How complex, how realistic and dynamic are the environments, uh, the agent's operating in? And the next is autonomy horizon. Um, are agents being tested and evaluated at the lengths and the horizon lengths and the way people actually wanna use agents? Um, are they making safe and good decisions by themselves? And can they actually improve, um, their judgment as a co-pilot over time? And finally, output complexity.
- 15:18
We wanna capture the wide range of outputs in our day-to-day work. Um, and I would say that this is currently under explored. Um, new benchmarks are coming up in this direction, but, uh, still I see a big, uh, primary focus on, um, text artifacts and, um, creating more rigorous evaluations in, um, these settings is, uh, a big opportunity. Uh, progress is being made here. So on that note, um, I'm gonna
Open Benchmarks Grants
Open Benchmarks Grants funded this project as part of a $3M+ commitment to open benchmarks. Applicants go through submission and review, development, and publication and promotion.
Text in image
Open Benchmarks Grants A $3M+ commitment to open benchmarks IN PARTNERSHIP WITH Snorkel Hugging Face PRIME Intellect together.ai FACTORY harbor PyTorch
- 15:48
advertise our Open Benchmarks grants, which funded this project, and this is a multimillion-dollar commitment, um, to producing open benchmarks like this. Um, we've had a lot of success, Agent, uh, Last Exam, um, Judgment Bench, um, others are coming out and this is, um, really accelerating our ability to measure the frontier and shape the frontier. Um, so to apply, uh, check out our website. Um,
- 16:18
you'll go through a submission, uh, selection review process, develop, and then finally publication, launch, um, and promotion. And also wanna call out that we're hiring in professional research, um, you name it, AI engineers and, um, yeah, check that out. You can, uh, check out our website. And finally, thank you so much for attending. Um, you can use this QR
- 16:48
code to see the blog as well as pointers to all the artifacts that released, uh, on GitHub, Hugging Face. Um, yeah. Thank you all.