Our Smaller Model Beat Opus on Finance Tasks - Alexander Shevchenko, Ramp
Loading slides…
Read the talk
How Ramp’s 35B Model Beat Opus 4.6 on Financial Question Answering
Ramp Labs kept Opus in charge of financial modeling, trained a faster model to answer its questions, and found two other ways to save time on agent work: prepare the context overnight or run thousands of investigations in parallel.
From a talk by Alexander Shevchenko
At a glance
Ideas worth remembering
A planner’s recurring questions make a bounded target for post-training. Measure the specialist’s answer accuracy and response latency separately from completion of the full assignment.
For a scheduled workflow such as month-end close, identify the documents the agent will need and prepare their context before the user starts.
When comparing models for a parallel search, include the number of attempts a budget buys and the cost of deduplicating and reproducing findings.
To explain a token bill, classify the activities in user sessions and reconcile those categories with spend by key or user.
Why do research when waiting keeps paying off?
Alexander Shevchenko had spent about three years on Ramp’s applied AI team, working on products for financial professionals. He described Ramp as a company built on the bitter lesson. Over the preceding year, the number of use cases employing LLMs at Ramp had grown 123 percent. Several of its most-used features consumed more tokens in aggregate even as their unit economics improved. New frontier models let Ramp deliver the same accuracy at a lower cost per use. That record raised a fair question about Ramp Labs. What could an application company gain from doing its own research while model providers were already making its products faster and cheaper?
Shevchenko recalled building agents with GPT-4 in 2023. Once an agent took a wrong turn, it tended to keep going. The models lacked thinking tokens and had little reinforcement training to backtrack, so engineers wrote strict guardrails around the steps an agent could take. Even substituting a synonym in a prompt could change its behavior. Engineers also pared down the context because too much material could confuse the model. Ramp did that work so financial professionals could use Ramp Tour Guide and other agents without waiting for GPT-4.5 or GPT-5.
By 2026, engineers could give a model tools through MCP and access to a much wider body of context, and it could usually work out what the task required. Ramp’s reason to go further came from watching where people still had to wait. The first example surfaced in a product that already worked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A discounted cash flow model exposed the cost of reading the sheets
Ramp launched Ramp Sheets in November 2025, before either of the two frontier labs Shevchenko referred to had released a spreadsheet agent. Customers began using it as a general-purpose spreadsheet tool. Accountants and finance professionals spent much of their day in spreadsheets, as did people working inside Ramp. The team saw an opportunity to reuse the agent’s harness and underlying model in other parts of the product. As it did that, it looked for the step that caused users the most friction.
One revealing assignment was to give Ramp Sheets a collection of spreadsheets and ask it to build a discounted cash flow model or model a financial scenario. Opus 4.6 first had to make a plan. Then it had to explore the available sheets, establish what each contained and find the financial information the plan called for. From the user’s perspective, much of the wait came before the requested model took shape. From Ramp’s perspective, it was paying for Opus thinking tokens while the agent got its bearings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Opus keeps the plan and Qwen answers the financial questions
Ramp split the assignment where the planner needed an answer from the data. The frontier model remained the orchestrator. It could decide what it needed to know, hand a financial question to a faster model and use the answer to continue its plan. Shevchenko described the specialist first as a retrieval model and then, more precisely, as a financial question-answering model. Its job was to return an answer the planner could use, which gave Ramp a specific capability to train and measure.
Working with Prime Intellect, Ramp post-trained a 35-billion-parameter Qwen 3.5 model. On exact task accuracy for financial question answering, it scored above Opus 4.6, the frontier model Ramp was using at the time. Ramp could also serve the specialist at Haiku-level latency. Those comparisons describe two properties of the handoff: how often the planner got the right answer and how long it waited for that answer. Shevchenko gave no end-to-end timing or accuracy figure for the finished financial model. Ramp shipped the specialist and frontier orchestrator together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a KV cache cartridge the night before month-end close
A user can drop an unfamiliar workbook into Ramp Sheets at any time. Month-end close gives Ramp advance notice of both the task and much of its input. On the first of the month, a financial professional will need to reconcile and close the books. Ramp can anticipate documents such as the ledger and the aging report. Making those files available still leaves the agent with a job when the accountant arrives. It has to retrieve the relevant material and build up a picture of the accounts.
A cartridge moves that preparation offline. It compresses context into the model’s key-value cache, the internal representations the model uses to attend to earlier material. Shevchenko said cartridges could pack up to 20 to 30 times as much context into the cache. In the close example, Ramp could build a cartridge overnight from the anticipated documents. When the accountant opened the agent the next morning, it would begin with that context already available. The multiplier measures how much context fits. The time saved on a particular close depends on how much retrieval and preparation the cartridge covers.
Shevchenko found cartridges less work to build than a specialist model because the self-play procedure generalized more readily. He also saw a reason a frontier lab might eventually offer the capability through an API as part of its work on continuous learning. Ramp would still have learned which jobs it could anticipate, what material those jobs required and when to prepare it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
10,000 agents search Ramp and a second pass tests their findings
For penetration testing, Ramp used its cloud agent, Ramp Inspect, to supply the infrastructure for 10,000 agents running in parallel. The system organized their work as an agentic MapReduce. In the map phase, agents investigated Ramp for vulnerabilities and filed tickets for suspected problems. A ticket was a candidate finding. Several agents could discover the same issue, and an agent could offer a plausible account of an exploit that failed when another investigator tried it.
In the reduce phase, LLMs deduplicated the tickets, then tried adversarially to reproduce the suspected vulnerabilities. This gave Ramp a way to filter the search results before spending people’s time on them. The exercise found seven high-severity vulnerabilities, all of which Ramp patched. One was a Lua script injection deliverable by email. In Shevchenko’s account, each component along the path was safe in isolation. Discovering the injection required tracing how the components behaved together from receipt of the email through to script execution.
Automated filtering also changed the economics of model selection. A failed investigation cost tokens, but a person could be spared the work of pursuing it. Ramp tried open-weight models that were up to five times cheaper than frontier reasoning models. Their detection rate fell by a smaller factor than their price. Ramp could therefore buy more attempts with the cheaper tokens and bring the swarm’s detection rate close to the frontier result. Shevchenko gave no exact detection percentages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Token Spend Management accounts for usage and Ramp Router changes its cost
Shevchenko connected the experiments to two products. Token Spend Management connects to a company’s API keys and attributes spending by key or user. For finance teams, it also supports accounting categories and analysis of spending that may qualify for tax credits. For someone deciding where to improve an agent, attribution answers only the first question. Knowing who incurred a charge still leaves the question of what work those tokens bought. Ramp analyzes usage in context and builds a taxonomy of activities that can be reconciled with the bill.
Ramp Router applies research to the execution side. Ramp Labs helped develop routing strategies aimed at improving agent latency and cost per token. Shevchenko named NVIDIA Switch Yard and Smart Flex offloading among the examples. He described multiple routing strategies as available, without giving a selection rule or a benchmark for an individual strategy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Transfer a post-train, revise a monitor and discover usage categories
A project called Portal addresses a cost that the Qwen result leaves open. What happens to the investment in a specialist when a newer base model arrives? Shevchenko described doing a LoRA post-train on a model such as Qwen 3.5 and learning a latent representation that can be projected onto another model. Ramp explored transfers to later Qwen versions and across model families. Reusing what the first post-train learned lets the team spread its training cost across models.
In a separate project on self-improving monitoring, Ramp used LLMs to create Datadog monitors for product surfaces. Incoming alerts then became input to another step in which the system revised the monitors automatically. For semantic tagging, the input was a collection of user sessions. Agents annotated what happened in each session, and an agentic MapReduce assembled those observations into an open-ended taxonomy. Ramp could discover clusters of LLM use without first asking someone to enumerate every activity in the company. Those resulting categories fed back into Token Spend Management, where finance teams could match the work people actually did to what it cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…