Do Not Let Models Invent Numbers - Vinoo Ganesh, Kepler
Loading slides…
Read the talk
Do Not Let Models Invent Numbers
Vinoo Ganesh explains how Kepler separates model planning from financial data retrieval and arithmetic, then checks numerical output against a citation registry before showing it to an analyst.
From a talk by Vinoo Ganesh
At a glance
Ideas worth remembering
A financial figure needs more than a source link: its value, unit, scale and period must stay attached to the right fact.
Kepler assigns planning to the model, retrieval and arithmetic to tools, and numerical output checks to a post-processing filter.
Hybrid retrieval finds candidates; further ranking handles relevance and source trust, while Postgres reauthorization checks whether each hit is permitted.
Unresolved numerical claims are stripped, with selective citation repair attempted in some cases. This trades speculative freedom for traceability.
The model can still hallucinate. The claimed protection belongs to the platform’s execution and filtering path, and does not guarantee an investment outcome.
A source link cannot explain what “17” means
A sentence saying revenue grew 12.4% to $418.2 billion gives a financial analyst two figures to defend. Both need a traceable source, and the growth rate needs a defensible calculation. Vinoo Ganesh, introducing himself as Kepler’s CEO, starts from that working requirement: an analyst should be able to show a client or portfolio manager where a number came from. His experience leading compute at Palantir and business engineering at Citadel informs the architecture that follows.
Attaching a source after generating an answer leaves several independent ways to fail:
- Wrong source: Retrieval can find a semantically relevant opinion rather than reliable evidence. Ganesh’s deliberately absurd example is a personal “YOLO blog” predicting Palantir will reach $100,000 a share. Citing it would identify the opinion without making it dependable.
- Wrong interpretation: A 10-K or 10-Q can contain the number 17, but that token alone says almost nothing. It could mean millions, billions, yen, dollars, revenue from an earlier year, or simply a page number.
- Wrong arithmetic: Growth rates, margins and multiples require operations on selected inputs. Generating a plausible numerical answer does not provide the fidelity of executing a defined calculation.
These failures need different controls. Better search can improve source selection, but it cannot recover a missing currency or table scale by itself. Correctly extracting two values does not ensure that their growth rate was calculated correctly. Kepler’s governing decision is therefore to make the model cite numbers while other software supplies and computes their values.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model plans; the registry supplies the numbers
Kepler divides the work among retrieval, human-written deterministic code and a post-processing filter. Retrieval finds evidence. Predetermined code performs arithmetic. The filter removes numerical output that cannot be verified. The distinction between human-written calculation code and code improvised by the model matters here: the permitted computation belongs to the platform’s implementation.
The citation registry begins before a user asks a question. Ganesh says Kepler processed more than 30 million SEC documents and registered numerical facts extracted from filings. Those facts include financial-statement values and figures embedded in management discussion and analysis, such as a written statement that revenue grew 37%. A registered citation gives a fact an anchor back into its source document. Production output must resolve to verifiable evidence rather than merely carry a document link.
Consider the question “What is Google’s adjusted EBITDA?” The model’s useful job is to work out what answering it requires. Relevant facts may span multiple years of 10-Ks. An investor presentation or earnings call may disclose an adjustment, and a later restatement may change a previously reported value. The model plans those retrieval and reconciliation steps with knowledge of the available tools; tools then retrieve facts, compute results, build tables and register the resulting information.
The registry also preserves meaning. In Ganesh’s hypothetical example, a Palantir 2025 10-K contains a 2023 revenue figure of $1 billion. The fact must retain 2023 as its data period even though the document is from 2025. And the visible cell might contain only 1, with “billions” elsewhere in the table. Losing either relationship produces a numerically plausible but incorrectly scoped answer.
Planning does not stop a model from inventing additional figures while composing its response. The final filter exists because that behavior remains possible: if the platform cannot deterministically establish that a number came from a trusted source, it strips the number. The guarantee is enforced on the output path, after the model has had an opportunity to generate unsupported content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry value, unit, scale and period together
The total-net-sales example makes the extraction problem concrete. A value around 416,000 is not yet an interpretable financial fact. It needs a unit, a scale and a period alongside the value. These fields determine whether a downstream calculation compares like with like. The important relationship in a filing is often between a cell and information elsewhere on the page, rather than between adjacent words in a text chunk.
Structured filing data helps, but it cannot cover the whole research workflow. Ganesh challenges the idea that XBRL solves extraction completely, estimating that it is incorrect around 10% of the time; he does not define the error measure behind that estimate. More fundamentally, investment teams also use their own memos and internal documents, which do not all arrive as structured financial records.
Kepler therefore structures data where structure is available and searches where it is not. The intended corpus includes public filings alongside internal sources such as SharePoint, Box, Egnyte, OneDrive and Outlook. A turbopuffer namespace supports retrieval across that information, while the surrounding Kepler machinery remains responsible for interpreting facts and verifying numerical output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search widely, then rank for relevance and trust
Retrieval combines two search mechanisms over the namespace. Approximate nearest-neighbor search, or ANN, retrieves through embeddings; BM25 supplies lexical matching. Ganesh describes using 1,280-dimensional embeddings. The architecture searches broadly through both mechanisms before narrowing the candidates, rather than asking a single retrieval score to settle every question about relevance and source quality.
The candidates pass through reciprocal reranking and another ranking stage that accounts for source trust, with a cross-encoder used for reranking before results return to the model. Source trust matters when apparently equivalent facts disagree. An earnings-call transcript may report revenue differently from the 10-K. Treating these as two pieces of information to reconcile preserves the conflict; collapsing them into one interchangeable search result hides it.
In the diagram, ANN and BM25 run at the same time. Each retrieves three times the requested candidate count. Reciprocal rank fusion combines their lists using a rank constant of 60. The cross-encoder stage selects 20 documents, and the diagram shows 10 returned to the model, with a maximum of 25.
Keeping this pipeline current has an operating cost. New information can require regenerated embeddings, and Ganesh emphasizes the expense of doing that repeatedly. He presents efficient embedding generation, including the possibility of hosting the embedding model, as an important part of making the retrieval strategy practical. The ranking design depends on being able to refresh the searchable information economically.
Parallel retrieval feeds a narrower evidence set
Read the diagram as text
- Searchable namespace. Information available to the retrieval pipeline.
- ANN search. parallel-search — Embedding-based approximate nearest-neighbor retrieval.
- BM25 search. parallel-search — Lexical retrieval over the namespace.
- Reciprocal reranking. Ranks the broadly retrieved candidates.
- Further reranking. Accounts for trusted sources; uses a cross-encoder.
- Model receives evidence. The selected retrieval results return to the model.
- Searchable namespace → ANN search: Search through embeddings.
- Searchable namespace → BM25 search: Search lexical matches.
- ANN search → Reciprocal reranking: Supply candidates.
- BM25 search → Reciprocal reranking: Supply candidates.
- Reciprocal reranking → Further reranking: Pass ranked candidates.
- Further reranking → Model receives evidence: Return selected results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resolve the reference, repair selectively, strip when unproven
A model may already know a historical revenue figure from training. That familiarity does not establish that it recalled the right year, scope or value, and it does not supply a traceable extraction path. Ganesh uses revenue from 2005 to illustrate why even a convincing historical answer must pass through the evidence system.
Ganesh characterizes roughly a third of Kepler’s citations as “the math the model didn’t do.” The point is that a substantial share of cited output represents work delegated away from the model, rather than facts accepted from its training memory. Those citations can arise while working with web results, spreadsheets and filing exhibits as well as other retrieved material.
During inference, references are resolved against the registry. Ganesh’s analogy is dereferencing a pointer: a reference identifies an entry, and the registry supplies the information needed to trace it. If no matching entry can support a numerical claim, the filter strips the number. A fluent sentence is insufficient to let an unresolved value through.
That decision has a visible product cost. Kepler gives up some of the speculative freedom people enjoy in a conversational model. Ganesh explicitly says it is a poor fit for creative marketing content or for users who want the model to invent projections freely. The same restriction is useful when building a discounted cash flow model, or DCF, whose inputs must be accurate and traceable.
Removal is not always the first or only response. A post-processing step can attempt citation repair: when information cannot be reliably traced, an agentic process may try to establish which source supports it. Ganesh says Kepler does this in selected cases but leaves the selection rules and repair mechanism proprietary. The disclosed distinction is between attempting to recover a valid source relationship and allowing an unsupported number to remain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the filter caught in ordinary model output
Kepler’s slide headline says the filter refused 10,981 citations. The bars add up to that total only when they include 799 events labeled “passed unverified (soft).” Kepler classifies 3,810 events as numeric mismatches. The largest category, 4,542 quotes the filter could not find in their sources, illustrates how a traceability check can fail without establishing that a number is wrong. The chart cites production event logs from April through July 2026. Ganesh describes evaluations involving Claude or ChatGPT, but does not explain how those evaluations relate to the logs. The chart gives neither the total number of citations checked nor counts by model.
These results motivate the architectural objection to plain retrieval-augmented generation. Connecting a language model to a vector database can deliver relevant context, but relevance alone does not verify the numerical answer produced from it. A financial research system also needs controls over source identity, the meaning of extracted values, arithmetic and the final response. Retrieval is one component of that system.
The sharpest correction in the talk concerns who owns the guarantee. Ganesh begins to describe the model as structurally incapable of producing hallucinated numbers, then corrects himself: the platform enforces that restriction. The model still hallucinates. The filter and deterministic execution path are intended to prevent its unsupported numerical inventions from reaching the user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Click through the calculation—and keep the guarantee in scope
The product example asks for Apple’s fiscal 2025 revenue and how quickly it grew. The screen shows net sales of $416,161 million in fiscal 2025 and $391,035 million in fiscal 2024. Dividing the $25,126 million increase by the fiscal 2024 figure gives 6.4% growth after rounding. Clicking the revenue figure highlights the filing entry. Opening the growth calculation shows the two inputs, each with its own source.
That inspection path targets a familiar manual workflow. Analysts still extract information from S-1s and move figures from PDFs into Excel because producing a usable financial model requires more than a plausible answer. Kepler’s aim is to automate that transfer and calculation work while preserving the information needed to check it.
The closing guarantees remain specific:
- Citations: Models select references, and the platform verifies citations before presenting numerical output.
- Arithmetic: The compute layer performs calculations using deterministic code.
- Isolation: Tenant and document controls limit which research a user can retrieve.
- Recall: Kepler measures retrieval recall, acknowledging that finding the needed evidence is a separate concern from checking a value already retrieved.
Those controls do not amount to an autonomous hedge fund or a replacement for everyone making investment decisions. Ganesh expects more “quantamental” work, combining quantitative methods with fundamental research. The intended division of labor leaves people with the investment ideas they want to develop and gives AI the repetitive retrieval and assembly work they would rather avoid. Traceable numerical execution makes that handoff more useful without turning an investment judgment into a guaranteed outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The architecture case study Ganesh points readers toward for more detail on Kepler’s use of model planning and deterministic infrastructure.
Further reading
- Context Is the Easy PartArticle
Ganesh develops the surrounding data-engineering requirements: provenance, versioning, validation, reproducibility and observability when retrieved context changes.
Related talks
- How Kepler Built Verifiable AI for Financial Services
Develops the companion concepts of atomic provenance, scoped determinism and derivation chains, including how firm-specific processes affect verification.
- Respect The Process
Applies a comparable separation to sustainability calculations: agents propose changes through typed tools, while controlled deterministic execution validates consequential operations.
Full transcript
Loading transcript…