Do Not Let Models Invent Numbers - Vinoo Ganesh, Kepler

Vinoo Ganesh· CEO & Co-Founder, Kepler20:58

Loading slides…

Read the talk

Do Not Let Models Invent Numbers

Kepler separates retrieval, deterministic arithmetic and a filter that checks numerical citations.Open full source frame
Kepler separates retrieval, deterministic arithmetic and a filter that checks numerical citations.

Vinoo Ganesh explains how Kepler separates model planning from financial data retrieval and arithmetic, then checks numerical output against a citation registry before showing it to an analyst.

From a talk by Vinoo Ganesh

At a glance

Ideas worth remembering

  • A financial figure needs more than a source link: its value, unit, scale and period must stay attached to the right fact.

  • Kepler assigns planning to the model, retrieval and arithmetic to tools, and numerical output checks to a post-processing filter.

  • Hybrid retrieval finds candidates; further ranking handles relevance and source trust, while Postgres reauthorization checks whether each hit is permitted.

  • Unresolved numerical claims are stripped, with selective citation repair attempted in some cases. This trades speculative freedom for traceability.

  • The model can still hallucinate. The claimed protection belongs to the platform’s execution and filtering path, and does not guarantee an investment outcome.

A source link cannot explain what “17” means

A sentence saying revenue grew 12.4% to $418.2 billion gives a financial analyst two figures to defend. Both need a traceable source, and the growth rate needs a defensible calculation. Vinoo Ganesh, introducing himself as Kepler’s CEO, starts from that working requirement: an analyst should be able to show a client or portfolio manager where a number came from. His experience leading compute at Palantir and business engineering at Citadel informs the architecture that follows.

Attaching a source after generating an answer leaves several independent ways to fail:

  • Wrong source: Retrieval can find a semantically relevant opinion rather than reliable evidence. Ganesh’s deliberately absurd example is a personal “YOLO blog” predicting Palantir will reach $100,000 a share. Citing it would identify the opinion without making it dependable.
  • Wrong interpretation: A 10-K or 10-Q can contain the number 17, but that token alone says almost nothing. It could mean millions, billions, yen, dollars, revenue from an earlier year, or simply a page number.
  • Wrong arithmetic: Growth rates, margins and multiples require operations on selected inputs. Generating a plausible numerical answer does not provide the fidelity of executing a defined calculation.

These failures need different controls. Better search can improve source selection, but it cannot recover a missing currency or table scale by itself. Correctly extracting two values does not ensure that their growth rate was calculated correctly. Kepler’s governing decision is therefore to make the model cite numbers while other software supplies and computes their values.

0:121:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

The model plans; the registry supplies the numbers

Kepler divides the work among retrieval, human-written deterministic code and a post-processing filter. Retrieval finds evidence. Predetermined code performs arithmetic. The filter removes numerical output that cannot be verified. The distinction between human-written calculation code and code improvised by the model matters here: the permitted computation belongs to the platform’s implementation.

Language models plan analyses while Kepler’s tools retrieve facts, calculate values and register citations.Open full source frame
Language models plan analyses while Kepler’s tools retrieve facts, calculate values and register citations.

The citation registry begins before a user asks a question. Ganesh says Kepler processed more than 30 million SEC documents and registered numerical facts extracted from filings. Those facts include financial-statement values and figures embedded in management discussion and analysis, such as a written statement that revenue grew 37%. A registered citation gives a fact an anchor back into its source document. Production output must resolve to verifiable evidence rather than merely carry a document link.

Consider the question “What is Google’s adjusted EBITDA?” The model’s useful job is to work out what answering it requires. Relevant facts may span multiple years of 10-Ks. An investor presentation or earnings call may disclose an adjustment, and a later restatement may change a previously reported value. The model plans those retrieval and reconciliation steps with knowledge of the available tools; tools then retrieve facts, compute results, build tables and register the resulting information.

The registry also preserves meaning. In Ganesh’s hypothetical example, a Palantir 2025 10-K contains a 2023 revenue figure of $1 billion. The fact must retain 2023 as its data period even though the document is from 2025. And the visible cell might contain only 1, with “billions” elsewhere in the table. Losing either relationship produces a numerically plausible but incorrectly scoped answer.

Planning does not stop a model from inventing additional figures while composing its response. The final filter exists because that behavior remains possible: if the platform cannot deterministically establish that a number came from a trusted source, it strips the number. The guarantee is enforced on the output path, after the model has had an opportunity to generate unsupported content.

4:114:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:11 · section reference included

Carry value, unit, scale and period together

The total-net-sales example makes the extraction problem concrete. A value around 416,000 is not yet an interpretable financial fact. It needs a unit, a scale and a period alongside the value. These fields determine whether a downstream calculation compares like with like. The important relationship in a filing is often between a cell and information elsewhere on the page, rather than between adjacent words in a text chunk.

Kepler combines structured filing data with search over unstructured documents.Open full source frame
Kepler combines structured filing data with search over unstructured documents.

Structured filing data helps, but it cannot cover the whole research workflow. Ganesh challenges the idea that XBRL solves extraction completely, estimating that it is incorrect around 10% of the time; he does not define the error measure behind that estimate. More fundamentally, investment teams also use their own memos and internal documents, which do not all arrive as structured financial records.

Kepler therefore structures data where structure is available and searches where it is not. The intended corpus includes public filings alongside internal sources such as SharePoint, Box, Egnyte, OneDrive and Outlook. A turbopuffer namespace supports retrieval across that information, while the surrounding Kepler machinery remains responsible for interpreting facts and verifying numerical output.

8:188:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:18 · section reference included

Search widely, then rank for relevance and trust

Retrieval combines two search mechanisms over the namespace. Approximate nearest-neighbor search, or ANN, retrieves through embeddings; BM25 supplies lexical matching. Ganesh describes using 1,280-dimensional embeddings. The architecture searches broadly through both mechanisms before narrowing the candidates, rather than asking a single retrieval score to settle every question about relevance and source quality.

The candidates pass through reciprocal reranking and another ranking stage that accounts for source trust, with a cross-encoder used for reranking before results return to the model. Source trust matters when apparently equivalent facts disagree. An earnings-call transcript may report revenue differently from the 10-K. Treating these as two pieces of information to reconcile preserves the conflict; collapsing them into one interchangeable search result hides it.

Parallel ANN and BM25 retrieval feed reciprocal rank fusion and cross-encoder reranking.Open full source frame
Parallel ANN and BM25 retrieval feed reciprocal rank fusion and cross-encoder reranking.

In the diagram, ANN and BM25 run at the same time. Each retrieves three times the requested candidate count. Reciprocal rank fusion combines their lists using a rank constant of 60. The cross-encoder stage selects 20 documents, and the diagram shows 10 returned to the model, with a maximum of 25.

Keeping this pipeline current has an operating cost. New information can require regenerated embeddings, and Ganesh emphasizes the expense of doing that repeatedly. He presents efficient embedding generation, including the possibility of hosting the embedding model, as an important part of making the retrieval strategy practical. The ranking design depends on being able to refresh the searchable information economically.

Parallel retrieval feeds a narrower evidence set

Searchable namespaceInformation available to the retrieval pipeline.
ANN searchparallel-search — Embedding-based approximate nearest-neighbor retrieval.
BM25 searchparallel-search — Lexical retrieval over the namespace.
Reciprocal rerankingRanks the broadly retrieved candidates.
Further rerankingAccounts for trusted sources; uses a cross-encoder.
Model receives evidenceThe selected retrieval results return to the model.
ANN and BM25 retrieve candidates independently. Ranking combines and narrows those candidates before evidence returns to the model.
Read the diagram as text
  • Searchable namespace. Information available to the retrieval pipeline.
  • ANN search. parallel-search — Embedding-based approximate nearest-neighbor retrieval.
  • BM25 search. parallel-search — Lexical retrieval over the namespace.
  • Reciprocal reranking. Ranks the broadly retrieved candidates.
  • Further reranking. Accounts for trusted sources; uses a cross-encoder.
  • Model receives evidence. The selected retrieval results return to the model.
  • Searchable namespace → ANN search: Search through embeddings.
  • Searchable namespace → BM25 search: Search lexical matches.
  • ANN search → Reciprocal reranking: Supply candidates.
  • BM25 search → Reciprocal reranking: Supply candidates.
  • Reciprocal reranking → Further reranking: Pass ranked candidates.
  • Further reranking → Model receives evidence: Return selected results.
9:4811:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:48 · section reference included

Authorize each hit and preserve each fact’s identity

Internal research introduces a separate requirement: a relevant document may still be forbidden to the person asking. Ganesh uses two Citadel desks as the example. Desk One must not see Desk Two’s research simply because both belong to the same firm. Isolation therefore has to reach the document level.

A registered citation separates the source identifier, displayed value and underlying anchor.Open full source frame
A registered citation separates the source identifier, displayed value and underlying anchor.

Every retrieval hit is reauthorized in Postgres. The checks cover the citation, value and document through the computations, and builders add filters as queries are assembled. Only the authentication component can create a scoped token. These controls make permission checks part of the path that retrieves and uses information, rather than relying solely on an initial search scope.

A fact also needs an identity that survives its presentation. Kepler assigns unique identifiers to facts, with source IDs resolving to stored values and metadata in a deterministic engine. Entity recognition and resolution help determine which entity a fact belongs to. The display value can then be formatted for a requested presentation, such as thousands of dollars, while retaining the underlying fact and its source.

The user-facing invariant is that every number and fact can be clicked to inspect where it came from and how it was obtained. Ganesh connects this to reproducibility and describes an idempotent guarantee: repeated queries yield the same results. The deterministic values and calculations explain the intended basis for that guarantee; the talk does not specify how repeated queries are scoped when source data or restatements change.

11:1811:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:18 · section reference included

Resolve the reference, repair selectively, strip when unproven

A model may already know a historical revenue figure from training. That familiarity does not establish that it recalled the right year, scope or value, and it does not supply a traceable extraction path. Ganesh uses revenue from 2005 to illustrate why even a convincing historical answer must pass through the evidence system.

Kepler checks numerical citations against the registry while streaming a response.Open full source frame
Kepler checks numerical citations against the registry while streaming a response.

Ganesh characterizes roughly a third of Kepler’s citations as “the math the model didn’t do.” The point is that a substantial share of cited output represents work delegated away from the model, rather than facts accepted from its training memory. Those citations can arise while working with web results, spreadsheets and filing exhibits as well as other retrieved material.

During inference, references are resolved against the registry. Ganesh’s analogy is dereferencing a pointer: a reference identifies an entry, and the registry supplies the information needed to trace it. If no matching entry can support a numerical claim, the filter strips the number. A fluent sentence is insufficient to let an unresolved value through.

That decision has a visible product cost. Kepler gives up some of the speculative freedom people enjoy in a conversational model. Ganesh explicitly says it is a poor fit for creative marketing content or for users who want the model to invent projections freely. The same restriction is useful when building a discounted cash flow model, or DCF, whose inputs must be accurate and traceable.

Removal is not always the first or only response. A post-processing step can attempt citation repair: when information cannot be reliably traced, an agentic process may try to establish which source supports it. Ganesh says Kepler does this in selected cases but leaves the selection rules and repair mechanism proprietary. The disclosed distinction is between attempting to recover a valid source relationship and allowing an unsupported number to remain.

13:4614:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:46 · section reference included

What the filter caught in ordinary model output

Kepler’s slide headline says the filter refused 10,981 citations. The bars add up to that total only when they include 799 events labeled “passed unverified (soft).” Kepler classifies 3,810 events as numeric mismatches. The largest category, 4,542 quotes the filter could not find in their sources, illustrates how a traceability check can fail without establishing that a number is wrong. The chart cites production event logs from April through July 2026. Ganesh describes evaluations involving Claude or ChatGPT, but does not explain how those evaluations relate to the logs. The chart gives neither the total number of citations checked nor counts by model.

Horizontal bars count citation events by reason. The headline calls 10,981 citations refused, but the bars include 799 marked “passed unverified (soft).” Five red bars total 3,810 numeric mismatches.Open full source frame
The slide’s headline counts 10,981 citations as refused, including 799 labeled “passed unverified (soft).” Kepler identifies 3,810 numeric mismatches in its April–July 2026 production event logs. The chart gives no total for all citations checked.

These results motivate the architectural objection to plain retrieval-augmented generation. Connecting a language model to a vector database can deliver relevant context, but relevance alone does not verify the numerical answer produced from it. A financial research system also needs controls over source identity, the meaning of extracted values, arithmetic and the final response. Retrieval is one component of that system.

The sharpest correction in the talk concerns who owns the guarantee. Ganesh begins to describe the model as structurally incapable of producing hallucinated numbers, then corrects himself: the platform enforces that restriction. The model still hallucinates. The filter and deterministic execution path are intended to prevent its unsupported numerical inventions from reaching the user.

16:4017:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:40 · section reference included

Click through the calculation—and keep the guarantee in scope

The product example asks for Apple’s fiscal 2025 revenue and how quickly it grew. The screen shows net sales of $416,161 million in fiscal 2025 and $391,035 million in fiscal 2024. Dividing the $25,126 million increase by the fiscal 2024 figure gives 6.4% growth after rounding. Clicking the revenue figure highlights the filing entry. Opening the growth calculation shows the two inputs, each with its own source.

An Apple revenue example links displayed figures to filing evidence and the steps of the calculation.Open full source frame
An Apple revenue example links displayed figures to filing evidence and the steps of the calculation.

That inspection path targets a familiar manual workflow. Analysts still extract information from S-1s and move figures from PDFs into Excel because producing a usable financial model requires more than a plausible answer. Kepler’s aim is to automate that transfer and calculation work while preserving the information needed to check it.

The closing guarantees remain specific:

  • Citations: Models select references, and the platform verifies citations before presenting numerical output.
  • Arithmetic: The compute layer performs calculations using deterministic code.
  • Isolation: Tenant and document controls limit which research a user can retrieve.
  • Recall: Kepler measures retrieval recall, acknowledging that finding the needed evidence is a separate concern from checking a value already retrieved.

Those controls do not amount to an autonomous hedge fund or a replacement for everyone making investment decisions. Ganesh expects more “quantamental” work, combining quantitative methods with fundamental research. The intended division of labor leaves people with the investment ideas they want to develop and gives AI the repetitive retrieval and assembly work they would rather avoid. Traceable numerical execution makes that handoff more useful without turning an investment judgment into a guaranteed outcome.

18:3219:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:32 · section reference included

Resources

From the talk

  • Context Is the Easy PartArticle

    Ganesh develops the surrounding data-engineering requirements: provenance, versioning, validation, reproducibility and observability when retrieved context changes.

  • How Kepler Built Verifiable AI for Financial Services

    Develops the companion concepts of atomic provenance, scoped determinism and derivation chains, including how firm-specific processes affect verification.

  • Respect The Process

    Applies a comparable separation to sustainability calculations: agents propose changes through typed tools, while controlled deterministic execution validates consequential operations.

Transcript download
Full transcript

Loading transcript…