AI Can't Make Up a Number It Can't Look Up - Vinoo Ganesh, Kepler
Loading slides…
Read the talk
Who Writes the Number? How Kepler Grounds Financial AI in Facts It Can Look Up
Vinoo Ganesh follows a financial figure from a company filing to an analyst’s screen, showing where a model can exercise judgment and where a database lookup, a calculation or a refusal should decide what the analyst sees.
From a talk by Vinoo Ganesh
At a glance
Ideas worth remembering
Diagnose a wrong number at three separate points: source selection, fact selection within that source and arithmetic. The Palantir example can pass a document-level citation check while failing the reporting-year check.
Give reusable facts identifiers during ingestion. Let the model choose what to retrieve, then use code to fetch values and compute results.
Apply the caller’s document permissions during both candidate search and direct reads by ID. Check index freshness and population when investigating surprising results.
Test a displayed figure in both directions. Follow a valid reference through extraction and calculation, then inspect what the user sees when a reference cannot resolve.
The part a competitor copied
Three months before this appearance, Vinoo Ganesh, CEO and co-founder of Kepler, had argued at AI World’s Fair that institutions needed a way to verify AI-generated work. The day before this session, he had shared a stage with Jack, who had run Citadel’s biggest book and founded Candlestick Capital. Jack still distrusted AI. Meanwhile, by Ganesh’s count, roughly $270 million in venture capital had gone into formal verification layers in those three months.
Ganesh says a venture capitalist even passed Kepler’s pitch deck to a competitor, whose product appeared with the same tagline. What amused him was the part they copied. Kepler does run a verification pass on its output. But by the time that pass encounters a revenue figure, other parts of the system have already determined which documents the user can search, which fact the figure points to and which component gets to write the number. Those earlier decisions make the final check possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An investment firm wants originality and an accountable set of numbers
Ganesh often hears builders of retrieval-augmented generation systems predict that better models will eventually eliminate hallucinations. He sees a complication specific to investing. A firm earns alpha in part because its analysts can look at information available to other investors and arrive at a different judgment. That capacity to propose an unexpected explanation is valuable. The revenue figure an analyst uses to support it, on the other hand, has to survive review. And a model with access to public information alone misses the firm’s own research, which may be precisely where its distinctive judgment resides.
Firms were buying retrieval platforms, building their own systems and running agent pilots. One customer described itself to Ganesh that morning as AI-first on the strength of a single copilot agent. Yet fundamental investing had scarcely undergone the transformation those purchases suggested. Ganesh puts the obstacle in human terms. Before people will stand behind a piece of work, they need to know whom they can hold responsible when it goes wrong.
He cites figures of 80 percent of finance executives trusting numbers and 31 percent finding errors. The product feedback he recounts puts a finer point on the problem. A system demos well and behaves like a smart intern. Give it the right structured information and much of the answer comes back correctly. The reviewer still owns the last ten percent and needs some way to locate the mistake.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways to get Palantir’s revenue wrong
The first failure is choosing the wrong document. Even defining the right one takes judgment. If the question is what a stock will be worth, the firm might want its own internal analysis. It might instead want an estimate from an outside analyst whose work it particularly trusts. Both documents could discuss the correct stock and contain a plausible answer. Retrieval has to take account of whose assessment the user is asking for.
The second failure occurs inside a relevant source. Suppose the question concerns Palantir’s 2023 revenue and the system retrieves its 2025 Form 10-K. The filing includes the earlier year’s figure, so retrieving it is defensible. But the model can pick up the 2025 revenue and cite the filing. A reviewer following that citation will find Palantir, a genuine revenue figure and the correct document. To spot the error, the reviewer has to match the selected figure to its reporting year. A citation filter can check that association after generation, by which point the system has already produced the incorrect claim.
The third failure is arithmetic. A model can select the intended figures and still predict the wrong next token when asked to combine them. Ganesh’s rejoinder to the observation that deterministic models exist is that the CPU already knows how to add two numbers. Once the operands and the operation are known, code can perform the calculation directly. Better document search would do nothing to fix this particular error.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract the fact once, then let the model plan with its ID
Kepler pays the extraction cost before an analyst asks a question. A substantial preprocessing step takes unstructured documents, puts individual facts in context and gives them unique identifiers in a structured store. In the Palantir example, the unit worth identifying is revenue for a particular reporting year. An ID that names only the 10-K would leave every subsequent reader to choose between the two years again.
At query time the model acts as the analyst planning the work. It determines which source to consult, which entity to retrieve and what to do with the result. It refers to facts using pointers. Code retrieves the values and performs the math, and code writes the numbers that appear through this path. This division also changes the economics of repeated questions. Reading a 10-K with a model to extract a figure at search time consumes tokens each time. A fact extracted at ingestion is available for subsequent lookups.
The reader encounters this architecture in a small interface detail. A sentence describing how revenue changed from one amount to another has clickable chips on the figures. The analyst can select an amount and inspect how Kepler obtained it. The connection exists because the system used a reference to produce the figure in the first place. Kepler also runs a post-processing check, the part Ganesh says the competitor noticed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A portfolio manager’s research process suggests where search belongs
As head of business engineering at Citadel, Ganesh worked across the firm on software that could help generate alpha. He saw talented portfolio managers pair their judgment with deterministic tools. A buy-side researcher might build a discounted cash flow model, or DCF, using data already in hand. For information the researcher lacks, the next step could be a GLG or Guidepoint call, finding an expert or speaking with management. Knowing what remains to be learned is part of the skill.
Kepler makes a similar distinction. It stores what it can represent deterministically. For material it cannot handle that way, it preprocesses the information and uses hybrid storage that includes turbopuffer alongside other mechanisms. Reranking helps it choose among possible sources. Once it has a pointer, fetching the associated item can be as ordinary as calling a get method. Run a model-led search ten times with the same query and its choices can vary. Fetch an existing key and the store returns the value under that key. A bad choice of key will still faithfully retrieve the wrong year.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From 200 candidate documents to individual passages
Finding the key is a search problem. Even in a collection of SEC documents, the system has to decide both which source deserves attention and which candidates it can stop considering. Kepler starts with approximate nearest-neighbor search, or ANN, alongside BM25 keyword retrieval. The two methods give it a broad set of candidates, usually up to about 200 documents. Reciprocal rank fusion combines their rankings and narrows the set to roughly 60 documents.
Cross-encoders then score the query together with an individual document chunk. The unit of comparison has changed. A first pass can surface a filing about the right company. The later pass can examine how closely a particular passage bears on the question. This ordering reserves the more expensive joint scoring for material that survived the wider search.
Ganesh contrasts that with letting an LLM direct the entire investigation. Once the model finds a plausible first document, moving forward with it is cheap. Exploring many additional sources costs more even if one of them would better answer the question. Kepler builds breadth into retrieval before asking the system to narrow its attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scope the search and the direct read to the caller
Broad retrieval immediately raises two implementation questions for client documents. Whose OAuth token authorizes the request, and what can that user read? Ganesh identifies five ways the document path can fail. First, omit the tenant filter and a Citadel user could search Millennium’s documents. Second, accept a caller’s claim about which firm they represent and an impersonator can query another firm’s research. Correctly ranking the resulting documents only makes the disclosure more effective.
Third, search can return out-of-date information. The caller may be entitled to see a document that has since been superseded. Fourth, a direct read can skip the rules that governed search. Fetching one document by its ID still requires an access decision. An allowlist applied only when assembling search results leaves that second path exposed. Fifth, a query can hit an index that has yet to be populated. An empty result in that case tells the analyst little about whether the firm possesses the information.
These failures can look much the same from the answer screen. An expected fact is missing, or an unexpected one appears. To investigate, a builder needs to know who authorized the request, which documents were eligible to come back and what was actually indexed. For a firm whose internal analysis helps it generate alpha, constraining the corpus is part of answering the question correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A source hash gives verification something to check
Kepler’s source IDs are hashes. Preprocessing creates the references, and the model can use one to obtain a fact if the referenced item exists in the database. Ganesh’s analogy is a HashMap. Ask for a key that has no entry and the lookup gives you nothing. That outcome matters because it turns an unsupported reference into a condition application code can detect before presenting a sourced figure.
Kepler gives a citation three possible outcomes. It can verify the citation against a stored fact and present the figure with its reference. It can correct the citation during search. Or it can refuse the figure when it cannot resolve the pointer. The guarantee applies to facts in the corpus. Whether a retrieved fact is the most relevant one for this investor still depends on the source and period choices made earlier.
Ganesh suggests the same design for legal precedents. Store the cases that exist and let the model search among them. An invented case name then has no stored precedent to point to. His concern is what happens to adoption when practitioners repeatedly encounter convincing inventions. In his assessment, much financial AI use had settled on formatting PowerPoint presentations, while a trustworthy cradle-to-grave DCF remained out of reach. He argues that firms would have adopted such a system quickly if it already worked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production refusals fell, and generated numbers still needed a check
In Kepler’s production data, a refusal meant that the model could not resolve a pointer and therefore could not produce the requested number. Ganesh reports that refusals dropped sharply once the model had to resolve its own references. He does not say how large the drop was in percentage terms.
Refusals persisted. Ganesh also acknowledges that a probabilistic model can generate a stray number during inference. A lookup can prevent a nonexistent key from supplying a value, but the model is still capable of emitting numeric text. Kepler keeps a final post-processing check for that reason. The structured retrieval path gives that check something concrete against which to test the output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate a vendor by following one displayed number
Ganesh’s proposed vendor test starts at the number an analyst is expected to trust. Click it. If it was extracted, follow it to the source fact and check that the metric and period match the claim. If it was calculated, inspect the source inputs and the arithmetic. A figure can involve both operations, in which case the reader needs to follow the extraction into the calculation. A document link attached after the answer was written leaves that work to the analyst.
Then ask how the application enforces what the interface appears to promise. Does the model reference facts through pointers? Is every citation checked? Is arithmetic checked against its sources? What keeps a client file from reaching the wrong customer? Ganesh saves the most revealing question for last. What happens to a number that came from nowhere? A database lookup returns nothing when the requested value is absent. An analyst should be able to see that refusal just as clearly as the source of a verified figure. When Ganesh says AI is “out of the search phase,” this is the infrastructure he is asking builders to deliver: a way to carry either outcome from the database to the person making the decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…