AI Engineer Summit 2025

Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

Mike Conover· Founder & CEO, Brightwave21:10

Loading slides…

Read the talk

Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

Financial research agents need more than fluent summaries: they need bounded workflows, separate verification, contextual evidence, and interfaces that let analysts inspect consequential findings.

From a talk by Mike Conover

Before you start: Familiarity with language-model prompts, tool calls, and document retrieval will help; no finance specialization is required.

Thousands of pages before the term sheet

Before a term sheet is signed, a diligence team enters a data room containing thousands of pages. Competing teams are examining the same opportunity. The task is to reach conviction quickly while finding the risks that could undermine the asset’s performance. Reading faster helps only if the important exception—a problematic contract, an exposure, a fragile assumption—survives the compression into an investment judgment.

The same pressure appears elsewhere in financial research:

  • Earnings coverage: An analyst may cover 80–120 companies, combining calls, transcripts, and filings to understand developments at both sector and individual-ticker levels.
  • Contract diligence: Reviewers must find early termination clauses across vendor contracts, then recognize how companies throughout a portfolio negotiate those agreements.

These are both document-reading and synthesis problems. A reviewer must identify individual facts without losing the patterns that emerge across the collection.

Investment Research & Due Diligence slide with three illustrated columns covering competitive deal diligence, earnings season, and customer and vendor contract review.Open full source frame
Three investment research tasks: deal diligence, earnings coverage, and contract review.

Mike Conover, founder and CEO of Brightwave, approaches this problem after working at Databricks on Dolly, an early demonstration of instruction tuning for open-source language models. Working with finance professionals added another constraint to the technical problem: junior analysts are routinely assigned more reading and analysis than their deadlines realistically permit. The system must relieve that burden while preserving the judgment the work requires.

0:170:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Automating the work changes the analysis

Consider an accountant in 1978, before computational spreadsheets became commonplace. Running the numbers meant writing them by hand on wide sheets of paper. The work was demanding, important, and time-consuming. Software did not eliminate financial analysis; it changed the sophistication of the questions an analyst could afford to ask. Calculations that once consumed the working day became a foundation for further thought.

Conover reports customer feedback describing orders-of-magnitude improvements in efficiency and time to value from knowledge agents. He presents this as customer experience, without a task-level benchmark. But processing more material creates a new product problem: how do you make the consideration of ten thousand pages useful and legible to a human? A chat transcript is one possible interface, yet it is unlikely to expose everything an analyst needs to inspect.

2:122:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:12 · section reference included

Local plausibility is not workflow reliability

Conover characterizes non-reasoning models as performing greedy local search: an answer can look plausible at the current step without serving the eventual research objective. His concrete example is asking GPT-4o to extract organizations from a Reuters article and receiving products among the results. If the next step treats every extracted item as a company, that classification error becomes an input to further research.

Conover uses illustrative per-call error rates of 5–10% to explain how mistakes accumulate when calls are chained. These are not measured Reuters extraction results. The mechanism is dependency: later decisions inherit earlier mistakes, and a fluent final answer does not reveal which intermediate assumption failed.

His proposed direction is end-to-end reinforcement learning over tool use, with API results included in the sequence being optimized. That would allow an action that looks locally unhelpful to earn credit when it produces a better final result. At the time of the talk, OpenAI’s deep research launch account already described end-to-end RL for browsing and reasoning. The unresolved question here is broader: how to obtain reliably useful decisions across arbitrary tools, knowledge graphs, and financial research tasks.

A product still has to work before that general problem is solved. The Bitter Lesson supplies the pressure toward more compute, more data, and more capable models. Conover contrasts that direction with hand-built expert systems and dedicated named-entity recognition through tools such as spaCy. His practical answer is to constrain the agent’s scope of behavior. Treat that scope like a regularization parameter: fewer permitted behaviors reduce the complexity of the workflow and the opportunities for it to produce degenerate output.

Slide titled Non-Reasoning Models Perform Greedy Search, with text about RL with tool use and regularization beside a blue and yellow branching graphic.Open full source frame
Greedy search, reasoning over tool use, and workflow regularization.
4:044:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:04 · section reference included

Move conversational steering into the product

An effective multi-turn conversation follows a branching tree. The model responds; the user chooses a reaction; that reaction changes the next response. Skilled users learn which branches to pursue and which corrections move the model toward a useful answer. Professionals with demanding jobs may never invest in developing that skill.

Conover offers an activation-space interpretation of this process. The accumulated conversation tokens determine the model’s activations. A correction changes those activations, nudging the computation toward a state that can solve the problem. He interprets a reasoning monologue as serving a similar function: generating intermediate text can help establish the conditions for a better answer. In that account, the useful outcome is reaching the right computational state; readable intermediate prose is not itself the objective.

Conover estimates that deep prompting expertise can easily take a thousand hours. His product implication is that vertical workflows specify intent. Scaffolding can supply the decomposition, corrections, and constraints that an expert user would otherwise have to discover through conversation. That removes work from the user while giving the system a more precise task.

6:016:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:01 · section reference included

Distill findings, then check them separately

Suppose the research question is how a polypropylene resin manufacturer manages costs. Start by decomposing what an analyst would do:

  1. Find public-market comparables. Examine SEC filings and earnings-call transcripts for relevant companies.
  2. Bring in institutional knowledge. Consult a knowledge graph built from previous deals.
  3. Select relevant news and documents. Determine which collections bear on the cost-management question.
  4. Distill findings. Extract evidence that supports or challenges premises in the investment thesis.
  5. Enrich and correct those findings. Resolve gaps and errors before combining them into an account of the business.

This decomposition gives each stage a bounded purpose and an output the next stage can inspect.

After the first pass, intermediate notes should record what the system believes on the basis of what it found. Those notes turn a collection of passages into explicit provisional conclusions. Individual findings can then be enriched, rather than repeatedly asking the model to reconsider the entire corpus without a clear target.

Verification needs a specific question. For a factual claim, ask whether the source document entails it. For the Reuters extraction example, ask whether the extracted item is actually an organization. Conover reports that a separate verification call works better than asking the original call to check itself within its JSON output. His explanation is that the original generation is inclined to affirm what it has just asserted; a secondary call separates producing a candidate from evaluating it.

Only after those checks does synthesis weave findings from many documents into a coherent narrative. The architecture therefore distinguishes three operations that fluent text can otherwise blur together: proposing a finding, checking its support, and explaining how it relates to other findings.

7:588:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:58 · section reference included

Keep human judgment; compose simple tools

The research loop must remain open to human direction. An analyst may choose an interesting thread, add a directive, or redirect attention because of information that has never been digitized. A conversation with management may change the interpretation of a filing. A portfolio manager’s skepticism about a particular class of biotech may change which hypotheses deserve investigation. Analyst taste is therefore an input to the system, not merely a final approval step.

That does not require assigning human job titles to every component. Calling one node a portfolio-manager agent and another a fact-checker agent can make the architecture harder to change when the computational needs shift. Conover instead reaches for the Unix philosophy: simple tools that do one thing well, work together, and communicate through text. Bash pipes and tee illustrate the compositional idea. Organize nodes around the operations they perform, so their boundaries can change without having to preserve an imagined organizational chart.

Slide stating Anthropomorphizing compute creates negative value, above three diagrams illustrating Unix principles for simple programs, collaboration, and text streams.Open full source frame
Unix philosophy: do one thing well, work together, and handle text streams.
10:0210:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:02 · section reference included

Latency also slows the user’s learning

A price–performance plot from Latent Space introduces another constraint on the compute graph. The Pareto frontier may move outward as models improve, but a frontier still exists: each node needs an appropriate tool, system, or model. The best choice for synthesizing difficult evidence need not be the best choice for a narrow intermediate operation.

Adding research, verification, and error correction can produce the latency trap. It is tempting to assume that a workflow taking twenty-five minutes will deliver an excellent report. Conover points to the uncertainty users can feel even with a capable product such as OpenAI’s deep research: until the report arrives, they do not know whether its value will justify the wait. More elapsed computation does not by itself resolve that uncertainty.

There is also a learning cost. A user has an expected report in mind, receives the actual report, and updates their understanding of how a prompt produces behavior. Conover treats the gap between expectation and result as a loss signal for the user’s mental model. His illustrative eight- or twenty-minute feedback loops allow relatively few repetitions in a day. Slow feedback therefore limits not just throughput but the user’s opportunity to become proficient with the product.

11:2411:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:24 · section reference included

Give important subthemes their own output budget

A large input window does not guarantee a correspondingly detailed answer. Conover asks whether anyone in the audience has received a fifty-thousand-token response, then questions how a claimed hundred-thousand-token output capacity translates into ordinary use. He hypothesizes that instruction-tuning demonstrations have characteristic lengths, influencing how much prose a model tends to produce. Writing a very long, coherent, novel answer is itself difficult.

Conover reports typical o1 responses of roughly 2,000–3,000 tokens in his experience, describing them as longer than GPT-4o’s. That observation concerns generated responses, not a formal capacity limit. Current o1 documentation lists a 200,000-token context window and a 100,000-token maximum output budget; the reasoning guide explains that output budgets include hidden reasoning tokens as well as visible output. Those specifications do not establish how much visible prose the talk’s requests would produce.

The practical problem is compression. A model can receive a huge body of evidence and still compress it into a short answer. Compare a book report with a synopsis of every chapter:

Request structureWhere the available detail goes
One broad reportA small number of overarching themes
Separate chapter or subtheme analysesFocused treatment of each selected topic

The second structure lets the workflow deliberately allocate attention and output to areas that would otherwise disappear into a general summary.

Conover’s example is an analysis of the global financial crisis that gives the rise of shadow banking only three sentences. The remedy is to decompose the research instructions into narrower subthemes. A dedicated analysis of shadow banking can spend its available output on that mechanism, rather than competing with every other cause and consequence of the crisis. Decomposition controls what survives compression.

13:0013:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:00 · section reference included

Cross-document synthesis needs contextual evidence

Conover also suspects that instruction-tuning and post-training corpora contain relatively few demonstrations of recombinative reasoning. Writing alternative epilogues for The Great Gatsby amortizes the cost of understanding one book: read it once, then produce several variations. That is different from learning enough about many independent documents to say something useful about their combined evidence.

Biomedical literature synthesis makes the distinction concrete. The task is to read multiple papers and connect their findings, not simply summarize each one independently. Conover suggests generating a paper’s abstract from its bibliography as a possible post-training exercise. The idea targets evidence recombination, although producing thoughtful analysis across many documents remains difficult in practice.

Temporality makes that difficulty sharper. Pro forma financial statements associated with a merger or acquisition describe a different basis from statements before the event. A contract addendum can change how an earlier clause should be interpreted. Passages that look contradictory in isolation may instead describe different periods, scopes, or contractual states.

The evidence passed into synthesis should therefore include metadata explaining why a passage matters and how it relates to other evidence. For example, a contract-review record can preserve the original clause and the amendment relationship together:

json

{
  "question": "What early termination notice applies?",
  "evidence": [
    {
      "id": "contract-1",
      "passage": "Either party may terminate on 30 days' notice.",
      "document_type": "original_contract"
    },
    {
      "id": "addendum-1",
      "passage": "The termination notice period is amended to 60 days.",
      "document_type": "contract_addendum",
      "amends": "contract-1"
    }
  ],
  "review_status": "pending",
  "review_instruction": "Confirm the addendum applies before resolving the notice period."
}

In this illustrative record, the review is still pending. Keeping both passages and the amends relationship gives the next operation a way to reason about their interaction rather than treating them as interchangeable snippets. This is the purpose of propagating contextual metadata alongside evidentiary text.

14:3614:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:36 · section reference included

Make every finding interrogable

The interface question returns once the system has considered thousands of pages: how can a person recognize what deserves attention? Conover compares the experience to Facebook or LinkedIn connection recommendations. In a six-by-six grid of faces, a familiar person can stand out even if many suggestions are irrelevant. The useful design principle is to make recognition easy, then provide details on demand.

In the Brightwave demonstration, clicking a report citation provides more than the identity of a source document. It supplies context for interpreting the evidence. Structured interactive outputs also let the analyst pursue a specific issue, such as rising capital expenditure, without restarting the research from a broad prompt.

That interaction extends beyond citation markers. A reader can highlight any passage and ask for more information or its implications. Conover relates this to OpenAI Canvas and its ability to act on selected text, including changing a passage’s reading level. The broader requirement is a continuous research surface: any finding that attracts attention should be available for further questioning.

16:2916:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:29 · section reference included

The report is one view of the findings

A report is a convenient entry point into a larger structure. The system has read documents, developed a provisional view, and connected facts. Conover describes the resulting findings as a high-dimensional data structure, with the report representing just one view. An analyst should be able to turn that structure around and inspect its support.

Opening source documents is one level of auditability. Exposing an inventory of findings provides another. A fundraising timeline or an ongoing litigation item may attract attention even when it is not prominent in the narrative. From that inventory, the analyst can select a finding and investigate it further.

The examples become consequential quickly: patent litigation, or a factory fire in Mexico that destroys a critical single-source supplier. Either can change an investment assessment. The analyst’s taste determines which thread matters; the product provides a magnifying glass for the text and a route to additional detail. Conover explicitly acknowledges that Brightwave has not yet reached the final form of this class of product. The demonstrated direction is nevertheless clear: preserve access to the findings beneath the narrative, so the reader can investigate what the summary did not foreground.

Brightwave document findings table with an open detail popover describing an acquisition and showing Tell Me More and Implications buttons.Open full source frame
A finding opens into supporting detail with Tell Me More and Implications controls.
18:3618:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:36 · section reference included

A recruiting close with recursive incentives

The recording closes with a QR-code recruiting appeal for a small, experienced team. Conover names backgrounds at Goldman Sachs, UBS, Meta, Instagram, and Anaplan, along with a newly hired senior staff software engineer from Brave. He emphasizes staff and senior staff hiring, particularly product design and front-end engineering.

At the time of the talk, Conover advertised a $10,000 referral bonus for the listed roles and $1,000 for referring the person who referred the eventual hire. He compares the structure to MIT’s winning DARPA Red Balloon Challenge strategy: reward recruitment through a network, not only the person who supplies the final lead. He mentions further levels in the referral tree without specifying their payouts.

19:5220:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:52 · section reference included

Resources

From the talk

Updates since the talk

  • Current reference for o1's context window, maximum output budget and supported capabilities.

Transcript download
Full transcript

Loading transcript…