# Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

Mike Conover, Brightwave

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/MWTJIAwAAnk-trust-but-verify-high-fidelity-reasoning) · [Video](https://www.youtube.com/watch?v=MWTJIAwAAnk)

## Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

Financial research agents need more than fluent summaries: they need bounded workflows, separate verification, contextual evidence, and interfaces that let analysts inspect consequential findings.

Familiarity with language-model prompts, tool calls, and document retrieval will help; no finance specialization is required.

### Thousands of pages before the term sheet

[0:17](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=17s)

Before a term sheet is signed, a diligence team enters a data room containing thousands of pages. Competing teams are examining the same opportunity. The task is to reach conviction quickly while finding the risks that could undermine the asset’s performance. Reading faster helps only if the important exception—a problematic contract, an exposure, a fragile assumption—survives the compression into an investment judgment.

The same pressure appears elsewhere in financial research:

- **Earnings coverage:** An analyst may cover 80–120 companies, combining calls, transcripts, and filings to understand developments at both sector and individual-ticker levels.
- **Contract diligence:** Reviewers must find early termination clauses across vendor contracts, then recognize how companies throughout a portfolio negotiate those agreements.

These are both document-reading and synthesis problems. A reviewer must identify individual facts without losing the patterns that emerge across the collection.

![Investment Research & Due Diligence slide with three illustrated columns covering competitive deal diligence, earnings season, and customer and vendor contract review.](https://ai.engineer/api/talk-essay-media/9cde2ff35a719747a887b5c8c06db25051ef534562ff4c32bcd8977bdc0a67ac)

Three investment research tasks: deal diligence, earnings coverage, and contract review.

Mike Conover, founder and CEO of Brightwave, approaches this problem after working at Databricks on **[Dolly](https://github.com/databrickslabs/dolly)**, an early demonstration of instruction tuning for open-source language models. Working with finance professionals added another constraint to the technical problem: junior analysts are routinely assigned more reading and analysis than their deadlines realistically permit. The system must relieve that burden while preserving the judgment the work requires.

### Automating the work changes the analysis

[2:12](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=132s)

Consider an accountant in 1978, before computational spreadsheets became commonplace. Running the numbers meant writing them by hand on wide sheets of paper. The work was demanding, important, and time-consuming. Software did not eliminate financial analysis; it changed the sophistication of the questions an analyst could afford to ask. Calculations that once consumed the working day became a foundation for further thought.

Conover reports customer feedback describing orders-of-magnitude improvements in efficiency and time to value from knowledge agents. He presents this as customer experience, without a task-level benchmark. But processing more material creates a new product problem: how do you make the consideration of ten thousand pages useful and legible to a human? A chat transcript is one possible interface, yet it is unlikely to expose everything an analyst needs to inspect.

### Local plausibility is not workflow reliability

[4:04](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=244s)

Conover characterizes non-reasoning models as performing **greedy local search**: an answer can look plausible at the current step without serving the eventual research objective. His concrete example is asking **GPT-4o** to extract organizations from a Reuters article and receiving products among the results. If the next step treats every extracted item as a company, that classification error becomes an input to further research.

Conover uses illustrative per-call error rates of 5–10% to explain how mistakes accumulate when calls are chained. These are not measured Reuters extraction results. The mechanism is dependency: later decisions inherit earlier mistakes, and a fluent final answer does not reveal which intermediate assumption failed.

His proposed direction is **end-to-end reinforcement learning over tool use**, with API results included in the sequence being optimized. That would allow an action that looks locally unhelpful to earn credit when it produces a better final result. At the time of the talk, OpenAI’s [deep research launch account](https://openai.com/index/introducing-deep-research/) already described end-to-end RL for browsing and reasoning. The unresolved question here is broader: how to obtain reliably useful decisions across arbitrary tools, knowledge graphs, and financial research tasks.

A product still has to work before that general problem is solved. [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf) supplies the pressure toward more compute, more data, and more capable models. Conover contrasts that direction with hand-built expert systems and dedicated named-entity recognition through tools such as [spaCy](https://spacy.io/usage/linguistic-features). His practical answer is to constrain the agent’s scope of behavior. Treat that scope like a **regularization parameter**: fewer permitted behaviors reduce the complexity of the workflow and the opportunities for it to produce degenerate output.

![Slide titled Non-Reasoning Models Perform Greedy Search, with text about RL with tool use and regularization beside a blue and yellow branching graphic.](https://ai.engineer/api/talk-essay-media/f0e57c44d1c2a8f632eb9d68649b44420289b6f2a0a6b6869ca2a7b8f94b5034)

Greedy search, reasoning over tool use, and workflow regularization.

### Move conversational steering into the product

[6:01](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=361s)

An effective multi-turn conversation follows a branching tree. The model responds; the user chooses a reaction; that reaction changes the next response. Skilled users learn which branches to pursue and which corrections move the model toward a useful answer. Professionals with demanding jobs may never invest in developing that skill.

Conover offers an activation-space interpretation of this process. The accumulated conversation tokens determine the model’s activations. A correction changes those activations, nudging the computation toward a state that can solve the problem. He interprets a reasoning monologue as serving a similar function: generating intermediate text can help establish the conditions for a better answer. In that account, the useful outcome is reaching the right computational state; readable intermediate prose is not itself the objective.

Conover estimates that deep prompting expertise can easily take a thousand hours. His product implication is that **vertical workflows specify intent**. Scaffolding can supply the decomposition, corrections, and constraints that an expert user would otherwise have to discover through conversation. That removes work from the user while giving the system a more precise task.

### Distill findings, then check them separately

[7:58](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=478s)

Suppose the research question is how a polypropylene resin manufacturer manages costs. Start by decomposing what an analyst would do:

1. **Find public-market comparables.** Examine SEC filings and earnings-call transcripts for relevant companies.
2. **Bring in institutional knowledge.** Consult a knowledge graph built from previous deals.
3. **Select relevant news and documents.** Determine which collections bear on the cost-management question.
4. **Distill findings.** Extract evidence that supports or challenges premises in the investment thesis.
5. **Enrich and correct those findings.** Resolve gaps and errors before combining them into an account of the business.

This decomposition gives each stage a bounded purpose and an output the next stage can inspect.

After the first pass, **intermediate notes** should record what the system believes on the basis of what it found. Those notes turn a collection of passages into explicit provisional conclusions. Individual findings can then be enriched, rather than repeatedly asking the model to reconsider the entire corpus without a clear target.

Verification needs a specific question. For a factual claim, ask whether the source document entails it. For the Reuters extraction example, ask whether the extracted item is actually an organization. Conover reports that a separate verification call works better than asking the original call to check itself within its JSON output. His explanation is that the original generation is inclined to affirm what it has just asserted; a secondary call separates producing a candidate from evaluating it.

Only after those checks does synthesis weave findings from many documents into a coherent narrative. The architecture therefore distinguishes three operations that fluent text can otherwise blur together: proposing a finding, checking its support, and explaining how it relates to other findings.

### Keep human judgment; compose simple tools

[10:02](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=602s)

The research loop must remain open to human direction. An analyst may choose an interesting thread, add a directive, or redirect attention because of information that has never been digitized. A conversation with management may change the interpretation of a filing. A portfolio manager’s skepticism about a particular class of biotech may change which hypotheses deserve investigation. Analyst taste is therefore an input to the system, not merely a final approval step.

That does not require assigning human job titles to every component. Calling one node a portfolio-manager agent and another a fact-checker agent can make the architecture harder to change when the computational needs shift. Conover instead reaches for the Unix philosophy: simple tools that do one thing well, work together, and communicate through text. Bash pipes and `tee` illustrate the compositional idea. Organize nodes around the operations they perform, so their boundaries can change without having to preserve an imagined organizational chart.

![Slide stating Anthropomorphizing compute creates negative value, above three diagrams illustrating Unix principles for simple programs, collaboration, and text streams.](https://ai.engineer/api/talk-essay-media/4bdbadbc7b73af7f1719a85591127e73321adbb422860af88bfcdce0e8df995d)

Unix philosophy: do one thing well, work together, and handle text streams.

### Latency also slows the user’s learning

[11:24](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=684s)

A price–performance plot from Latent Space introduces another constraint on the compute graph. The **Pareto frontier** may move outward as models improve, but a frontier still exists: each node needs an appropriate tool, system, or model. The best choice for synthesizing difficult evidence need not be the best choice for a narrow intermediate operation.

Adding research, verification, and error correction can produce the **latency trap**. It is tempting to assume that a workflow taking twenty-five minutes will deliver an excellent report. Conover points to the uncertainty users can feel even with a capable product such as OpenAI’s deep research: until the report arrives, they do not know whether its value will justify the wait. More elapsed computation does not by itself resolve that uncertainty.

There is also a learning cost. A user has an expected report in mind, receives the actual report, and updates their understanding of how a prompt produces behavior. Conover treats the gap between expectation and result as a loss signal for the user’s mental model. His illustrative eight- or twenty-minute feedback loops allow relatively few repetitions in a day. Slow feedback therefore limits not just throughput but the user’s opportunity to become proficient with the product.

### Give important subthemes their own output budget

[13:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=780s)

A large input window does not guarantee a correspondingly detailed answer. Conover asks whether anyone in the audience has received a fifty-thousand-token response, then questions how a claimed hundred-thousand-token output capacity translates into ordinary use. He hypothesizes that instruction-tuning demonstrations have characteristic lengths, influencing how much prose a model tends to produce. Writing a very long, coherent, novel answer is itself difficult.

Conover reports typical **o1** responses of roughly 2,000–3,000 tokens in his experience, describing them as longer than GPT-4o’s. That observation concerns generated responses, not a formal capacity limit. Current [o1 documentation](https://developers.openai.com/api/docs/models/o1) lists a 200,000-token context window and a 100,000-token maximum output budget; the [reasoning guide](https://developers.openai.com/api/docs/guides/reasoning) explains that output budgets include hidden reasoning tokens as well as visible output. Those specifications do not establish how much visible prose the talk’s requests would produce.

The practical problem is **compression**. A model can receive a huge body of evidence and still compress it into a short answer. Compare a book report with a synopsis of every chapter:

| Request structure | Where the available detail goes |
|---|---|
| One broad report | A small number of overarching themes |
| Separate chapter or subtheme analyses | Focused treatment of each selected topic |

The second structure lets the workflow deliberately allocate attention and output to areas that would otherwise disappear into a general summary.

Conover’s example is an analysis of the global financial crisis that gives the rise of shadow banking only three sentences. The remedy is to decompose the research instructions into narrower subthemes. A dedicated analysis of shadow banking can spend its available output on that mechanism, rather than competing with every other cause and consequence of the crisis. **Decomposition controls what survives compression.**

### Cross-document synthesis needs contextual evidence

[14:36](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=876s)

Conover also suspects that instruction-tuning and post-training corpora contain relatively few demonstrations of **recombinative reasoning**. Writing alternative epilogues for The Great Gatsby amortizes the cost of understanding one book: read it once, then produce several variations. That is different from learning enough about many independent documents to say something useful about their combined evidence.

Biomedical literature synthesis makes the distinction concrete. The task is to read multiple papers and connect their findings, not simply summarize each one independently. Conover suggests generating a paper’s abstract from its bibliography as a possible post-training exercise. The idea targets evidence recombination, although producing thoughtful analysis across many documents remains difficult in practice.

Temporality makes that difficulty sharper. Pro forma financial statements associated with a merger or acquisition describe a different basis from statements before the event. A contract addendum can change how an earlier clause should be interpreted. Passages that look contradictory in isolation may instead describe different periods, scopes, or contractual states.

The evidence passed into synthesis should therefore include metadata explaining why a passage matters and how it relates to other evidence. For example, a contract-review record can preserve the original clause and the amendment relationship together:

```json
{
  "question": "What early termination notice applies?",
  "evidence": [
    {
      "id": "contract-1",
      "passage": "Either party may terminate on 30 days' notice.",
      "document_type": "original_contract"
    },
    {
      "id": "addendum-1",
      "passage": "The termination notice period is amended to 60 days.",
      "document_type": "contract_addendum",
      "amends": "contract-1"
    }
  ],
  "review_status": "pending",
  "review_instruction": "Confirm the addendum applies before resolving the notice period."
}
```

In this illustrative record, the review is still pending. Keeping both passages and the `amends` relationship gives the next operation a way to reason about their interaction rather than treating them as interchangeable snippets. This is the purpose of propagating contextual metadata alongside evidentiary text.

### Make every finding interrogable

[16:29](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=989s)

The interface question returns once the system has considered thousands of pages: how can a person recognize what deserves attention? Conover compares the experience to Facebook or LinkedIn connection recommendations. In a six-by-six grid of faces, a familiar person can stand out even if many suggestions are irrelevant. The useful design principle is to make recognition easy, then provide **details on demand**.

In the Brightwave demonstration, clicking a report citation provides more than the identity of a source document. It supplies context for interpreting the evidence. Structured interactive outputs also let the analyst pursue a specific issue, such as rising capital expenditure, without restarting the research from a broad prompt.

That interaction extends beyond citation markers. A reader can highlight any passage and ask for more information or its implications. Conover relates this to [OpenAI Canvas](https://openai.com/index/introducing-canvas/) and its ability to act on selected text, including changing a passage’s reading level. The broader requirement is a continuous research surface: any finding that attracts attention should be available for further questioning.

### The report is one view of the findings

[18:36](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1116s)

A report is a convenient entry point into a larger structure. The system has read documents, developed a provisional view, and connected facts. Conover describes the resulting findings as a high-dimensional data structure, with the report representing just one view. An analyst should be able to turn that structure around and inspect its support.

Opening source documents is one level of auditability. Exposing an inventory of findings provides another. A fundraising timeline or an ongoing litigation item may attract attention even when it is not prominent in the narrative. From that inventory, the analyst can select a finding and investigate it further.

The examples become consequential quickly: patent litigation, or a factory fire in Mexico that destroys a critical single-source supplier. Either can change an investment assessment. The analyst’s taste determines which thread matters; the product provides a magnifying glass for the text and a route to additional detail. Conover explicitly acknowledges that Brightwave has not yet reached the final form of this class of product. The demonstrated direction is nevertheless clear: preserve access to the findings beneath the narrative, so the reader can investigate what the summary did not foreground.

![Brightwave document findings table with an open detail popover describing an acquisition and showing Tell Me More and Implications buttons.](https://ai.engineer/api/talk-essay-media/1f3a6e121db80d045bae49ceb2e819080f26596d9df61e8f976e4645a8ec985c)

A finding opens into supporting detail with Tell Me More and Implications controls.

### A recruiting close with recursive incentives

[19:52](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1192s)

The recording closes with a QR-code recruiting appeal for a small, experienced team. Conover names backgrounds at Goldman Sachs, UBS, Meta, Instagram, and Anaplan, along with a newly hired senior staff software engineer from Brave. He emphasizes staff and senior staff hiring, particularly product design and front-end engineering.

At the time of the talk, Conover advertised a $10,000 referral bonus for the listed roles and $1,000 for referring the person who referred the eventual hire. He compares the structure to [MIT’s winning DARPA Red Balloon Challenge strategy](https://news.mit.edu/2009/red-balloon-challenge-1211): reward recruitment through a network, not only the person who supplies the final lead. He mentions further levels in the referral tree without specifying their payouts.

## Resources

- [Databricks Dolly](https://github.com/databrickslabs/dolly) — Instruction-following models with training and inference examples, dataset details and documented limitations.
- [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf) — Rich Sutton's essay on why general methods that scale with computation tend to prevail.
- [spaCy linguistic features](https://spacy.io/usage/linguistic-features) — Documentation for linguistic processing, including named-entity recognition.
- [Introducing deep research](https://openai.com/index/introducing-deep-research/) — OpenAI's launch account of multi-step web research, tool-use training and expected completion times.
- [o1 model specifications](https://developers.openai.com/api/docs/models/o1) — Current reference for o1's context window, maximum output budget and supported capabilities.
- [Introducing canvas](https://openai.com/index/introducing-canvas/) — The original Canvas interface announcement, including selected-text editing and reading-level controls.
- [MIT's Red Balloon Challenge incentive design](https://news.mit.edu/2009/red-balloon-challenge-1211) — How MIT recruited a distributed search network using rewards for both balloon finders and their recruiters.

## Slides

### Investment Research & Due Diligence

Brightwave’s research agent addresses three document-heavy tasks: competitive deal diligence, earnings analysis, and contract review. Each requires finding consequential details across large collections of financial content.

[0:32](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=32s) [Three investment research use cases: deal due diligence, earnings season, and customer and vendor contract review.](https://assets.ai.engineer/sites/aiecode2025/sha256/66e51ae9d92ff3227ebd4cbe0c0e705917fd755948627d704525f232e4f232e8.webp)

Brightwave

Investment Research & Due Diligence

Due Diligence in a Competitive Deal Process
High stakes decision making based on fact patterns spread across huge collections of interrelated documents.

Crush of Earnings Season
Analysts must act on subtle changes in management guidance across an expansive coverage universe.

Customer & Vendor Contract Review
Identify material risks to critical supply chain inputs or non-market clauses across thousands of pages of customer contracts.

### Non-Reasoning Models Perform Greedy Search

The speaker argues that reinforcement learning over tool calls can make locally suboptimal choices for better global outputs, while workflow specificity limits exploration.

[4:09](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=249s) [Slide discussing reasoning over tool use and workflow specificity, alongside an abstract blue and yellow illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b15ce50929689a81334ae4d77a88cdac0c9c30237a3ccfbe4dd030795c6d824.webp)

Brightwave

Non-Reasoning Models Perform Greedy Search
(and have generally poor judgement)

RL with Tool Use. Winning systems will perform reasoning over tool use to achieve globally superior outputs.

Calculation, public sources, knowledge graph, etc.

Regularization. Workflow specificity acts a control against exploring useless parts of the state space.

### Basic Knowledge Agent

The graph routes subqueries to a knowledge graph, data room and filings. Evidence passes through calculation and enrichment/error correction to synthesis, with human oversight at control disposition.

[7:51](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=471s) [Slide showing a knowledge agent workflow from subquery generation and source routing through evidence, calculation, enrichment, correction, synthesis and human oversight.](https://assets.ai.engineer/sites/aiecode2025/sha256/fecb0dcb5e4914a284622c2b06877da12dc24c9f05b7b95f9fd3582424de3fad.webp)

Basic Knowledge Agent
Semi-autonomous Compute Graph

Brightwave

Mimic Human Decision Processes

A private plastic packaging maker claims stable polypropylene resin costs, citing supplier invoices and a multi-year LyondellBasell contract. How does this compare to Berry Global or Amcor’s disclosed feedstock and logistics challenges?

Enrichment
Error Correction
Synthesis

Subquery Generation
Subqueries
Index Routing
Knowledge Graph
Data Room
Filings
Evidence
Calculation
Enrichment
Error Correction
Synthesis
Human Oversight
Control Disposition

### Basic Knowledge Agent

The agent breaks a resin-cost question into subqueries, routes them to a knowledge graph, data room and filings, then combines evidence through calculation, correction and synthesis with human oversight.

[8:23](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=503s) [Slide describing a semi-autonomous knowledge agent, with an example question and a compute graph from subquery generation through synthesis and human oversight.](https://assets.ai.engineer/sites/aiecode2025/sha256/81fe74a769ac3deeccf70c950e3b36584ba5ebef8eadd1fc802201b107286f2d.webp)

Brightwave

Basic Knowledge Agent
Semi-autonomous Compute Graph

Mimic Human Decision Processes

A private plastic packaging maker claims stable polypropylene resin costs, citing supplier invoices and a multi-year LyondellBasell contract. How does this compare to Berry Global or Amcor’s disclosed feedstock and logistics challenges?

Error Correction
Synthesis

Subquery Generation
Subqueries
Index Routing
Knowledge Graph
Data Room
Filings
Evidence
Enrichment
Calculation
Error Correction
Synthesis
Human Oversight
Control Disposition

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Anthropomorphizing Compute Creates Negative Value

Giving compute-graph nodes human roles can constrain flexibility as design needs change. The Unix philosophy favors focused programs that work together and use text streams as a universal interface.

[10:39](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=639s) [Three Unix principles advocate focused programs, cooperation, and text streams as a universal interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/8992994fa013340181beff33d166987e759bc8eb012691d270f484a392939346.webp)

Brightwave

Anthropomorphizing compute creates negative value.

Unix Philosophy (1978, Bell Labs Technical Journal)

Write programs to do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.

### Anthropomorphizing Compute Creates Negative Value

The slide pairs focused programs with cooperation through text streams. The speaker calls for choosing a tool, system, or model for each node in a compute graph.

[11:51](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=711s) [The Unix philosophy slide returns, showing three principles for programs and text interfaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/58df80ea2e15c110f932f05cbfc9bcfa12203de3ca3fa545fb671b2ec14542d3.webp)

Brightwave

Anthropomorphizing compute creates negative value.

Unix Philosophy (1978, Bell Labs Technical Journal)

Write programs to do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.

Interface text:
AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Synthesis

A single report compresses a large input. The speaker proposes decomposing research into themes, then combining them into a report, so each theme can receive a more focused analysis.

[13:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=780s) [A synthesis slide pairs an example financial-crisis report and problem-decomposition diagrams with a statement about the length and complexity of post-training demonstrations. The presenter partially covers one diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/513fb3f55ab58d6b817f1734ee855cbd57f65cb7ef47dfc3618bf3ab55f78b03.webp)

Synthesis

PROBLEM DECOMPOSITION

2.4. Deregulation in the 1990s: The Seeds of Trouble

The 1990s witnessed continued deregulation efforts, especially in the United States. Three major moments are frequently highlighted by economic historians:

1. Repeal of Glass-Steagall (1999): Enacted during the Great Depression, the Glass-Steagall Act had separated commercial banking (taking deposits, making loans) from investment banking (underwriting securities, engaging in trading). Its partial repeal through the Gramm-Leach-Bliley Act opened the door for the formation of financial conglomerates—so-called universal banks—that could engage in a wide range of financial activities. This increased the complexity of large financial institutions and, as critics argue, potentially amplified systemic risk.
2. Commodities Futures Modernization Act (2000): This legislation effectively exempted many over-the-counter (OTC) derivatives, including credit default swaps, from direct regulatory oversight. As a result, massive derivatives positions could accumulate out of sight of regulators.
3. Rise of the Shadow Banking System: Non-bank financial entities—investment banks, hedge funds, money market funds—took on roles resembling traditional banking but were not subject to the same reserve requirements or oversight. They relied heavily on short-term funding (repo markets) and engaged in maturity transformation (borrowing short-term funds to invest in longer-term assets). This structure made them susceptible to “bank runs” in a crisis.

Diagram labels: Report; Theme; Theme; ..; Theme; Report
Omnibus

CHARACTERISTIC OUTPUT
There aren’t many 20,000 token post-training demonstrations, and even fewer that deeply reason across heterogeneous inputs.

Length
Complexity

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

[13:39](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=819s) [A synthesis slide contrasts report generation with theme-based decomposition and discusses output length and complexity.](https://assets.ai.engineer/sites/aiecode2025/sha256/32267ff891eebdd92bca27e9a524f8485777dc0d45e9aea07935a7d892ad868e.webp)

Synthesis

PROBLEM DECOMPOSITION

2.4. Deregulation in the 1990s: The Seeds of Trouble

The 1990s witnessed continued deregulation efforts, especially in the United States. Three major moments are frequently highlighted by economic historians:

1. Repeal of Glass-Steagall (1999): Enacted during the Great Depression, the Glass-Steagall Act had separated commercial banking (taking deposits, making loans) from investment banking (underwriting securities, engaging in trading). Its partial repeal through the Gramm-Leach-Bliley Act opened the door for the formation of financial conglomerates—so-called universal banks—that could engage in a wide range of financial activities. This increased the complexity of large financial institutions and, as critics argue, potentially amplified systemic risk.
2. Commodities Futures Modernization Act (2000): This legislation effectively exempted many over-the-counter (OTC) derivatives, including credit default swaps, from direct regulatory oversight. As a result, massive derivatives positions could accumulate out of sight of regulators.
3. Rise of the Shadow Banking System: Non-bank financial entities—investment banks, hedge funds, money market funds—took on roles resembling traditional banking but were not subject to the same reserve requirements or oversight. They relied heavily on short-term funding (repo markets) and engaged in maturity transformation (borrowing short-term funds to invest in longer-term assets). This structure made them susceptible to “bank runs” in a crisis.

Diagram labels: Report; Theme; Theme; ..; Theme; Report
Omnibus

CHARACTERISTIC OUTPUT
There aren’t many 20,000 token post-training demonstrations, and even fewer that deeply reason across heterogeneous inputs.

Length
Complexity

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Tracing a finding to its source

Brightwave links a $10 million EcoCharge finding to a stockholder-consent excerpt, letting the analyst inspect the source behind the risk explanation.

[19:32](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1172s) [Brightwave action history displays a risk explanation with a source-document excerpt.](https://assets.ai.engineer/sites/aiecode2025/sha256/765506037950c3d1f90b53f05813462006df03ae4aa60868691633d4156775dc.webp)

company's financial resources, especially if the product line does not achieve expected market success. The need for stockholder approval underlines the financial significance of this initiative. If the EcoCharge line fails to generate sufficient revenue, it could impact the company's profitability and limit its ability to invest in other strategic areas.

EcoTech Solutions' decision to allocate $10 million 1 for the development and la[illegible]
represents a signific[illegible]
appears to be well-p[illegible]
jeopardizing its finan[illegible]
Series C equity finan[illegible]
confidence in its stra[illegible]
to utilize internal cas[illegible]
long-term debt, whil[illegible]
below 0.7. This dive[illegible]
robust financial health and its ability to support large-scale investments like EcoCharge.

Moreover, the company has a clear strategic vision for its capital expenditures, aiming to drive innovation, expand its market presence, [illegible] efficiency. The EcoCharge initiative aligns [illegible] EcoTech's existing technologies to [illegible]

NOW, THEREFORE, BE IT RESOLVED, that the undersigned stockholders hereby approve the development and launch of the EcoCharge EV Charging Solutions product line, including the allocation of $10 million in research and development funds for this initiative;

Amendment to Stockholder Constent.pdf
PDF

Interface text:
Action History
Ecotech: Pre-Term Sheet Diligence Questionnaire
Related Party Risk
Mergers & Acquisitions
Fundraising Timeline
AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Exploring an acquisition finding

An expanded merger finding describes EcoTech Solutions acquiring Renewable Storage Solutions for $8 million, including integration costs. The popover offers “Tell Me More” and “Implications” for further exploration.

[19:37](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1177s) [Brightwave diligence table with an acquisition finding expanded in a popover.](https://assets.ai.engineer/sites/aiecode2025/sha256/bec5abb973ea3d2c209a306dd67b15fbd934dc75c1c40d208e73ea4d27928c13.webp)

21 Documents
Risks | Related Party Risk | Mergers & Acquisitions | Fundraising Timeline

Amendment to Stockholder Constent.pdf
Patent litigation expenses may impact EcoTech Solutions’ [illegible]
Significant capital expenditure on EcoCharge may strain [illegible]
Environmental compliance issues in Shanghai may lead to [illegible]
Related Party Risk: No
Mergers & Acquisitions: No
Fundraising Timeline: No Findings

Capital Expenditure Plan & Forecast.pdf
R&D initiatives may face technological uncertainties [illegible]
Acquisition of Renewable Storage Solutions, Inc. may lead [illegible]
2 more
Related Party Risk: No
Mergers & Acquisitions: Yes
Fundraising Timeline: EcoTech Solutions Outlines $70 [illegible]

The Capital Expenditure Budget mentions a merger and acquisition involving EcoTech Solutions acquiring Renewable Storage Solutions, Inc. for $8 million, including integration costs.
Tell Me More
Implications

Corporate Social Responsibility Policy.pdf
Dependence on local sourcing may expose EcoTech to [illegible]
Ambitious carbon neutrality goal by 2030 may lead to [illegible]
Zero landfill goal by 2030 could increase operational costs [illegible]
Related Party Risk: [illegible]
Mergers & Acquisitions: [illegible]
Fundraising Timeline: [illegible]

Debt Schedule.pdf
Balloon payment due in 2028 may strain EcoTech Solutions [illegible]
Environmental compliance loan restrictions may limit EcoTech [illegible]
LIBOR rate increase may lead to higher interest expenses [illegible]
Related Party Risk: No
Mergers & Acquisitions: Yes
Fundraising Timeline: EcoTech Solutions’ Fundraising [illegible]

EcoTech - Confidential Investment Memo.pdf
Supply chain disruptions could hinder EcoTech’s production [illegible]
EcoTech’s reliance on government incentives may lead to [illegible]
Technological obsolescence may impact EcoTech’s [illegible]
Related Party Risk: No
Mergers & Acquisitions: No
Fundraising Timeline: EcoTech Solutions Secures $50 [illegible]

Patent infringement by competitors could threaten EcoTech [illegible]
Related Party Risk: Yes
Mergers & Acquisitions: Yes
Fundraising Timeline: Company Expands Financial Cap[illegible]

Interface text:
Brightwave
Home
Ecotech: Pre-Term Sheet Diligence Questionnaire
Action History
Report
Upload Documents
Search for additional companies and documents
Filter
Sorted: Date
AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Brightwave is hiring

Brightwave lists four open roles. The speaker says the company is hiring staff and senior staff professionals and offers a $10,000 referral bonus.

[20:03](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1203s) [Brightwave hiring slide lists product designer, product engineer, AI/ML engineer, and systems engineer roles with QR codes.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a9c378c5b77d06f7aac83eded6fdaa5d371e29b3f84179d40ae18b5a7688a52.webp)

Hiring

Brightwave
Create something incredible.

Product Designer
app.liftoff.xyz/n3FFxv

Product Engineer
app.liftoff.xyz/D5fZTS

AI/ML Engineer
app.liftoff.xyz/Kg6HyR

Systems Engineer
app.liftoff.xyz/OsApcU

## Full transcript

[0:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=0s) [on-hold music] I'm Mike Conover.

[0:17](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=17s) I am founder and CEO of Brightwave. Uh, we build a research agent that digests very large corpuses of content in the financial domain. So you can think of due diligence in a competitive deal process.

[0:30](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=30s) You are pre-term sheet, you step into a data room with thousands of pages of content. Uh, you need to get to conviction quickly ahead of, uh, other teams. You need to spot, uh, critical risk factors that would, would diminish asset performance.

[0:44](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=44s) Um, it's a fairly non-trivial task. Um, you think about mutual fund analysts. It's earnings season. You've got a universe of coverage of eighty, one hundred and twenty names. There are calls, transcripts, filings.

[0:56](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=56s) It, it's, um, a fairly non-trivial problem to understand, uh, at a sector level, but also at the individual, uh, ticker level what's, what's happening in the market. Um, or goodness, you get into confirmatory diligence, and you've got eight-- eighty, eight hundred vendor contracts, and you need to spot, uh, t- early termination clauses.

[1:16](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=76s) You need to understand thematically, how is my entire portfolio, uh, negotiating their vendor contracts? It's, um, frankly not a human-level intelligence task. And the reality, as we've stepped into this space, um, is that these, uh, these professionals, uh, just get put in a meat grinder.

[1:36](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=96s) Junior analysts are, um, tasked to do the impossible on extremely tight deadlines. Um, I come from a, a technical background. Um, prior to Brightwave, I was at Databricks and created a language model called Dolly, uh, that was one of the earlier models to demonstrate the power of instruction tuning, um, for eliciting, uh, instruction following behavior from, from

[1:57](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=117s) open source technologies. And, um, as I have met with these professionals, I have developed a deep sense of empathy for, um, the stakes and the human cost of doing this work, uh, manually.

[2:12](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=132s) When it comes to the role of the individual in, uh, finance workflows and financial research, um, we think of the parallels to early, early spreadsheets. You go to an accountant or finance professional in 1978 before the advent of computational spreadsheets.

[2:29](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=149s) You say, "What's your job?" "Well, I run the numbers." It's cognitively demanding. These people write this stuff out by hand on literally wide pieces of paper called spreadsheets. It's cognitively demanding.

[2:38](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=158s) It's important to the business, and it's time-intensive. It feels like real work. And now, nobody wants that job. And it's not because there aren't finance professionals. It's not because nobody's doing analysis.

[2:50](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=170s) It's the sophistication of the thought that you can bring to bear on the problem has increased so substantially because there are tools that allow us to think more effectively, more efficiently.

[3:01](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=181s) What we're seeing, what we're hearing from our customers is that a system like Brightwave that is able to diges-- and not just Brightwave. These-- this class of knowledge agents is able to digest volumes of content and perform meaningful work that accelerates by orders of magnitude, um, the efficiency and also, uh, time to, to value in, in these

[3:23](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=203s) markets. And so the purpose of this talk is to relate, mm,

[3:30](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=210s) sort of the intelligence that we've developed, uh, in, in the course of building this high-fidelity research agent, um, and just things that we're seeing both technically but also in terms of product affordances.

[3:40](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=220s) I mean, the, the design problem that you have to solve is how do you reveal the thought process of something that's considered ten thousand pages of content to a human in a way that's useful and legible?

[3:51](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=231s) That is not a UI/UX problem. It's not a product architecture problem that existed three years ago. And the final form factor has not been determined. Chat, everybody's very target fixated on chat.

[4:04](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=244s) Um, that's probably not enough. So the, the first thing that I'll observe is that non-reasoning models are performing greedy local search. So the, the Bloomberg talk highlighted that sort of fidelity issue.

[4:17](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=257s) Like a really concrete example, you put a Reuters article in 4o, and you ask it to extract all the organizations, goodness if it's not gonna give you products. And if you have an, a ten, five percent error rate, and you chain calls like that, you're gonna introduce, um, sort of in an exponential way, uh, the likelihood of,

[4:33](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=273s) of error being in these models. And so the, the winning systems will perform end-to-end RL over tool-use calls, where the results of the API call are in fact part of the RL'd, um, sequence of decisions so that you can make locally suboptimal decisions in order to get globally optimal outputs.

[4:51](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=291s) Um, the reality is that that's still an open research problem. You know, how do I avail myself of a knowledge graph or--

[4:59](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=299s) I did not do that. [audience laughing] Um, uh, how do you avail yourself of these tools in an intelligent way, um, so that you get globally optimal outputs? It, it does seem like that that is not a solved question.

[5:12](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=312s) So the reality, and I think it's, like, heartening to see, um, this is a theme, and I think everybody in this room can be sort of comforted by this.

[5:20](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=320s) You got, you gotta build a product today, and, like, you're g- you're gonna-- there's gonna be this talk of the bitter lesson that more data, more compute, better models dominate all other approaches.

[5:28](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=328s) Like, nobody wants an expert system. Nobody, nobody wants to use spaCy to do named entity recognition. Um, the-- Sorry. Um, that was not in the speaker notes. Uh, it-- you can think of being more circumspect about what is the scope of behaviors that the system, the agent is going to engage in, sort of like a regularization parameter,

[5:48](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=348s) which constrains the complexity of the model, and that limits the likelihood, reduces the likelihood that it will go truly off the rails and begin to produce, uh, degenerate output.

[5:57](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=357s) You can think of it sort of like a-

[6:01](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=361s) Like multi-turn-- The most interesting interactions I've had with language models are deep into a conversational tree, where you can think of selecting at each branch, each response. There are a set of, uh, reactions that I can have to the model output, and I'm steering, I'm choosing.

[6:15](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=375s) This is what knowing how to use language models, it's, it's a skill. Um, and many people who have real full-time jobs may not invest in developing that skill. This is not dissimilar to what these RL systems are doing.

[6:28](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=388s) And if you can think of a multi-turn conversation as not just establishing a, a human-orchestrated chain of thought, but really that set of tokens defines the activations of the model.

[6:41](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=401s) And if you think of the activations of the model as defining a program, what you are doing when you respond to the model and say, "No, not quite like that, more like this," is if you think of the, the activation weights at, or the activations as a point in a vector space, you are n-nudging the activations to

[7:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=420s) a place where they can finally solve the problem at hand. And I think that's what the chain-of-thought process or the sort of reasoning monologue is performing. It's, it's getting the activations to a position where it can actually solve the problem.

[7:11](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=431s) So it's actually not-- I don't-- It's cute that it, you can interpret it, but I would prefer if it just got to the right set of activations automatically. Um, and so from a product affordance standpoint, [sighs]

[7:23](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=443s) people are not going to want to really become prompting experts in a deep way. And, and frankly, it takes, you know, easily a thousand hours. Um, and so the scaffolding that products put in place in order to orchestrate these workflows and, and shape and the, the, the behavior of these systems, um, I think ha-- you know, these

[7:42](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=462s) verticalized product workflows are, are probably going to be enduring because they specify intent. They take that weight off the user. Um, so some of the things that we see with respect to archetypal design patterns in this space, consider a, a basic autonomous agent.

[7:58](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=478s) You really wanna mimic the human decision-making process and decompose what is it that a person would do. Well, if I need to understand how this, uh, poly, uh, polypropylene resin, uh, manufacturer, um, is, is managing costs, I might look for public-market comparables, and that would inc-- that would, you know, maybe entail going to the SEC filings or

[8:18](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=498s) earnings-call transcripts. And I would assess content potentially from a knowledge graph constructed from, uh, p-previous deals that, that, that, you know, I as a, as a private equity investor have done.

[8:28](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=508s) Um, news corpuses assess which, which document sets are relevant to me, distill down from those documents findings that substantiate, mm, premises or hypotheses that I might have about this question or this investment thesis.

[8:42](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=522s) Um, and then enrich and error-correct those findings. And so a couple points on this. One is that, um, it is actually-- So I forget who it was, but they were talking about-- It was the Deep Research team talking about, um, on that next step, what are my intermediary notes?

[8:58](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=538s) What is it that I believed on the basis of what I found? That's actually an extremely useful think out loud about what do we believe given the facts as they, uh, have materialized on that first pass through the n- the, the dataset.

[9:11](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=551s) Um, enriching individual findings that are distilled down from documents is an extremely powerful, um, design pattern. Likewise, um, it's, it's ki-- you can ask these models, you know, is this accurate?

[9:24](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=564s) For that Reuters example, you can say, uh, is this factually entailed by this document, or is this actually an organization? Um, and the model can frequently self-correct. And what we've noticed is that it is-- You can do that in the JSON, um, as sort of like a chain-of-thought behavior, but it's also-- it's actually more powerful to do

[9:41](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=581s) it as a secondary call because the model's kind of primed to be credulous. It says, "Well, you know, I, I told you it was, and so, yeah, I'm probably right."

[9:50](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=590s) Um, so it's interesting how you can tease apart some of these steps into multiple different, uh, calls. And then through this process of synthesis, you're able to weave together fact patterns across many, many, many documents into a coherent narrative.

[10:02](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=602s) Um, and that control loop, we think that obviously human oversight is extremely important. Um, the ability to nudge the model, um, with directives or, or sort of selecting, "The-- This is an interesting thread I want you to pull," that is extremely important.

[10:18](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=618s) And that's because the human analyst always is going to have access to information that has not been digitized. That's that conversation with management. That's, uh, your portfolio manager thinks b-- this class of biotech is just harebrained.

[10:30](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=630s) Um, that taste-making, I think, is going to be where you see, um, the most powerful, uh, products lean. I firmly believe with respect to the nodes in that knowledge graph, and we probab-- many people in this room probably have reached this un-this conclusion as well, but you still see this, oh, we got a portfolio manager agent, and

[10:47](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=647s) this is the fact checker. And, and that sort of it needless like anthropomorphizing of these systems, um, it constrains your flexibility if the design needs of your compute graph change.

[10:59](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=659s) And this is, this nineteen seventy-- I think it was 1978, and Bell, you know, the Unix philosophy, it's like you think about piping and teeing on the Bash command line, or I guess I date myself, I still use Bash, not zsh.

[11:10](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=670s) Um, y- just simple tools that do one thing and that work together well. And text is the universal interface. Um, it's forty years ago. Fifty years ago. Geez. Um, so our friends at Latent Space put together this plot.

[11:24](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=684s) With respect to the structure of these graphs, I'd-- obviously, that Pareto frontier, which is the sort of efficiency frontier. [phone ringing] It's two. Batting a thousand today. Um, the efficiency frontier for compute and performance trade-off or price-performance trade-off, um, that frontier is going to continue to move out, but there will, I believe there will, for at least enduring time,

[11:43](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=703s) be a frontier. And what's notable about that is that you have to select then which tool, which system, which model am I going to use for each node in the compute graph.

[11:53](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=713s) And the reason that this is important is

[11:59](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=719s) what I call the latency trap If you think about the plot of time to value and realized value for agentic systems, and I think this is extremely important, it's very easy to think, "Oh man, it's gonna do all of these things.

[12:11](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=731s) It's going to, you know, I'm gonna check it and error-correct, and then it, you know, in, in twenty-five minutes it's gonna be banger." And I think even with high-quality products like OpenAI's Deep Research, it's, hmm, you're not always sure that what you're going to get out is high quality.

[12:26](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=746s) So there's, there's kind of like a question of like which side of the diagonal, it's probably not a straight line, but is that product on? But also from a reps standpoint, the impulse response for the user, how well, how well refined is...

[12:38](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=758s) You can think of, like, my expectation for what the report is going to look like and what the report actually looks like is the loss. And the user's mental model is developing a sense for how do, how do my prompts elicit behaviors from these models?

[12:49](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=769s) If it's a eight-minute feedback loop, it's a twenty-minute feedback loop, goodness, you're not gonna do many of those in the course of a day. And your faculty with the system or the product is going to be low.

[13:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=780s) So synthesis is, is really where a lot of the magic happens in these systems, and, um, y- a couple observations. So n- notice that it-- I don't know, has anybody in this room ever had a fifty thousand token response from any model?

[13:17](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=797s) No? They say that it's, you know, o1 is a hundred thousand context, output context length. Um, I'm not so sure, and it's because the instruction tuning demonstrations, these human-gener- synthetic or human-generated outputs that are used to post-train the models, have a characteristic output length.

[13:34](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=814s) It's hard to write fifty thousand coherent, uh, novel words. And so the likelihood that the models are able to produce, I mean, even, um, o1 still is about two thousand, three thousand tokens.

[13:47](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=827s) It's better than 4o. And so what happens, it's kind of like a com-- there's a compression problem. If I have a very, very large context window for input, I'm compressing that information into a set of tokens.

[13:57](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=837s) And so it's the, like the difference between writing a book report and a synopsis of each chapter. You can dis-- you can be more focused and, um, specific about what is it that I want those n thousand tokens to be focused on.

[14:11](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=851s) Um, here we have, uh, you know, I said write, write, uh, an analysis of the global financial crisis. Goodness if I don't think the rise of the shadow banking system warrants more than three sentences.

[14:23](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=863s) And so if you, if you can be more granular and more specific, um, you can get higher quality, higher fidelity, more information-dense outputs out of these systems by decomposing your research instructions into multiple subthemes.

[14:36](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=876s) Um, additionally, the last point I'll make on this problem is that of, uh, the, the presence of recombinative reasoning demonstrations in the instruction tuning and post-training corpuses is low.

[14:48](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=888s) So it is, uh, easy to say, here, you know, given the text of The Great Gatsby, this is the epilogue, and write a new epilogue for The Great Gatsby because the cost of internalizing that corpus is fixed, effectively.

[15:00](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=900s) You read the book, and then you write five epilogues, and it's like, "Goodness, I, I got it."

[15:06](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=906s) Synthesis really is about weaving together disparate fact patterns from multiple documents. Think about the applications to biomedical literature synthesis. I need to read all of these papers and then have something useful to say that actually brings together the facts from these documents.

[15:21](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=921s) Now, there's like a, a cute trick you could try, which is to say, given the bibliography of any given paper, write the abstract as an ins-- as a, as a post-training exercise.

[15:30](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=930s) But it's just really hard to get high-quality, intelligent, thoughtful analysis of many, many, many different documents. And so there are limitations in practice for, uh, even state-of-the-art models in terms of how they are able to manage complex real-world situ-situations, uh, factors like temporality, um, the per-plexity at a...

[15:53](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=953s) Well. So temporality is hard. Um, and being able to understand, you know, something like a merger and an acquisition, um, you know, this, these pro forma financial statements are different from those that came, um, before the event, um, if they're addendums to contracts.

[16:08](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=968s) It's important to propagate with, um, evidentiary, um, passages, metadata that contextualizes why do I care about this? What do we think about this document? Um, what-- How should I consider this in relation to the other dem-- uh, pieces of evidence in, in the cor- in the context window?

[16:29](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=989s) Um, so I'll now shift a little bit with some, some examples from, from the, our, the product that we've built, which is, um, how do you reveal the thought process of something that's considered ten thousand pages of text?

[16:42](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1002s) And I think that it is more like a surface, and one where you're able to, um...

[16:50](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1010s) It's kind, so it's kinda like this. Like, people you may know, the Facebook and LinkedIn recommendation algorithm for, for, um,

[16:59](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1019s) connections, uh, feels uncanny good in part, not because... I mean, the algorithms are okay, not great, um, have gotten a lot better over time. But in your visual cortex, there is a, a bundle of nerves that are, uh, exclusively dedicated to face recognition and the ability to say in a, in a f- you know, six by six

[17:20](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1040s) grid of faces, "Goodness, I know that person." And so you attend to the things that matter, even if it's actually a low-precision product experience. And so the ability to give the person, um, details on demand is extremely important.

[17:34](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1054s) Um, we'll see. So here we have a Brightwave report. Um, we, you know, I think the ability to click on a citation and then get additional context about this, not just what document is it from, but how should I be thinking about this?

[17:47](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1067s) What was the model thinking in the course of this? Um, as well as structured interactive outputs that give you the ability to pull the thread and say, "Well, tell me, tell me more about that rising CapEx spend."

[17:59](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1079s) In Brightwave, um, you can highlight any passage of text. So it's not just the citations, but you can highlight any passage of text and say, "Tell me more. What are the implications of this?"

[18:08](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1088s) I think OpenAI gestures towards this with respect to Canvas and the ability to increase the reading level of, of a passage. Having a continuous surface that's not just these citations, um, but in fact any, uh, finding should be interrogable.

[18:24](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1104s) Um- Likewise, you can think of... I'm actually gonna pause. Is that gonna-- It's not gonna pause. I'm gonna go back and do this again. Um,

[18:36](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1116s) you can think of the set of things that the model has discovered. It reads all of these documents. It develops a view. It weaves the facts together, um, as a h- as a high-dimensional data structure, and the report is one view on that data structure.

[18:49](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1129s) It's kind of a low c- low-effort point of entry into the, the space of ideas. You wanna be able to turn over that cube and see, especially in finance, um, the receipts.

[19:01](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1141s) What's the audit trail for this system that's read all of these materials? And so being able to, in this example, click into the documents is one level, but having all of the findings laid out for you, whether it's a fundraising timeline, um, ongoing litigation, I'm able to, if something catches my attention, click on it, and this is

[19:18](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1158s) where that, the investor... Hello? [chuckles] Investor or analyst taste comes into play. I'm able to say, "Tell me more about that." It's like a magnifying glass for text. Something catches my eye, this patent litigation.

[19:30](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1170s) Goodness, that seems important. Um, you had a factory fire in Mexico that wiped out a m- you know, critical single-source supplier. Um, what are you gonna do about that?

[19:39](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1179s) That ability to drill in and get additional details on demand is extremely important in these systems, and I think candidly, um, we, we do not yet have the final version, the final form factor of this class of products.

[19:52](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1192s) Um, but it, it's an extremely interesting design problem, and I will say, uh, we are, we are hiring. So these QR codes, not only is it a great place to work, we've got, uh, people from Goldman Sachs and UBS and Meta and Instagram and Anaplan, and we just hired a senior staff software engineer from Brave.

[20:09](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1209s) Um, goodness, we got a stacked team. We also have a $10,000 referral bonus, so I'm gonna see a lot more phones come out now. [laughs]

[20:17](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1217s) Um, $10,000 referral bonus for all of these roles, primarily the product designer and the front-end engineer. We're hiring staff and senior staff-level professionals. We t- we have a small team of extremely experienced individuals, um, and this is structured like the DARPA Red Balloon Challenge, if you're familiar.

[20:34](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1234s) Um, so if you refer the person that refers the person that we hire, you get 1,000 bucks, and so on, and so on, and so on, all along that, uh, exponentially exploding, uh, referral tree.

[20:43](https://www.youtube.com/watch?v=MWTJIAwAAnk&t=1243s) So we're Brightwave. We build knowledge agents for finance workflows. I appreciate your time today. [upbeat music]
