Your Company Has No Model of Itself. Agents Can Build One - Joao Wenzel, Citadel Securities
Loading slides…
Read the talk
Your Company Has No Model of Itself. Agents Can Build One
João Wenzel explains how Citadel Securities builds a history of what teams reported, what the evidence supported and what happened next, down to the unresolved dependency on an otherwise green slide.
From a talk by Joao Wenzel
At a glance
Ideas worth remembering
Keep the reported status, supporting evidence, agent assessment and rationale separately. Carry corrections forward so an earlier judgment can be evaluated on the evidence available at the time.
Date each claim and track both the last collection attempt and the last successful read. An old estimate and a silent collector call for different interpretations.
Evaluate observations and predictions separately. Count the useful updates a workflow finds, then back-test its flags against later outcomes with a clearly defined comparison group.
When a stakeholder can explain a recurring judgment, capture the rule and distinguish their confirmation from the agent’s original inference.
What the colleague who survived seventeen reorgs knows
Think of the person at a company who has been through seventeen reorgs. They know the gossip and the politics, and somehow they can predict what will come of a big strategic move. João Wenzel calls that person the company’s world model. Colleagues can ask them how a decision is likely to play out, but there is only one of them, and hardly anyone writes down what they say. At Citadel Securities, Wenzel and his colleagues have been trying to approximate that capability with agents.
Citadel Securities builds its own technology down to the silicon, and more than two-thirds of its employees write code. Leadership had seen what coding agents could produce and wanted to know whether they could also help define and refine company strategy. Borrowing a visual from Tim Urban, Wenzel breaks that job into three questions. Where do we stand now? What happens next? How did we get here?
Each question requires some institutional memory. Two teams that used to collaborate may have stopped speaking after a reorg. A project’s current scope may make sense only to someone who remembers why it was narrowed several times. Even a meeting record takes interpretation. Who was in the room, what were they willing to say, and what did they leave unsaid?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two views of a project and three tests for new evidence
The top-down view comes from strategy documents, decks, meeting reports and status updates. These are tidy accounts of what people want the organization to know. Their authors also exercise judgment about what to include and how to characterize it. Wenzel calls the extreme case a “watermelon project,” green on the outside and red on the inside. The bottom-up view comes from meeting notes, Slack threads, code changes and sessions with coding agents. A detail that explains a project’s condition may be buried in the middle of a much noisier exchange.
Wenzel borrows the state estimator from robotics to bring those views together. Start with a belief about something that matters, such as whether a project can meet its target. As time passes without a supporting observation, that belief grows less certain. A deck, a thread or a meeting note supplies another observation. It can give the system grounds to renew its confidence or to change its assessment.
Three considerations govern that update. How long has it been since the signal arrived? Where did it come from, and is that source still trustworthy? Does it agree with the other sensors? The third question matters even for recent evidence. A person may have posted a sincere assessment five minutes ago while another participant already has measurements that call it into question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reading a status slide takes both XML and vision
Consider the monthly status deck that someone in business operations already produces. Unpack the PowerPoint file and a script can read its XML. Now the script needs to decide what green is. Which hex code counts as the green that means a project is on track? A person can read the intended status at a glance, while the file exposes a color value whose significance the extractor has to establish.
Sending a screenshot of the slide to a vision model solves a different part of the problem. The model can interpret the visual presentation, but the slide’s author may have pasted in a screenshot to begin with. Even a high-resolution image of the finished slide can leave Claude struggling to distinguish 31 from 81 in the embedded image. Wenzel’s team uses both the PowerPoint XML and the vision model’s output to extract the evidence.
Accurately reading one slide still leaves two kinds of change to account for. Across decks, a project spontaneously changes names in March, and the system needs to preserve its history. Between decks, the evidence can change in five days. Someone writes in Slack that the project looks on track. A colleague replies with a picture of a dashboard that undermines that assessment. Reading only the deck misses the exchange. Reading only the message misses the reply.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a March 16 status query misses
The system organizes its work into five stages. Update carries the record forward. Age accounts for the time since a claim was supported. Observe examines incoming material for evidence. Grade compares the system’s expectations with what subsequently happened. Correct gives people a way to supply feedback and steer future judgments. Citadel Securities uses agents throughout this loop. The model proposes changes, and code and people decide how to update the records.
Wenzel illustrates the difference with a retrieval-augmented generation (RAG) query. Ask on March 16 whether a project is on track. A model can retrieve a couple of reassuring Slack messages and slides and confidently answer green. Those sources explain why someone might say green. To advise a manager, the agent also needs to know which project the sources describe, how its commitments have evolved and whether an outstanding issue qualifies the latest report.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry Project North Star forward one month at a time
For the monthly update, the inputs are last month’s project record and this month’s slide. Pairing them gives the agent a specific continuity question to resolve. When the name changes in March, the people involved and the goals they are pursuing still connect the new entry to Project North Star. The agent can infer that this is the same undertaking and carry its history forward under the new name. At the time of Wenzel’s presentation, the team had accumulated three years of these records.
The update distinguishes four things a single status field would collapse. It records what the team reported, the evidence the agent found, the assessment that evidence supports and the reason for the assessment. In Wenzel’s example, green is the reported status. A comment someone left on the slide identifies an outstanding dependency. The agent flags the project as yellow and cites the dependency as its reason. A manager can see both why the project appeared healthy in the deck and why it deserves another look.
Corrections flow into the next record. Earlier months stay as they were, including the judgments the agent could reasonably have made with the evidence then available. That choice gives the team an audit history across renames and revisions. It also preserves something a continuously rewritten summary would lose, namely the ability to test a February judgment using only what was knowable in February.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delivery dates show how quickly a claim ages
In January, the projected go-live is Q2. It moves to Q3, then to October. Each target belongs in an account of how the project got here. To answer when it is likely to go live today, the system needs the date on which each target was claimed. Otherwise, a January slide promising Q2 remains available to support an answer long after the team has moved on.
The three-year history gave Wenzel a way to measure this effect. On average, half of delivery dates changed within two months. Of the dates that changed, 90 percent moved later than originally proposed. This gives practical meaning to his observation that a fact has a half-life. Even before a replacement target arrives, the age of the last estimate matters. Attaching a date to each claim lets an agent weigh recency when answering questions about a project, a person or the company’s intentions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialist observers found more details worth recording
The team’s first approach to observation was to give Claude or another model the entire collection of materials and see what it found. The appeal was simplicity. The failure was attention to detail. A model could take in the corpus and still overlook individual observations that should change a project record, much as a reader scanning the green indicators could miss the comment about a dependency.
The team broke the job into narrower assignments. Specialist agents examine the materials for particular kinds of information, and other agents review their findings. Wenzel reports that the number of updates the team considered insightful for a project more than doubled with this workflow. That comparison concerns what the observers found worth recording. The team used a separate evaluation to learn whether any of those observations helped identify future trouble.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Back-test a February warning against the next status label
To grade the model, pick a date in the past and reconstruct the record as it stood then. Wenzel uses February as the example. Give the agent the evidence available by that point and see which apparently healthy projects it flags as likely to slip. Then consult the status labels that arrived afterward. Because later updates were carried forward, a warning discovered in March cannot silently improve the agent’s February call.
Wenzel reports that projects flagged while green were twice as likely to turn yellow or red in later status reports. He leaves the comparison group implicit. A manager can use the flag to decide which project to inspect first, even while the cause of a future status change and its effect on delivery remain open questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask stakeholders for the rule behind a correction
Wenzel describes having 18 cases with a flagged verdict, then turns to the cases where the agent lacks a verdict. This is partly a product-design problem. Someone with the missing context has to see the question and find it easy enough to answer. The team put work into collecting feedback in the app a stakeholder already uses, in Slack or by email. Every additional step between seeing the question and answering it makes that loop harder to sustain.
The wording determines how far a correction travels. Asking for a fact can settle one disputed output. Asking about the rule the person used to reach that judgment can improve other assessments to which the same rule applies, now and in future updates. The project record distinguishes what the agent initially inferred from what a person subsequently confirmed. A later reader can tell where human confirmation entered the history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the thread, the pull request and the work that preceded it
Bottom-up sources expose different parts of the work. Meetings capture what people say aloud. Slack offers a continuous, noisy account close to the work itself. The unit the team examines is a thread, including its participants and attachments. For a claim worth recording, the agent identifies when it was made, who was responsible or whom it concerns, which project it belongs to and where to find the supporting evidence. That last link makes it possible to revisit the dashboard image in its original exchange.
Pull requests show what changed and how a project is evolving. For engineers who opt in, coding-agent session traces also show how the work reached the pull request. The original ask may have evolved, tests may have failed along the way, or the developer may have made a decision that explains the final change. Reading the trace alongside the PR gives the system access to decisions and difficulties that the finished change alone may leave implicit.
Collection itself can fail. The system once treated a channel that appeared to be going quiet as evidence about the people or project using it. Wenzel compares that inference to driving down a highway and concluding that your speed has dropped to zero because the speedometer reading disappeared. The team began tracking two times for each source, when it last tried to collect information and when collection last succeeded. If those times diverge, the apparent quiet may tell the manager more about the collector than about the project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Open the leadership meeting where the evidence disagrees
Return to the green project. The assembled record shows that the go-live has slipped twice and people are discussing failing tests. The evidence points to yellow. Leadership can open with that discrepancy and discuss the work behind it, sparing everyone another pass through slides they have already skimmed. The records also feed a weekly read for managers and leaders on what stuck and what went quiet.
Wenzel sees a further use in giving other agents a common source of context. In its client business, Citadel Securities is exploring whether a similar model could bring internal and external evidence together to explain who holds what, what clients are interested in and how to anticipate their needs. That extension exposes a harder identity question than the North Star rename. One client can have four records in four systems with no canonical identifier. Wenzel says the team still lacks a clear way to establish that those records belong together.
Stakeholder attention is another unresolved constraint. Wenzel distinguishes three cases. For a low-stakes update the model is confident about, apply it automatically. For a low-stakes update it is unsure about, wait for more evidence. When the stakes are high, the choice of whom to ask, when to ask and how to collect the answer matters. He closes by encouraging teams to start collecting history early. Thousands of decks gave his team material to work with, but the dated observations, subsequent outcomes and human corrections are what made that history usable in daily work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…