Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis
Loading slides…
Read the talk
Why a 100x Faster Engineer Still Needs a Process the Company Can Trust
Basis co-founder Mitchell Troyanovsky uses a bank reconciliation to show how a firm can specify the decisions an agent should make, inspect those decisions in production, and investigate the ones that went wrong.
From a talk by Mitchell Troyanovsky
At a glance
Ideas worth remembering
Choose consequential procedures at the firm or team level so each employee can delegate work without configuring the agent's method.
Write each important behavior with a recognizable trigger and an observable action. Score reconciliation runs for discrepancy handling only when a discrepancy occurred.
For a bad decision, fork after it to ask for an explanation or before it to test revised context. A sub-agent simulation offers a cheaper first experiment if it can reproduce the mistake.
Set review depth using both the consequences of an error and the agent's demonstrated competence on that kind of work.
What a personal speedup leaves out
Mitchell Troyanovsky, one of the two co-founders of Basis in New York City, frames his research around a practical question. What would it take for accounting firms across the country to trust agents with production work from beginning to end? An engineer who uses Cowork, Codex, or Claude Code may already have experienced an enormous increase in personal output. Troyanovsky estimates that his own productivity may have risen a hundredfold. Yet the average firm, he observes, has seen nothing like a tenfold or hundredfold increase. Why does a gain that one engineer can see in their own work prove so hard to reproduce across a company?
For an explanation, he looks at how people already work with intelligent systems whose actions they cannot predict exactly. Their coworkers fit that description. The head of an accounting firm can have very little idea what the most junior people are doing on a given day and still have good reason to trust the work. Understanding how the firm earns that confidence helps explain what an individually productive agent user would have to reproduce across an organization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A monthly close contains a decision about who handles a PDF
An organization establishes processes that it expects to produce good results across many cases. The people carrying them out can exercise judgment, and any one person can make a mistake. The process gives the head of the firm a basis for delegation anyway. Someone with responsibility for the work has thought through how it should be done, and people throughout the firm have a way to follow that decision.
Follow one piece of monthly accounting work down to its smallest dependency. To reconcile cash for a client, the accountant needs the bank data. If the data comes in a PDF, even the extraction step can have a procedure as simple as using the firm's approved software. An accountant who searches the web for a PDF extractor and picks one may get perfectly usable data. If everyone does that independently, though, the firm has lost its say over how client statements are processed. The choice matters several levels below the task the head of the firm thinks they have delegated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Giving everyone an agent opens two gaps
Troyanovsky calls the first problem the competency gap. A coding-style agent invites the user to describe a job and send it off to work. Behind that simple interaction, someone still has to manage skills and memories, decide when to use the agent, and anticipate how it will interpret the request. Experienced users develop a theory of mind for their agent. A firm distributing the same product to its knowledge workers is implicitly asking each person to acquire that skill alongside the expertise their job already requires.
The process gap persists even among capable users. Give every employee an agent to customize and each employee gets to decide how the firm does the work. Troyanovsky argues that senior leadership should set the firm's procedure. An accountant may know exactly how to configure a favorite PDF extractor, but the firm still needs everyone handling client statements to use the software it approved. Troyanovsky points out that a firm would have had the same objection to everyone inventing their own extraction procedure in 2018.
Closing the first gap means an accountant can give the agent work as naturally as asking a coworker, without having to reason through its internals. Closing the second means the accountant knows what to expect. Even a consistent limit is useful. People can learn what the agent can handle today and entrust it with more as it improves. An agent that displays expertise one day and misses a basic step the next is harder to place on a team.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An opinion settles a choice the user would otherwise make
Consider what the request to read a PDF leaves open. The agent could pass the document directly to a vision-language model (VLM). It could write Python that calls an open-source optical character recognition (OCR) library, or use a specialist OCR model provider. It could run OCR first and ask a VLM to check the extraction. Even the checking step admits variations. Troyanovsky imagines five VLMs reviewing the work, then a hundred VLMs and a selection of the best result. A correct extraction on one document reveals little about which of those procedures the agent will choose on the next.
Being opinionated means settling consequential choices like these on behalf of the user. Some opinions can start with domain expertise. For tax work, Troyanovsky expects an agent that reads the primary source before replying to be more accurate. The team can adopt that expectation and look for the behavior without first building an evaluation set to discover whether consulting primary sources is a good idea.
Other opinions turn on what users value. An agent making a PowerPoint presentation can take a picture of the rendered deck and inspect it before delivery. That gives it a chance to catch a formatting error, at the cost of extra latency on every deck. Suppose the agent makes a formatting mistake only once in a hundred presentations and the users can tolerate that frequency. The team might reasonably skip the visual check. The rate is hypothetical. What matters is that the team can weigh a specific error against the time users spend waiting, then make a choice that applies across deliveries.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a trajectory can reveal without a known answer
An outcome-based evaluation judges the work against a known answer or a rubric. A team might have a hundred such cases, or a thousand. In production, Troyanovsky asks it to consider hundreds of thousands of possible situations. As agents undertake longer jobs, an early choice affects which choices become available later. The number of possible paths through the work grows rapidly. Even a carefully selected set of final answers leaves the team with a question about how well those cases represent the work ahead.
A trajectory, the record of a run and the decisions made along the way, offers another kind of evidence. The team may lack the ground truth needed to certify the result and still be able to tell whether the agent followed a procedure it believes produces good results. For example, someone reviewing tax work can ask whether the agent consulted the primary source before replying. Observing that step supports an inference about how the agent will handle similar work. The accuracy of the particular answer remains a separate question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reconciliation that ties can still violate the firm's policy
Suppose the books and the bank statement disagree. An agent can post an entry for the difference, a plug that makes the reconciliation tie. The totals will then agree, while the discrepancy that prompted the entry still needs an explanation. Troyanovsky wants the agent to explain the difference. He allows that another customer might choose the balancing entry. What would make the service difficult for either customer to rely on is receiving an explanation one month and an unexpected entry the next.
Getting that choice into a production agent takes more than identifying a place to put two sentences. A narrow agent might follow a system-prompt instruction consistently. In a larger system, the system prompt works alongside instructions disclosed progressively as the job unfolds, descriptions of the tools the agent can use, and behavior learned during model training. The agent makes its decision under the combined influence of all of them.
Troyanovsky suggests treating the context the agent sees as “runtime training data.” He means this as an analogy. The model's weights need no update for a tool description or a newly disclosed instruction to change its next action. An engineer investigating an unwanted entry therefore needs to care about the actual words available at the moment of the decision. Reading the paragraph where someone originally wrote the reconciliation rule gives only a partial view.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A behavior spec gives reviewers two questions to ask
Basis writes a behavior spec before writing or revising the agent's context. Like a product spec, it records what the team intends. The team selects its most important opinions. Trying to document every detail would duplicate information already maintained elsewhere, and one desired behavior may depend on several pieces of context. A concise spec gives the people changing prompts, skills, or tool descriptions a common account of the decision they are trying to produce.
The same spec becomes a rubric for reading a trajectory. First, did the agent reach a situation in which the behavior was relevant? For the reconciliation policy, that requires a discrepancy. A run in which the balances already agreed offers no evidence about how the agent handles a mismatch. Second, when the situation did arise, did the agent exhibit the specified behavior? The reviewer can distinguish an attempt to explain the difference from an entry posted to make it tie.
Those questions make more runs available for supervision. The reviewer can identify a policy violation without independently completing the entire reconciliation. They can also record that the agent pursued an explanation while leaving its correctness open. Outcome evals remain valuable for judging the finished work. The behavior spec lets the team learn from decisions in testing and production much faster than it can supply complete ground truths for new cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fork before the bad decision to test a change
A failed behavior check identifies a place to investigate. Troyanovsky likens forking to the control beneath a message in a coding agent that lets the user branch from that point in the interaction. Start with a trajectory from an eval, a test, or production. Find the point where the agent chose to post the plug. A fork immediately after the decision lets an engineer ask the agent why it did that. A fork immediately before the decision supports a different experiment. Change the context, let the agent face the discrepancy again, and observe which action it takes.
The usefulness of that experiment depends on how faithfully the fork recreates the original conditions. In a cloud system, copying the conversation may be the easy part. The engineer may also need the database environment, the prompts as they appeared at that point, and the surrounding environment the agent was working in. If the bank data differs between branches, a different decision is harder to attribute to the revised instruction.
The advantage over starting from a bad final result is the location of the failure. Troyanovsky imagines a five-hour trajectory containing a thousand inference calls. Learning at the end that the outcome was wrong leaves a considerable search through those calls. A meaningful behavior check points to a particular choice and gives the engineer something observable to test when revising the agent's harness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reproduce the mistake with a sub-agent before rebuilding the environment
Troyanovsky's quicker experiment starts inside a coding agent. Ask it to create a sub-agent to play the role of the production agent. Give the sub-agent the system prompt and skills rendered as they would appear during the job. Instruct it to stay within the information available in that role. Then describe the situation from the eval or production run and ask what it would do. The first result to look for is the original mistake. A simulation in which the agent immediately does the right thing tells the engineer little about how to repair the observed failure.
If the sub-agent reproduces the bad call, vary the context and ask again. The engineer can try several ideas without repeatedly restoring an entire environment. The tradeoff is fidelity. A sub-agent describing what it would do in a supplied scenario gives weaker evidence than a fork taking an action under the reconstructed conditions. Troyanovsky used simulations for more than a year before Basis built forking into its internal systems.
The original opinion gives these experiments a common target. The team can spot the unwanted decision in a real run, reproduce it, change the context, and inspect later trajectories for the same behavior wherever it applies. That closes a feedback loop around a production decision even when the team cannot grade every production outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delegate according to the agent's track record and the cost of a mistake
An agent can start with a narrow assignment. People who work with it get to observe what it handles reliably, where it needs help, and whether its competence improves. Troyanovsky compares the resulting trust to reviewing a pull request. Two things determine how closely a colleague will read it: the blast radius of the change and their assessment of its author. A change to a core service from someone whose work inspires little confidence warrants close inspection. An internal documentation edit from a trusted contributor can receive a lighter review.
Left to choose its own method, an agent might discover a better one. Troyanovsky grants the possibility of a surprising “move thirty-seven,” then asks who wants an agent to try one on their tax return. An accounting firm needs to deliver returns reliably, at scale, in a way its people understand. For agents to spread through the firm, employees need to find them simple to use. To trust them with more work, employees need to see them behave reliably. Specifying how the agent should act also gives its builders something to learn from. When a production run departs from the expected procedure, they can investigate the decision, revise the context, and check whether the agent handles the next such case better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…