# Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis

Mitchell Troyanovsky, Basis

[Talk page](https://ai.engineer/talks/JGtkTjn4120-why-ai-makes-you-100x-faster-but) · [Video](https://www.youtube.com/watch?v=JGtkTjn4120)

## Why a 100x Faster Engineer Still Needs a Process the Company Can Trust

Basis co-founder Mitchell Troyanovsky uses a bank reconciliation to show how a firm can specify the decisions an agent should make, inspect those decisions in production, and investigate the ones that went wrong.

### What a personal speedup leaves out

[0:12](https://www.youtube.com/watch?v=JGtkTjn4120&t=12s)

Mitchell Troyanovsky, one of the two co-founders of Basis in New York City, frames his research around a practical question. What would it take for accounting firms across the country to trust agents with production work from beginning to end? An engineer who uses Cowork, Codex, or Claude Code may already have experienced an enormous increase in personal output. Troyanovsky estimates that his own productivity may have risen a hundredfold. Yet the average firm, he observes, has seen nothing like a tenfold or hundredfold increase. Why does a gain that one engineer can see in their own work prove so hard to reproduce across a company?

For an explanation, he looks at how people already work with intelligent systems whose actions they cannot predict exactly. Their coworkers fit that description. The head of an accounting firm can have very little idea what the most junior people are doing on a given day and still have good reason to trust the work. Understanding how the firm earns that confidence helps explain what an individually productive agent user would have to reproduce across an organization.

### A monthly close contains a decision about who handles a PDF

[2:21](https://www.youtube.com/watch?v=JGtkTjn4120&t=141s)

An organization establishes processes that it expects to produce good results across many cases. The people carrying them out can exercise judgment, and any one person can make a mistake. The process gives the head of the firm a basis for delegation anyway. Someone with responsibility for the work has thought through how it should be done, and people throughout the firm have a way to follow that decision.

Follow one piece of monthly accounting work down to its smallest dependency. To reconcile cash for a client, the accountant needs the bank data. If the data comes in a PDF, even the extraction step can have a procedure as simple as using the firm's approved software. An accountant who searches the web for a PDF extractor and picks one may get perfectly usable data. If everyone does that independently, though, the firm has lost its say over how client statements are processed. The choice matters several levels below the task the head of the firm thinks they have delegated.

![The month-end close diagram expands cash reconciliation into four steps. The first, getting bank data, leads to a bank statement and the firm’s chosen extraction method. Two improvised alternatives are crossed out.](https://ai.engineer/api/talk-essay-media/6a21168168f67fe3a2db328557ac5728c523d618b293856d0e2edc7c6867eb1f)

Even the smallest step has a process

### Giving everyone an agent opens two gaps

[3:34](https://www.youtube.com/watch?v=JGtkTjn4120&t=214s)

Troyanovsky calls the first problem the competency gap. A coding-style agent invites the user to describe a job and send it off to work. Behind that simple interaction, someone still has to manage skills and memories, decide when to use the agent, and anticipate how it will interpret the request. Experienced users develop a theory of mind for their agent. A firm distributing the same product to its knowledge workers is implicitly asking each person to acquire that skill alongside the expertise their job already requires.

The process gap persists even among capable users. Give every employee an agent to customize and each employee gets to decide how the firm does the work. Troyanovsky argues that senior leadership should set the firm's procedure. An accountant may know exactly how to configure a favorite PDF extractor, but the firm still needs everyone handling client statements to use the software it approved. Troyanovsky points out that a firm would have had the same objection to everyone inventing their own extraction procedure in 2018.

Closing the first gap means an accountant can give the agent work as naturally as asking a coworker, without having to reason through its internals. Closing the second means the accountant knows what to expect. Even a consistent limit is useful. People can learn what the agent can handle today and entrust it with more as it improves. An agent that displays expertise one day and misses a basic step the next is harder to place on a team.

### An opinion settles a choice the user would otherwise make

[6:48](https://www.youtube.com/watch?v=JGtkTjn4120&t=408s)

Consider what the request to read a PDF leaves open. The agent could pass the document directly to a vision-language model (VLM). It could write Python that calls an open-source optical character recognition (OCR) library, or use a specialist OCR model provider. It could run OCR first and ask a VLM to check the extraction. Even the checking step admits variations. Troyanovsky imagines five VLMs reviewing the work, then a hundred VLMs and a selection of the best result. A correct extraction on one document reveals little about which of those procedures the agent will choose on the next.

![A flowchart routes a PDF by the document type inferred from its filename. Semi-structured documents go to OCR A and structured documents to OCR B. Both paths add a vision-model check if OCR may miss tables or diagrams.](https://ai.engineer/api/talk-essay-media/9241c537730346962ebbacfd658ffa177bb41b1a0dfe217d8e29882689c34d57)

An example opinion: reading any PDF

Being opinionated means settling consequential choices like these on behalf of the user. Some opinions can start with domain expertise. For tax work, Troyanovsky expects an agent that reads the primary source before replying to be more accurate. The team can adopt that expectation and look for the behavior without first building an evaluation set to discover whether consulting primary sources is a good idea.

Other opinions turn on what users value. An agent making a PowerPoint presentation can take a picture of the rendered deck and inspect it before delivery. That gives it a chance to catch a formatting error, at the cost of extra latency on every deck. Suppose the agent makes a formatting mistake only once in a hundred presentations and the users can tolerate that frequency. The team might reasonably skip the visual check. The rate is hypothetical. What matters is that the team can weigh a specific error against the time users spend waiting, then make a choice that applies across deliveries.

### What a trajectory can reveal without a known answer

[9:06](https://www.youtube.com/watch?v=JGtkTjn4120&t=546s)

An outcome-based evaluation judges the work against a known answer or a rubric. A team might have a hundred such cases, or a thousand. In production, Troyanovsky asks it to consider hundreds of thousands of possible situations. As agents undertake longer jobs, an early choice affects which choices become available later. The number of possible paths through the work grows rapidly. Even a carefully selected set of final answers leaves the team with a question about how well those cases represent the work ahead.

A trajectory, the record of a run and the decisions made along the way, offers another kind of evidence. The team may lack the ground truth needed to certify the result and still be able to tell whether the agent followed a procedure it believes produces good results. For example, someone reviewing tax work can ask whether the agent consulted the primary source before replying. Observing that step supports an inference about how the agent will handle similar work. The accuracy of the particular answer remains a separate question.

![Three bold lines mark outcome evals on a bell-shaped distribution of production situations. A callout enlarges one situation in the right tail, showing several possible paths with one behavior highlighted.](https://ai.engineer/api/talk-essay-media/ccbb342b3631cb81f591a5424f2ed1bdab051b516f1afff473bd6f994c115718)

How do you know your outcome evals generalize to production?

### A reconciliation that ties can still violate the firm's policy

[10:18](https://www.youtube.com/watch?v=JGtkTjn4120&t=618s)

Suppose the books and the bank statement disagree. An agent can post an entry for the difference, a plug that makes the reconciliation tie. The totals will then agree, while the discrepancy that prompted the entry still needs an explanation. Troyanovsky wants the agent to explain the difference. He allows that another customer might choose the balancing entry. What would make the service difficult for either customer to rely on is receiving an explanation one month and an unexpected entry the next.

Getting that choice into a production agent takes more than identifying a place to put two sentences. A narrow agent might follow a system-prompt instruction consistently. In a larger system, the system prompt works alongside instructions disclosed progressively as the job unfolds, descriptions of the tools the agent can use, and behavior learned during model training. The agent makes its decision under the combined influence of all of them.

Troyanovsky suggests treating the context the agent sees as “runtime training data.” He means this as an analogy. The model's weights need no update for a tool description or a newly disclosed instruction to change its next action. An engineer investigating an unwanted entry therefore needs to care about the actual words available at the moment of the decision. Reading the paragraph where someone originally wrote the reconciliation rule gives only a partial view.

### A behavior spec gives reviewers two questions to ask

[11:49](https://www.youtube.com/watch?v=JGtkTjn4120&t=709s)

Basis writes a behavior spec before writing or revising the agent's context. Like a product spec, it records what the team intends. The team selects its most important opinions. Trying to document every detail would duplicate information already maintained elsewhere, and one desired behavior may depend on several pieces of context. A concise spec gives the people changing prompts, skills, or tool descriptions a common account of the decision they are trying to produce.

The same spec becomes a rubric for reading a trajectory. First, did the agent reach a situation in which the behavior was relevant? For the reconciliation policy, that requires a discrepancy. A run in which the balances already agreed offers no evidence about how the agent handles a mismatch. Second, when the situation did arise, did the agent exhibit the specified behavior? The reviewer can distinguish an attempt to explain the difference from an entry posted to make it tie.

![A behavior spec and a production run feed into a model judge. The run reports $0.00 variance after posting a $412.06 adjustment. The judge flags the adjustment because it lacks a source.](https://ai.engineer/api/talk-essay-media/bf18ef9059e7fe5f1f41bc73d9ea2a0f2970b803b6fabc0fd836e1abd545f0ad)

The spec and the judge

Those questions make more runs available for supervision. The reviewer can identify a policy violation without independently completing the entire reconciliation. They can also record that the agent pursued an explanation while leaving its correctness open. Outcome evals remain valuable for judging the finished work. The behavior spec lets the team learn from decisions in testing and production much faster than it can supply complete ground truths for new cases.

### Fork before the bad decision to test a change

[13:24](https://www.youtube.com/watch?v=JGtkTjn4120&t=804s)

A failed behavior check identifies a place to investigate. Troyanovsky likens forking to the control beneath a message in a coding agent that lets the user branch from that point in the interaction. Start with a trajectory from an eval, a test, or production. Find the point where the agent chose to post the plug. A fork immediately after the decision lets an engineer ask the agent why it did that. A fork immediately before the decision supports a different experiment. Change the context, let the agent face the discrepancy again, and observe which action it takes.

![A saved point before the $412.06 decision branches into three changes, each tested in five forks. The skill and prompt changes each yield “Sometimes.” The review tool change yields “Yes.”](https://ai.engineer/api/talk-essay-media/b1fd8dc8e215abb8e734713bcfce69d89faa13dd88d736ec2be488ac0061cc5a)

Forking

The usefulness of that experiment depends on how faithfully the fork recreates the original conditions. In a cloud system, copying the conversation may be the easy part. The engineer may also need the database environment, the prompts as they appeared at that point, and the surrounding environment the agent was working in. If the bank data differs between branches, a different decision is harder to attribute to the revised instruction.

The advantage over starting from a bad final result is the location of the failure. Troyanovsky imagines a five-hour trajectory containing a thousand inference calls. Learning at the end that the outcome was wrong leaves a considerable search through those calls. A meaningful behavior check points to a particular choice and gives the engineer something observable to test when revising the agent's harness.

### Reproduce the mistake with a sub-agent before rebuilding the environment

[15:15](https://www.youtube.com/watch?v=JGtkTjn4120&t=915s)

Troyanovsky's quicker experiment starts inside a coding agent. Ask it to create a sub-agent to play the role of the production agent. Give the sub-agent the system prompt and skills rendered as they would appear during the job. Instruct it to stay within the information available in that role. Then describe the situation from the eval or production run and ask what it would do. The first result to look for is the original mistake. A simulation in which the agent immediately does the right thing tells the engineer little about how to repair the observed failure.

If the sub-agent reproduces the bad call, vary the context and ask again. The engineer can try several ideas without repeatedly restoring an entire environment. The tradeoff is fidelity. A sub-agent describing what it would do in a supplied scenario gives weaker evidence than a fork taking an action under the reconstructed conditions. Troyanovsky used simulations for more than a year before Basis built forking into its internal systems.

![Six steps describe simulating a production agent. Three checked examples show the simulated agent rereading a page after an OCR error, posting a documented bank fee and flagging a missing page for review.](https://ai.engineer/api/talk-essay-media/19fdcd3d01f275bb84dcac4c3576fcf2cdb24fcdb5d2dc64026a3a6d60525b85)

Simulations

The original opinion gives these experiments a common target. The team can spot the unwanted decision in a real run, reproduce it, change the context, and inspect later trajectories for the same behavior wherever it applies. That closes a feedback loop around a production decision even when the team cannot grade every production outcome.

### Delegate according to the agent's track record and the cost of a mistake

[17:10](https://www.youtube.com/watch?v=JGtkTjn4120&t=1030s)

An agent can start with a narrow assignment. People who work with it get to observe what it handles reliably, where it needs help, and whether its competence improves. Troyanovsky compares the resulting trust to reviewing a pull request. Two things determine how closely a colleague will read it: the blast radius of the change and their assessment of its author. A change to a core service from someone whose work inspires little confidence warrants close inspection. An internal documentation edit from a trusted contributor can receive a lighter review.

![A chart plots trust from low to high across the page and the consequences of errors up the page. Arrows connect three starred steps, from reading a statement through cash reconciliation to the month-end close.](https://ai.engineer/api/talk-essay-media/b7ee6bf4dd36b9bac04d78ea7990ef7c5325733427a7676f34eff55ab7363b99)

Back to the firm

Left to choose its own method, an agent might discover a better one. Troyanovsky grants the possibility of a surprising “move thirty-seven,” then asks who wants an agent to try one on their tax return. An accounting firm needs to deliver returns reliably, at scale, in a way its people understand. For agents to spread through the firm, employees need to find them simple to use. To trust them with more work, employees need to see them behave reliably. Specifying how the agent should act also gives its builders something to learn from. When a production run departs from the expected procedure, they can investigate the decision, revise the context, and check whether the agent handles the next such case better.

## Key takeaways

- Choose consequential procedures at the firm or team level so each employee can delegate work without configuring the agent's method.
- Write each important behavior with a recognizable trigger and an observable action. Score reconciliation runs for discrepancy handling only when a discrepancy occurred.
- For a bad decision, fork after it to ask for an explanation or before it to test revised context. A sub-agent simulation offers a cheaper first experiment if it can reproduce the mistake.
- Set review depth using both the consequences of an error and the agent's demonstrated competence on that kind of work.

## Slides

### How do organizations scale?

[1:38](https://www.youtube.com/watch?v=JGtkTjn4120&t=98s) [An accounting firm’s head oversees two managers, four reviewers and eight preparers. Clients send their books every month. Dashed boxes show a vineyard and a hotel joining the client list.](https://assets.ai.engineer/sites/aiecode2025/sha256/d39138978e5ded9b4885cf51ff32f17e9a2a988c32f5fc49616ed1a51614f514.webp)

AI Engineer NYC                                                   2026

How do organizations scale?

An accounting firm
Head of the firm
Managers
Reviewers
Preparers

How can the head of the firm trust it'll work?

Their books, every month
Its clients                                      New clients keep coming
Dental practice    SaaS startup    Coffee roaster    Medical clinic    Real estate LLC    Vineyard
Construction co.    Family office    Bakery    Car dealership    Biotech    Hotel

Basis

### A process the firm signed off on

[2:19](https://www.youtube.com/watch?v=JGtkTjn4120&t=139s) [The head of the firm signs off on a four-step process repeated for five clients. A bracket under the last three steps ties those choices to the firm’s risk tolerance, work and clients.](https://assets.ai.engineer/sites/aiecode2025/sha256/1218710aae1eb23c2cf63d1f9537a161d0b9e051477d6b0fe18e92459b458894.webp)

AI Engineer NYC                                                   2026

A process the firm signed off on

Head of the firm    ✓ Signed off
Categorize → Reconcile cash → Explain changes → Review

Clients
Dental practice    Categorize → Reconcile cash → Explain changes → Review
SaaS startup    Categorize → Reconcile cash → Explain changes → Review
Coffee roaster    Categorize → Reconcile cash → Explain changes → Review
Medical clinic    Categorize → Reconcile cash → Explain changes → Review
Real estate LLC    Categorize → Reconcile cash → Explain changes → Review

Chosen for our risk tolerance, our work, our clients.

Basis

### Even the smallest step has a process

[3:05](https://www.youtube.com/watch?v=JGtkTjn4120&t=185s) [The month-end close diagram expands cash reconciliation into four steps. The first, getting bank data, leads to a bank statement and the firm’s chosen extraction method. Two improvised alternatives are crossed out.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a21168168f67fe3a2db328557ac5728c523d618b293856d0e2edc7c6867eb1f.webp)

AI Engineer NYC                                                   2026

Even the smallest step has a process

Month-end close    Categorize → Reconcile cash → Explain changes → Review
Cash rec    Get the bank data → Match to the books → Chase differences → Workpaper

Bank statement
The firm's way:
the extraction software we chose, used the way we use it.

Google it    Figure it out on the fly

Firms scale on process and trust.

Basis

### Coding-agent-style products outsource complexity to the user

[4:29](https://www.youtube.com/watch?v=JGtkTjn4120&t=269s) [Three users produce different numbers of checks and crosses. Below, three ways of matching to the books produce documents that look alike, even when one user chases differences and another forces a tie. The heading overlaps the start of the subtitle.](https://assets.ai.engineer/sites/aiecode2025/sha256/73b65cdb7f180b83993524c4cd3f135c118381a01b2459f74ffe1f5b280760c8.webp)

AI Engineer NYC                                                   2026

Coding-agent-style products outsource complexity to the
user
of mind over an alien intelligence.

Competency gap
Slop cannon
× × ✓ × × ×
× ✓ × × ✓ ×
Not productive
✓
Really good
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓

Process gap
Each person decides the process for themselves.
Match to the books → Chase differences
Match to the books
Match to the books → Force a tie
Look the same

Basis

### What does it take to build production agents?

[5:09](https://www.youtube.com/watch?v=JGtkTjn4120&t=309s) [Two arrows map the competency gap to simplicity for the user and the process gap to reliable performance across situations.](https://assets.ai.engineer/sites/aiecode2025/sha256/766d8a5fc2c814f2bbeac5bb52f1b8f5d96ec7fa8962b255b28963fbf110a49d.webp)

AI Engineer NYC                                                   2026

What does it take to build production agents?

Solve these two gaps

Competency gap → Simple
No agent whisperer required.

Process gap → Reliable
Consistent across the universe of situations.

Basis

### What does simple and reliable look like?

[5:59](https://www.youtube.com/watch?v=JGtkTjn4120&t=359s) [A user hands a bank statement to an agent with several possible extraction methods. Below, document icons illustrate the range of inputs, from digital bank statements and scans to layouts the agent has yet to see.](https://assets.ai.engineer/sites/aiecode2025/sha256/eca1d9f473b5b811e30f0d3e379ba8bc59d49e219f4932616fd4de8984864cd5.webp)

AI Engineer NYC                                                   2026

What does simple and reliable look like?

Simple
No opinion needed.
No special prompt.
It just works.

You
Get the bank statement out.
Agent
Run one OCR
Just look at it
Run ten OCRs, compare, investigate
OCR, then visually verify everything
Write code to parse it
•••

Reliable
Not just this one.
All bank statements,
or even all PDFs.

Big bank, digital
Credit union
Scanned
Table across pages
New layout
Not seen yet

Basis

### Getting it right isn't the task

[6:15](https://www.youtube.com/watch?v=JGtkTjn4120&t=375s) [Three paths lead from a home-office deduction question to the same right answer. The primary-source path is highlighted. Below, filled dots represent familiar tax situations and open dots represent situations the agent has yet to encounter.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba45a2dce2dc544fbbc0634f5487887c650fd43b58f0801952158be8b35cf403.webp)

AI Engineer NYC
2026

Getting it right isn't the task

Can this client deduct their home office?
Agent
From pretraining
From Wikipedia
From the primary sources
Right answer

At Basis: primary sources before any tax conclusion.

Home office
Rental property
Stock sales
Partnership K-1
Moved states
Crypto sales
Situations you haven't seen yet

Getting it right reliably, at scale, is.
Basis

### An example opinion: reading any PDF

[6:52](https://www.youtube.com/watch?v=JGtkTjn4120&t=412s) [A flowchart routes a PDF by the document type inferred from its filename. Semi-structured documents go to OCR A and structured documents to OCR B. Both paths add a vision-model check if OCR may miss tables or diagrams.](https://assets.ai.engineer/sites/aiecode2025/sha256/9241c537730346962ebbacfd658ffa177bb41b1a0dfe217d8e29882689c34d57.webp)

AI Engineer NYC
2026

An example opinion: reading any PDF

mar_statement.pdf
Infer its type from the name
Semi-structured
OCR A
Structured
OCR B

Tables or diagrams OCR may miss?
Yes
Also verify with a vision model
More expensive

Simple   The user never chooses how.
Reliable   The same rule on every PDF.
Basis

### How do you develop opinions?

[7:40](https://www.youtube.com/watch?v=JGtkTjn4120&t=460s) [Three rows show six outcome evaluations with five checks and one cross, four targets to consider, and a range from bank statements to any PDF. The “Us” marker sits at the any-PDF end.](https://assets.ai.engineer/sites/aiecode2025/sha256/4bb6c8efca5a3d3bdadb04387f870cf6eb971c7692d3ada008f9dc8b9258524f.webp)

AI Engineer NYC
2026

How do you develop opinions?

Outcome evals
✓  ✓  ×  ✓  ✓  ✓
What actually works

Your targets
Latency
Cost
What the model can do
What you're OK with

Your situations
Only bank statements
Specific approach

Us
Any PDF
General approach
Basis

### How do you know your outcome evals generalize to production?

[9:31](https://www.youtube.com/watch?v=JGtkTjn4120&t=571s) [Three bold lines mark outcome evals on a bell-shaped distribution of production situations. A callout enlarges one situation in the right tail, showing several possible paths with one behavior highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/ccbb342b3631cb81f591a5424f2ed1bdab051b516f1afff473bd6f994c115718.webp)

AI Engineer NYC
2026

How do you know your outcome evals generalize to production?

Your outcome evals
A situation in production
Situations in production

One situation
The behavior

The evals are the only ones with true ground truth

I can't prove this one is right.
I can test whether the behavior is happening.

Process supervision
Behavior specs
Forking
Simulations
Basis

### Here's a behavior we want

[10:17](https://www.youtube.com/watch?v=JGtkTjn4120&t=617s) [A cash reconciliation shows a $412.06 difference between the bank statement and the books. A note traces the discrepancy to an OCR error. The stated rule directs the agent to investigate and explain the difference.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d9fb08dbe207aa3f9ceffae3b873eeda5e4ff461f8c3b445f7a62749f8df70d.webp)

AI Engineer NYC
2026

Here's a behavior we want
Example

When a reconciliation doesn't tie, the agent
investigates and explains the difference.
It never posts an entry just to make it tie.

Cash rec
Bank statement   $48,210.55
Books   $47,798.49
Difference   $412.06

Why: that scan's OCR misread one line,
and it flowed into the cash rec.
Basis

### What actually decides the behavior?

[10:52](https://www.youtube.com/watch?v=JGtkTjn4120&t=652s) [Five cards show what shapes the agent’s reconciliation behavior. Highlights connect the instruction to bring the account into agreement, the review tool’s balance check, and the model’s tendency to pursue a passing result. A dashed border distinguishes the unwritten model instincts.](https://assets.ai.engineer/sites/aiecode2025/sha256/92b8d22ef4d83fd884ec0250074ca3d32ab8f3e71db1e18826902a6b33d9748b.webp)

AI Engineer NYC
2026

What actually decides the behavior?
Example

System prompt
You prepare work that ends up in clients' books. Deliver work a reviewer can sign off on: complete, reconciled and explained.

Month-close skill
Reconcile each cash account to its bank statement for the period. The adjusted book balance should agree to the statement. Investigate differences, then record the adjustments needed to bring the account into agreement.

Review tool
check_reconciliation
Checks a completed reconciliation. Returns PASS when the adjusted book balance equals the statement balance; otherwise FAIL, with the variance.

Adjustment tool
post_adjustment
Records a reconciling adjustment to an account, with an amount and a memo.

Model instincts
written nowhere
Wants the check to pass. Treats ‘agrees’ as the goal.

In a complex system that handles a massive distribution of situations, agent behavior is the combination of all of this.

You can't write every desired behavior into context explicitly.
Basis

### Behavior specs

[12:02](https://www.youtube.com/watch?v=JGtkTjn4120&t=722s) [Three builders point to one agreed behavior spec. A second diagram places the spec outside the agent’s context, beside the system prompt, skills and tools the agent reads.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c03b91f68abfcfe5676b47dab68265dd03bf4b5e9e745b3aa4542319845cb27.webp)

AI Engineer NYC
2026

Behavior specs
Behavior specs   Forking   Simulations

1   Align on the opinions.
A forcing function, for us as builders.

Builders
Behavior spec
One opinion, agreed

2   Define the ones that matter most.
Apart from the prompts, skills and tools.

System prompt
Skills
Tools
What the agent reads

Behavior spec
What we want
Not in its context

Open-sourced with Braintrust · agentbehavior.dev

Basis

### The spec and the judge

[12:36](https://www.youtube.com/watch?v=JGtkTjn4120&t=756s) [A behavior spec and a production run feed into a model judge. The run reports $0.00 variance after posting a $412.06 adjustment. The judge flags the adjustment because it lacks a source.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf18ef9059e7fe5f1f41bc73d9ea2a0f2970b803b6fabc0fd836e1abd545f0ad.webp)

AI Engineer NYC
2026

The spec and the judge   Example
Behavior specs   Forking   Simulations

Behavior spec
When a reconciliation doesn't tie, the agent
investigates the difference and explains it with a
source. It never posts an adjustment just to
make it tie.

Applies: any reconciliation with a difference.
Not: a difference already explained by a sourced item,
like a bank fee on the statement.

Production run
Searched the ledger for $412.06. Nothing found.
Posted “Reconciling adjustment, $412.06”
check_reconciliation: PASS
Reported “$0.00 variance”

Judge (a model)

Applies?   Yes
Followed?   No

Evidence
“Posted an adjustment with no
source. Never re-checked the
statement.”

The rec ties.
The judge still caught it.
No answer key needed.

Basis

### Forking

[13:21](https://www.youtube.com/watch?v=JGtkTjn4120&t=801s) [A saved point before the $412.06 decision branches into three changes, each tested in five forks. The skill and prompt changes each yield “Sometimes.” The review tool change yields “Yes.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b1fd8dc8e215abb8e734713bcfce69d89faa13dd88d736ec2be488ac0061cc5a.webp)

AI Engineer NYC
2026

Forking   Example
Behavior specs   Forking   Simulations

The flagged run
Followed?   No

Saved point
just before it
decides what to do
about the $412.06

Change the harness   Forks   Followed?

Month-close skill
“...record adjustments only for differences you
can explain with a source.”
× 5   Sometimes
The review tool still fails anything
that doesn't tie.

System prompt
Add “Never force a reconciliation to tie.”
× 5   Sometimes
For the same reason.

Review tool   check_reconciliation
“...PASS when every difference is explained by a
sourced item or flagged for the reviewer.”
× 5   ✓ Yes
No side effects.

Iterate on the harness. The fix was the tool, not the prompt.

Basis

### Simulations

[15:14](https://www.youtube.com/watch?v=JGtkTjn4120&t=914s) [Six steps describe simulating a production agent. Three checked examples show the simulated agent rereading a page after an OCR error, posting a documented bank fee and flagging a missing page for review.](https://assets.ai.engineer/sites/aiecode2025/sha256/19fdcd3d01f275bb84dcac4c3576fcf2cdb24fcdb5d2dc64026a3a6d60525b85.webp)

AI Engineer NYC
2026

Simulations   Example
Behavior specs   Forking   Simulations

1   Render the agent's prompt, skills and
tool descriptions into files.
2   A coding agent spawns subagents.
3   Each reads the files and pretends to be
your production agent.
4   Give it a situation.
5   See what it decides.
6   Ask why. Find the cause in the files.

The tool fix, across situations

A $412.06
OCR misread
Re-reads the page,
finds it   ✓

A $35 bank fee
on the statement
Posts a sourced
adjustment   ✓
A real adjustment, not a plug

Page 3 of 5
missing
Flags it for
the reviewer   ✓

Plays your agent

Cheaper, faster than forking. A smoke test while you iterate on the harness.

Basis

### Is it generalizing?

[16:21](https://www.youtube.com/watch?v=JGtkTjn4120&t=981s) [A bracket marks outcome evals across the middle of a bell-shaped curve of production situations. Check marks extend beyond the bracket on both sides. Two crosses mark failures in the right tail.](https://assets.ai.engineer/sites/aiecode2025/sha256/285dc40664adcbf2bd58ae94e3b9512cf69948d973f0f2b5061b17ab784f13c0.webp)

AI Engineer NYC
2026

Is it generalizing?

Your outcome evals

Situations in production

✓   Behavior specs check it on runs, even without ground truth.
It shows you where the behavior holds, and where it doesn't.

Basis

### Opinions close the loop

[16:50](https://www.youtube.com/watch?v=JGtkTjn4120&t=1010s) [Written opinions feed a signal box that contrasts one final outcome score with behavior checks along a run. Arrows connect that signal to testing fixes, improving the system and observing the agent in evals and production.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c790da3590892aeb4627e18d8f8b6b7fcb53c27328d4f352bc007d9933b3679.webp)

AI Engineer NYC
2026

Opinions close the loop

Opinions
written down

Signal
Outcome evals
A few cases, one score at the end.
Which step, which behavior?
Behavior
specs
Every run, in evals and in production.

Forks and simulations: test the fix

A reward function,
one level up.

Improve the system
the harness, sometimes the
opinion

The agent works
in evals and in
production

Agents can run more of this
loop,
if they know what better means.

Basis

### Back to the firm

[17:07](https://www.youtube.com/watch?v=JGtkTjn4120&t=1027s) [A chart plots trust from low to high across the page and the consequences of errors up the page. Arrows connect three starred steps, from reading a statement through cash reconciliation to the month-end close.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7ee6bf4dd36b9bac04d78ea7990ef7c5325733427a7676f34eff55ab7363b99.webp)

AI Engineer NYC
2026

Back to the firm

What breaks if they're wrong
The month-end close
Line by line
The cash rec
Read the statement
Evidence builds
Process supervision
A glance

Low
How much you trust them
High

The head of the firm
trusts agent work the
way they trust their
people's.

Basis

### Won't opinions stop it finding a better way?

[18:08](https://www.youtube.com/watch?v=JGtkTjn4120&t=1088s) [Three rows pair a Go board marked 37 with a line about someone's close, four checkmarks with reliable work at scale, and two Opinion boxes joined by an upward arrow with improving models.](https://assets.ai.engineer/sites/aiecode2025/sha256/a2b75e736480f970590e13c3b2e0130376808d8848e847c92a302cf5d818997a.webp)

AI Engineer NYC
2026

Won't opinions stop it finding a better way?

37
We're not paid to Move 37 someone's close.

✓ ✓ ✓ ✓
We're paid to do the work reliably, at scale.

Opinion
↑
Opinion
Opinions move up as models improve.

Basis

### If you take anything away:

[18:41](https://www.youtube.com/watch?v=JGtkTjn4120&t=1121s) [A dark slide lists four numbered takeaways about agent adoption, team trust, how work gets done, and feedback for agents in production. Bold type emphasizes “simple,” “reliable,” “opinions,” and “additional signal.”](https://assets.ai.engineer/sites/aiecode2025/sha256/ce499dc30c5c2f26729b15ffca5202fec5c1de5053d1bbb212fa3670a76703e7.webp)

AI Engineer NYC
2026

If you take anything away:

1  For agents to diffuse, they need to be simple.

2  For teams to trust them, they need to be reliable.

3  Simple and reliable come from opinions about how the work gets done.

4  Opinions let you build additional signal that helps close the loop on self-
   improving agents in production.

Basis

## Full transcript

[0:12](https://www.youtube.com/watch?v=JGtkTjn4120&t=12s) I'm Mitch. I'm one of the two co-founders of Basis. We are based right here in New York City, and the thing that we focus on is how can we get agents to do production scale work reliably enough so that we are trusted and used end to end by accounting firms across the country. So a lot of our research really focuses on this. How can you get these agents to do real work in the real economy? So

[0:43](https://www.youtube.com/watch?v=JGtkTjn4120&t=43s) I wanted to start with a question, which is if you're in this, at this conference, honestly, if you're watching this talk, you probably increase your own productivity a lot. You're using Cowork or, uh, Codex or Cloud Code, whatever it is. And yet if you go look at the average company, the average enterprise, the average firm, their productivity hasn't gone up ten x, a hundred x. I know my productivity has gone up maybe a

[1:13](https://www.youtube.com/watch?v=JGtkTjn4120&t=73s) hundred x, but if I were working inside of a company, it wouldn't be anywhere near that order of magnitude. And so why is that?

[1:22](https://www.youtube.com/watch?v=JGtkTjn4120&t=82s) At Basis, one of the things we have is that humans are already used to working with non-deterministic systems. It's just those systems are normally their coworkers, not their computers. And so I actually think to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies. So how do organizations scale their own productivity, right? Organizations need to trust that the

[1:52](https://www.youtube.com/watch?v=JGtkTjn4120&t=112s) people doing the work at every level are doing it in a way that is gonna produce good outputs consistently at scale, right? If you look at this image here, the, uh, here I'm showing an accounting firm, but this could honestly be any company, traditional company. The person who is the head of the firm, they have literally no idea about the exact day-to-day of the people at the very bottom, and yet somehow they trust the work is gonna be done. How does that happen?

[2:21](https://www.youtube.com/watch?v=JGtkTjn4120&t=141s) It doesn't happen through deterministic systems, right? People are not deterministic. They're intelligent entities. And instead, there are processes that are put in place, and those processes, if we believe the people who, you know, put them in place think that those processes will generalize to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process. This is true, by the way, even in the smallest steps. So in this

[2:51](https://www.youtube.com/watch?v=JGtkTjn4120&t=171s) example, earlier I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm, but even within that process, say you wanted to see, how do I reconcile the cash? Well, I need to get the bank data, and even that has a process. The process could be really basic. It could just be use the approved software to extract the data, but there's still some process that is put in place. You wouldn't expect the firm to just let anyone Google, you know, extract

[3:21](https://www.youtube.com/watch?v=JGtkTjn4120&t=201s) PDF and then just use whatever they want to go extract it, because now you wouldn't have process.

[3:29](https://www.youtube.com/watch?v=JGtkTjn4120&t=209s) Now let's talk about agents.

[3:34](https://www.youtube.com/watch?v=JGtkTjn4120&t=214s) As I said earlier, everyone already has Cowork, ChatGPT, Codex, Cloud Code, but where is that productivity? Our opinion is there are two main problems that come with what I'll call coding style agent products, where you're working with an agent and you just tell it to go and do something. The first problem is that there is a competency gap because agents, as you would think of them as, you think of them today, they outsource the complexity to the user,

[4:04](https://www.youtube.com/watch?v=JGtkTjn4120&t=244s) right? If I'm the user, I now have to ma-manage my skill and memories and think about how to, uh, when I should use the agent. I have to have, like, theory of mind over it. This is really hard.  And for those who are here, maybe you guys find it easy, but I promise for the wide majority of knowledge workers, they have no idea how to do any of this.

[4:29](https://www.youtube.com/watch?v=JGtkTjn4120&t=269s) And there's a second problem, which is there is a process gap, right? If you are saying, "Hey, user, here's your agent. Do what you want," well, now suddenly the way the work is being done is essentially up to whoever the user is to customize the way the agent is behaving, which goes entirely against the way that organizations scale. The process should be coming from senior leadership down, not from any individual employee deciding how they're gonna do

[4:59](https://www.youtube.com/watch?v=JGtkTjn4120&t=299s) their job, right? If you, going back to my example, if every employee just made up their own way of extracting data, you know, even back in twenty eighteen, that wouldn't be a good outcome. And so we believe strongly that the key to building production agents that actually do work end to end is solving these two problems. You have to solve the competency gap by making it simple. To the user, you shouldn't even have to know it's an agent, or you probably know, but you don't have to really feel it. It should be as simple

[5:29](https://www.youtube.com/watch?v=JGtkTjn4120&t=329s) as just working with a coworker. You just tell it what you want it to do, and all the things you expect to work should work. And the second is the process gap. You need agents to be reliable, not having a stroke of genius one day and forgetting something the other day. It's not about being perfect. Humans are not perfect, right? Nobody is looking for perfect. What you're looking for is reliability and consistency and trustworthiness. So what does that actually look like?

[6:00](https://www.youtube.com/watch?v=JGtkTjn4120&t=360s) We think that to be simple and reliable, where you know it's-- you don't have to think about what to do, the agent can just do it for you without the user knowing the internals. To do it the same way consistently, you have to be, uh, you have to be opinionated. Just to show you actually one slide back here, I wanna point out that getting it right isn't actually the main task, because again, humans are imperfect themselves. The

[6:30](https://www.youtube.com/watch?v=JGtkTjn4120&t=390s) point is to do it in a similar way. It's okay if you, you can't do something, but you should be consistently able to not do something, and then you get better, just like a person does. If you suddenly one day are an expert at something, and then you miss something very basic that would be expected, now you're not reliable.

[6:48](https://www.youtube.com/watch?v=JGtkTjn4120&t=408s) So you have to be opinionated. What's an example of an opinion? Let's say we have an agent, and we need to go and read a PDF. So in this case, what is the best way to read a PDF? There's many ways. You can throw it into a VLM. You can go write some Python code and use an open source OCR library. You can work with one of the many OCR-specific model providers. You could do an OCR, and then you could have the VLM check its work. You could have five VLMs

[7:18](https://www.youtube.com/watch?v=JGtkTjn4120&t=438s) check its work. You could do a hundred VLMs and take the best of them. There's literally infinite possibilities here. And so if you just said, "Agent, extract Excel... uh, extract PDF," you might get good outcomes, but they're not gonna be reliable, right? Not because the process it did was bad, but because the process it did was not opinionated. So

[7:45](https://www.youtube.com/watch?v=JGtkTjn4120&t=465s) the way we develop opinions is you use initial intuition, right? Some things you know what the opinions are from expertise. For example, if I'm trying to get an agent to do tax work well, I know that if the agent is gonna-- goes and reads the primary source before returning to the user, it is more likely to be accurate. I don't need a set of evals to tell me that. I don't need to divine that from first principles. We know that, right? I know that if I have an agent make a

[8:15](https://www.youtube.com/watch?v=JGtkTjn4120&t=495s) PowerPoint, if it takes a picture of the PowerPoint before it returns to the user, it is less likely to have a formatting issue. That is obvious. And so there are lots of different ways to develop opinions, and one of the key points here I'll just make before I move on to talking about how to use the opinions is that the opinions are not just about accuracy, right? We just talked about the example of taking a picture of a PowerPoint. Well, that adds latency, right? You may not want that. Maybe the agent only makes formatting mistakes one out of a hundred times,

[8:45](https://www.youtube.com/watch?v=JGtkTjn4120&t=525s) and you're okay with that, and so therefore, you don't have it take a picture of the PowerPoint. The point isn't exactly what the opinion is, and this talk we're gonna focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale in production. So

[9:06](https://www.youtube.com/watch?v=JGtkTjn4120&t=546s) say you have a bunch of outcome-based evals, and when I say outcome-based evals, I mean what we traditionally think of as evals. It has a right answer. We have some rubric. We're trying to judge what the agent did. How do you know that the evals generalize to production, right? You might have a thousand cases or a hundred cases, but in production, if you're doing real work, you're gonna have hundreds of thousands of situations, and that's only gonna increase as the trajectories get much longer

[9:35](https://www.youtube.com/watch?v=JGtkTjn4120&t=575s) because the amount of decisions that are made in a trajectory, uh, just grow exponentially as the hor-- like long-horizon abilities of agents increase.

[9:47](https://www.youtube.com/watch?v=JGtkTjn4120&t=587s) And so I can't prove if I-- If you're looking at all these distribution of, uh, trajectories, you can't prove every one of them is correct. You don't have a ground truth for every trajectory. But you can potentially look at the trajectories to decide if the behavior, the opinion that you have is actually happening. And you can then infer that if it's actually happening, it's more likely to generalize to production. It's not perfect. You don't know for sure, but at least you are having an opinion.

[10:18](https://www.youtube.com/watch?v=JGtkTjn4120&t=618s) Let's run through a real example. So let's say that I, uh, for behavior, this is a, an accounting example, where if there is a, a mistake for, uh, in, in the bank reconciliation, um, uh, I don't want the agent to like post an entry to make it tie. I want it to, uh, sort of explain the difference instead of putting a plug. Again, this is subjective. You might want the other way. But what

[10:48](https://www.youtube.com/watch?v=JGtkTjn4120&t=648s) you don't want is a customer getting one way one time and another way the other time. And so what would you do? Well, first, let's understand what governs the behavior, right? The behavior is not just you, you write the two sentences into your system prompt and it just works. That might work for very simple, narrow agents. But when you're building these big agent systems, the behavior is actually a combination of the system prompt, your progressive disclosures, y- uh, potentially tool descriptions, obviously the training of the model. There's a lot of things that govern the behavior.

[11:19](https://www.youtube.com/watch?v=JGtkTjn4120&t=679s) And one tip, by the way, I have for when you think about building production agents is really think of the context the agent is gonna see as it's working as runtime training data. The word prompt, I think, trivializes it a bit, right? At the end of the day, this is the data that the agent is being trained on at runtime. Not literally trained on, but effectively trained on. And so you wanna care about every single token and understand how that data causes the agent to behave in certain ways. So when

[11:49](https://www.youtube.com/watch?v=JGtkTjn4120&t=709s) you're putting, uh, behaviors into production, it, it might be very difficult for one behavior to put it in exactly one, you know, paragraph that you know exactly because, again, the behavior is all of the data at once, not just one thing. And one of the things that we developed at Basis was this concept of a behavior spec. And the idea of a behavior spec is essentially saying, before we update the context or write the context, uh, what do we

[12:19](https://www.youtube.com/watch?v=JGtkTjn4120&t=739s) actually want? What is our intent? You can kind of think of this a little bit as like a product spec for the agent era. And you can't write-- You shouldn't be writing every detail down. You should be taking the most important opinions and putting it, because if you put everything, you'll be duplicating a lot of states. So you define the ones that matter the most. And then you can use them as a way to align your team from a spec perspective, but then you can also use them as a rubric. And you can say, in this particular trajectory,

[12:49](https://www.youtube.com/watch?v=JGtkTjn4120&t=769s) whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant? And then if so, did it actually exhibit the behavior? And this way, you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome-based evals, ground truths, and rubrics that you have. Those are really important, but you cannot scale those anywhere near as fast as you can scale the ability to

[13:19](https://www.youtube.com/watch?v=JGtkTjn4120&t=799s) supervise what the behaviors of the agent are.

[13:24](https://www.youtube.com/watch?v=JGtkTjn4120&t=804s) Let's just take a quick example here. Um, this is-- So this is actually the second concept, um, which is called forking, which we can combine here with behavior specs to think about. Forking, if you've ever used, uh, Cloud Code or, or Codex, the web apps, there's this little fork icon, uh, under each message, uh, where you can essentially create a copy of the agent using the same thing, which works really well on your local computer. It's hard to build a system that does this in the cloud perfectly with recreating the database environment

[13:54](https://www.youtube.com/watch?v=JGtkTjn4120&t=834s) and recreating the, uh, the prompts, uh, all the, all the different environments. But forking is another tool that we use to check the behavior. So a flow that you can imagine is saying, we had a trajectory, could be in an eval set, it could be in testing or in production, and it failed the behavior eval. And now you're trying to understand why. Why did it fail, fail the behavior eval? Why did it not exhibit the behavior? And then you can go into the trajectory, go right

[14:23](https://www.youtube.com/watch?v=JGtkTjn4120&t=863s) before the point in which the agent made the decision that was in contrast to the behavior, assuming again, your behavior is good, and it's actually producing signal, and then you can fork it. And when you fork it, you can ask the agent, uh, or you would fork it right after it made the decision. You could ask it, "Why'd you do that?" Or you can go and update the, uh, context of the agent and see if that improves the behavior. And so with forking, you can start to actually take the

[14:53](https://www.youtube.com/watch?v=JGtkTjn4120&t=893s) signal and start to figure out, okay, how can we then improve the harness to now, uh, leverage the signal? Versus if you're just looking at signal from an outcome-based eval, you can't do that anywhere near as easily because it failed the outcome. How do you know why? You have potentially a thousand inference calls in your five-hour-long agent trajectory run. And then we have the last tool that I like to talk about, this is actually my favorite, which is called simulations. Uh, it's gonna sound silly, but I promise it's very

[15:23](https://www.youtube.com/watch?v=JGtkTjn4120&t=923s) effective, which is that you can, when working with coding agents, ask the coding agent to make a sub-agent that pretends to be your production agent. And you have to tell it to not cheat, and you give it the system prompt and the skills, and you render everything perfectly. And what you can do is you can ask it what it would do in some scenario, you can give it the situation, and see if you can recreate the situation that you saw in your eval or in production. And if you can recreate it, now you can tweak your context to see if the sub-agent would change its behavior based

[15:53](https://www.youtube.com/watch?v=JGtkTjn4120&t=953s) on your tweaks. And the reason to do this, it's obviously far worse than forking, but you do this because this is much cheaper, much faster, and allows you as an agent builder to really easily iterate on ideas w-without needing to go through the expensive process of actually forking an entire environment. And you can do this today without building anything. Um, I did this for literally year, year and a half plus before we, uh, built forking into our internal systems.

[16:23](https://www.youtube.com/watch?v=JGtkTjn4120&t=983s) And the main idea here is understanding that if you look at the, the distribution from earlier, you're trying to understand, is the beha-- is the, uh, agent generalizing to that distribution? And so you don't know b- purely based on the outcome evals. You can try to choose the best outcome evals that sh- represent the curve, but you don't really know. And so supervising the process through some of these methods gives you some additional signal in order to understand if it's generalizing properly. And what's really

[16:53](https://www.youtube.com/watch?v=JGtkTjn4120&t=1013s) cool is as you start to get more signal, you can actually start to close the loop. And an opinion is what lets you do that, right? If you don't have opinions about your system, you're not gonna be able to develop that signal.

[17:10](https://www.youtube.com/watch?v=JGtkTjn4120&t=1030s) And now if we go back to the firm, we talk about, talk about supervision, we talked about how the agent, uh, if user opinionated, can be more reliable and actually can be simpler. What that means is the agent, maybe it starts doing something narrow, but because people will work with it, they see that it's easy to use, and they see that it's reliable. As it gets better and better, they will trust it more. It's just like a coworker. You have a coworker, day one, imagine someone sends you a PR. You're

[17:40](https://www.youtube.com/watch?v=JGtkTjn4120&t=1060s) not gonna... If it's-- You're looking at two axes whenever someone sends you a PR. It's these two axes. What's the blast radius of the PR, and what is your perception of how competent that person is, right? If the least competent person at your company is sending you a PR that's editing a core service, you're gonna look at it.  If the most competent person is sending you a PR that's editing an internal document, you may not look at it as much. And so this idea is true with humans working with agents just like it is with humans working with humans.

[18:10](https://www.youtube.com/watch?v=JGtkTjn4120&t=1090s) And I think I just wanna make one last point to close here, which is that one of the things that people say when they hear this is, "Well, wait a second. If you let the agents just do what they want, won't they find a better way? Won't they move thirty-seven your work?" And the answer is, they might. But that's not how you scale GDP and scale output. Nobody wants their agent to move thirty-seven their tax return. They want it done reliably, at scale, in a way that they can trust, and in a way that their

[18:40](https://www.youtube.com/watch?v=JGtkTjn4120&t=1120s) people know how to work with. And so the thing I will leave you with is for agents to diffuse, they need to be simple. For teams to trust them, they need to be reliable. The way you get to simple and reliable is you develop opinions on how the agent behaves, and then opinions allow you to build that additional signal that helps you start to close the loop so that you can achieve self-improving agents in production. Thank you very much.
