Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI

Read the talk

What to Build Before the Next Model Arrives

Blackstone CTO John Stecher weighs the value of shipping six months early, while Norm AI CTO Scott Worland explains how tens of thousands of expert-labeled examples gave his team the confidence to replace software that worked.

From a talk by John Stecher and Scott Worland

At a glance

Ideas worth remembering

  • Trace the data a proposed agent needs back to its source systems. Map the workflow separately so ordinary code can carry predictable steps straight through and a model can handle decisions that warrant it.

  • Keep expert-labeled evaluations usable across implementations. Rerun them when the model, harness or data changes, and use the same cases to test whether removing workflow steps improves results.

  • For a proposed six-month delay, estimate the value of having the application during those six months. When usefulness is uncertain, a clickable proof of concept can resolve that question before a production commitment.

  • Give people explicit ownership of sensitive platform primitives. For the internal applications built above them, compare the next round of maintenance with the current cost and value of a replacement.

Two views of the same deployment decision

John Stecher sees technology proposals from three positions at Blackstone. He runs the firm’s technology, advises on investments and serves as a technical resource for its portfolio companies. When a team proposes an agent, he has to consider what it could do for the people using it and what it would take for the organization to operate it. Scott Worland, who introduces Stecher, comes at the question from the supplier’s side. His company, Norm AI, builds an enterprise AI platform for legal and regulatory work. Some of the world’s largest financial institutions use it. So does Norm Law, an affiliated law firm.

Worland opens with a question prompted by the rush to deploy agents. As more companies commit to building them, how does Blackstone decide what it takes to get one right? Stecher begins well before model selection, with the path a piece of information has to travel before an agent can use it.

0:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Get proprietary and public data out of transactional systems

Stecher dates a noticeable improvement in usefulness to around February or March of the previous year. Context windows had grown, and models could keep working on a task for significantly longer. People could get more value out of a run. But when he advises portfolio companies that want to build as many agents as possible, the first impediment he looks for is whether an agent can obtain the material the company expects it to work on.

At Blackstone, an asset manager, proprietary information and public information are woven together in the work people do. Some of the relevant material lives in transactional systems. A larger context window helps once the material reaches the model. It does little for a task whose inputs are still scattered across those systems. Stecher wants a cohesive strategy for moving that data somewhere the firm can manage it and from which an agent can retrieve it. The practical question for a proposed agent is whether that path exists for the particular mix of information its job requires.

1:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:17 · section reference included

Use agents to build straight-through processing and to handle specific decisions

Stecher gives agents two different jobs in the same application. During development, he has used Factory and other coding tools to help him work through what people actually do and craft software around that workflow. His design objective is still straight-through processing. Where the steps are understood and can run in sequence, he wants code that carries the work through. Looking closely at the workflow also reveals the places where progress has depended on a person making a decision, sometimes a fairly rote one. Those are places to consider putting model intelligence into the running application.

The distinction affects who gets time back. An investment professional or finance professional may have had to stop and make a routine judgment each time a piece of work crossed their desk. An engineer may already know exactly how an application should behave but still have to write the functional test cases, write the performance test cases and bring the libraries underneath the application up to newer versions. Each is a specific piece of work to hand off, with an identifiable person who can use the time for something else.

Dependency upkeep is a particularly personal example for Stecher. He says he has disliked that recurring chore throughout 27 years in engineering. An agent that can go into his source, renovate the code and take on some of the testing has made a material difference to how quickly his team delivers. That experience informs the question he puts back to Worland. Which parts of the work has Norm found it worthwhile to give to an agent?

2:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:17 · section reference included

How Norm got consistent regulatory outputs in early 2024

Norm was founded in 2023. Worland remembers the team’s first get-together, a week on Stanford’s campus, as a period when every assumption about model inputs seemed to be changing at once. In his recollection, context windows went from 4K to 8K to 128K, and multimodal capabilities arrived too. He qualifies the chronology himself. What stuck with him was the pace. Then improvements from the frontier seemed to slow for a while, just as Norm needed to turn the models it had into production software.

By early 2024, Norm’s engineers were making decisions explicitly at several layers. They knew certain content would eventually have to enter a model’s context and specified how to get it there. They prescribed the sequence in which models ran. Where a task required consistency checks, they inserted those steps, then distilled the results into an output someone could use. Much of the application’s knowledge about how to complete the job lived in the code coordinating those calls.

Worland likens the result to a large MapReduce problem. The comparison conveys how much work went into organizing individual operations and bringing their results together. He starts to describe this as scaffolding for agents, then corrects himself. The software was much closer to a workflow controlling how models ran. It earned its place by producing accurate results with considerable consistency. For regulatory work, getting a good answer on one run was only part of what Norm needed from the system.

3:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:46 · section reference included

Tens of thousands of labeled examples made the old workflow testable against an agent

The size of the workflow created a regression-testing problem. Norm needed a way to tell whether a change had disturbed the quality it had worked so hard to achieve. In building that test capability, it accumulated tens of thousands of examples labeled by experts. Their first purpose was to test the existing implementation. Their lasting value was that they described acceptable work in a form Norm could also use to test a substantially different implementation.

Worland uses a concrete definition of an agent. Give an LLM a set of tools, a goal and some way to verify its work against the goal. Then let it decide how to get there. That changed which decisions Norm had to make in advance. Its workflow had settled the sequence of model calls before a particular task began. The agent could choose how to proceed on the task in front of it. As soon as that approach became viable, Norm could run it against the benchmark it already had.

For the engineers, the result was uncomfortable. They were looking at hundreds of thousands of lines of thoughtfully written code, much of which now seemed obsolete. Some of the upfront structure was actively getting in the way. Worland later says they could get better performance by removing it. He gives the scale of the evaluation set and the codebase, but no numerical uplift. The consequential comparison was between two ways of doing work Norm already knew how to assess.

Worland took the engineering leadership team to upstate New York in a rented Sprinter van after a big snowstorm. At an Airbnb, they asked what platform they would build if they were starting that day. They came away with two requirements for the rebuild. Clients needed agents that could carry out work on their behalf, and Norm’s engineers needed a platform that AI coding tools could help them develop. Worland says the team has been rebuilding the platform to those specifications ever since.

5:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:16 · section reference included

A model, a harness or a data change can move the answers

Stecher picks up on the asset that made Norm’s comparison possible. In financial services, he wants an evaluation set tied to the actual work the agent performs. Does the output make sense for that work? Is it accurate? Can the firm obtain a consistent, repeatable result? These questions matter after deployment as much as they do when choosing an architecture.

He identifies three changes that can affect answers. First, a model version can change underneath an application. Stecher has seen the answers change with it and the accuracy of the information shift. Second, a change to the harness that runs the model can produce different answers. Third, a change in the data can do the same. Running the evaluations consistently gives the team a measure of the health of the assembled system. A model comparison performed when the team first bought or built the software cannot establish how that software behaves under today’s combination of model, harness and data.

6:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:44 · section reference included

The bitter lesson reaches the project roadmap

Worland connects Norm’s experience to Rich Sutton’s bitter lesson, which he dates to 2019. Clever people can invest considerable effort encoding how to solve a problem, only to see an approach that scales with compute eventually outperform their system. He puts the idea in a longer history, from the excitement around Rosenblatt’s perceptron in the 1950s and 1960s to the disputes between proponents of expert systems and connectionist approaches. He recalls early press coverage crediting the perceptron with the prospect of original thought, though he is unsure which publication ran it.

His most pointed example comes from an anecdote about IBM’s translation lab in the 1980s. A lab leader joked that the models improved each time he fired a linguist. The technical issue underneath the joke was where to put effort. Linguists could keep trying to formulate the rules of language, while a statistical translation model could learn from probabilities. Norm had encountered its own version of that choice when removing carefully prescribed steps improved performance. Worland asks what an engineering leader should do with this knowledge when someone brings in a project idea that has to be funded today.

7:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:40 · section reference included

What would Blackstone gain by shipping six months earlier?

Stecher starts with the return on the actual request. The request might be to build an agent, put data where it can be used, make a dashboard or develop a trading platform. He works through its value with engineers, product people, the site reliability engineering function and the stakeholders asking for it. On the other side of the comparison is the true cost of building it with the capabilities available today. A belief that enough future compute will make a problem solvable leaves both sides of that investment decision open.

Timing makes the comparison more precise. Suppose a more sophisticated model and sufficient compute are likely to make the problem cheaper to solve in six months. What business value depends on having the application during those six months? The total eventual value of the application may be considerable even when the value of getting it early is small. Blackstone has chosen to defer work when the need was modest, the expected build cost would fall and producing the application immediately offered little additional return.

Six months is Stecher’s example of a decision horizon. He describes making the trade-off regularly, with the urgency and economics of the particular request determining the answer. There is also a cost to repeatedly deciding to wait. If the open question is what someone would actually do with the proposed application, the next model release may arrive without the team having learned anything about demand.

8:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:54 · section reference included

Build the version a stakeholder can click through

For that question, Stecher shortens the time horizon. Start with a product person or the stakeholder, use a tool such as Fable to work through the concept and plan, then write enough code for someone to use it. His threshold for a workable proof of concept is straightforward. A person can click through it, interact with it and react to what it does. That gives the engineer and stakeholder something more specific to discuss than a PowerPoint describing the proposed software.

Stecher encourages his engineers to get to a proof of concept or proof of value frequently. Spend the tokens, put the result in front of people and iterate while their reactions can still shape it. If the result serves no purpose, discard it. He judges the expense by the value of what the team produces or learns. Counting tokens alone misses the time and money the team can save by finding out early that an idea needs changing or dropping.

This speed depends on the scope of the commitment. The clickable version can exist before cybersecurity review and before the infrastructure work required to run a service at scale in a financial institution. The team can therefore settle a question about usefulness today while leaving a larger production investment for a separate decision.

10:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:24 · section reference included

Claude Code changed what Worland expected an agent to generalize

Norm’s founders discussed the bitter lesson in 2023, before building the workflow they later reconsidered. Knowing the principle did not tell them how much to spend on each component. By the time Worland faced that choice for Norm’s agent platform, the team had built a harness for running agents, a sandbox for execution and controls over what could leave it. Customers needed those capabilities, and a frontier lab or a third party might release something Norm would want to adopt. Worland came to see investment as a matter of degree, with some parts worth building heavily and others worth keeping easy to replace.

Claude Code prompted him to revise his judgment about generalization. The striking feature for him was the combination of actions available to the model. It could write code, execute that code and use a computer in a way that let it check the result. A developer no longer had to foresee the sequence of operations for every kind of question. Faced with a request outside the distribution of familiar examples, an assistant could spin up a computer and work on the request in the background.

Worland saw that execution-and-verification loop as an ingredient for a much wider range of knowledge work. A consumer could ask an unfamiliar question and let an assistant use a computer to work out the answer. Worland was seeing the same capability in enterprise tools. For Norm, that sharpened the investment question. Agents needed a usable operating environment, and as models could figure out more of the work themselves, Norm had reason to invest in the information it gave them and in measuring what they accomplished with it.

11:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:44 · section reference included

Norm Law helps score the work a lawyer needs done

Worland identifies data capture, distillation and making the result available to models as state as areas for sustained investment. Alongside those sit benchmarks and evaluations. Norm Law grew up with the technology company, giving the engineering team direct access to lawyers who can help define what success looks like. That relationship matters when the team has to decide whether a general improvement to the platform has actually improved legal work.

Consider the difference between two roadmap questions. One asks when Norm will deliver an agent that can do three specified things. The other asks how successfully the work gets done across the dimensions a lawyer cares about. For the latter, lawyers and engineers develop a scorecard that can apply to a human completing the task as well as to an agent. The team can follow the scores over time and look for improvements an attorney would recognize in the workflow. It can also change how the agent arrives at a result without treating a departure from the original sequence as a failure. That freedom mattered precisely because the old upfront structure had sometimes reduced performance.

13:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:45 · section reference included

Production needs traceable software and a human owner for sensitive changes

Stecher returns to what happens after a promising proof of value. The application has to run inside the infrastructure the customer actually has. Vendors selling AI tools can underestimate how much work falls to the financial institution on the other side of that deployment. Its people have to get the software through cybersecurity review and make it supportable. Stecher wants the software’s behavior to be traceable and to feed into the firm’s SRE platform so the people responsible for the live service can monitor it. Coding agents are getting better at parts of this work, he says, but the gains have been less evident here than in producing prototypes.

Worland describes the ownership rule Norm uses for its core primitives. People oversee the design and implementation, own the code and approve changes. Engineers and agents can build on top of those components. If something built above them needs to change a primitive itself, that change requires human review. The boundary gives developers room to use the platform while keeping a person accountable for changes to the mechanisms protecting sensitive data.

A good demo can make this tension especially clear. Worland describes the experience of seeing an exciting capability and, at the same time, recognizing how much information it would expose if offered with the controls currently available. For Norm’s customers, protecting that information takes precedence over delivering the capability quickly. Worland wants to invest heavily enough in the core to uphold that commitment and more conservatively in layers likely to be superseded. His image for the latter is a conduit through which improved model capabilities can reach the application. An old workflow that insists on directing every step can effectively dam them up.

14:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:43 · section reference included

Compare the next maintenance cycle with the cost of a replacement

Stecher says that ten years earlier, designing and writing an internal application consumed enough time, money and engineering attention to commit a company to it for a long time. Suggesting a rewrite to the CFO would have seemed absurd. Today he is comfortable building an internal application for a particular purpose and point in time, using it while it serves that purpose and letting it disappear much sooner.

The decision comes due when engineers consider bringing the current version forward again. How much does that round of maintenance cost? How much would a replacement cost to write now, and how much business value might a fresh application deliver? The work already invested in the first version tells the team little about today’s answers. Worland sees a timing problem even within a single year. An assumption about what was expensive in June can still be directing engineering investment in October, after the practical cost has changed considerably.

17:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:24 · section reference included

A lower cost of scaffolding brings experienced engineers back to the keyboard

For Stecher, there is a personal consequence to that change in cost. He spent ten years as a low-level engineer at IBM, using Notepad and vi to write instruction-level code for RISC machines. He still remembers how much he enjoyed it. As his responsibilities grew, getting far enough into an implementation to do useful work became harder. Even starting something that processed data from a stream could mean dealing with enough scaffolding to keep him from getting started at all.

He had retained the ability to architect the system. Coding agents gave him a way to put that knowledge into working software again. He sees the same possibility for engineers with ten, twenty or thirty years of experience. They understand the complexity of system design and can now express more of that understanding directly in code. After a discussion about which software to wait for and which software to discard, that ability to sit down and build again is what Stecher says makes him excited to go to work.

19:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:16 · section reference included

Transcript download
Full transcript

Loading transcript…