▶ Watch ↗20:45
Conference affiliation: Norm AI
Scott Worland has led engineering as chief technology officer at Norm AI, building software for legal and regulatory work. His team’s experience exposed a useful distinction: stronger models could make carefully written workflow code obsolete, while the expert judgments collected to test that code remained valuable. Scott used those evaluations to guide a platform redesign around agents that could choose more of their own steps.
Norm develops technology alongside professional services businesses, including Norm Law and Norm Compliance. Professionals help shape the shared platform through their work. For Scott’s engineering team, that relationship provides both a way to define successful legal work and demanding constraints on how agents handle sensitive institutional information.
In early 2024, Norm’s engineers prescribed much of the route from source material to a finished work product. Their software selected the context models received, sequenced model calls, checked intermediate results for consistency and combined the outputs. Scott compared the architecture to a large MapReduce job. That structure helped the team produce accurate, consistent results for regulatory work.
To catch regressions, the team accumulated tens of thousands of expert-labeled examples. Those examples later let it test a different architecture on familiar cases: give a model tools, a goal and a way to check its work, then let it choose how to proceed. Scott described finding that much of the team’s elaborate sequencing code had become obsolete and could get in the way of better performance. The evaluations outlasted parts of the implementation they had been built to test.
Scott brought Norm’s engineering leadership to upstate New York to specify the platform they would build if they were starting again. They set two requirements: enable clients to delegate work to agents, and make the codebase suitable for AI coding tools to work alongside Norm’s developers. In his conversation with Blackstone CTO John Stecher, Scott described the redesign as work the team was continuing to execute.
Scott had discussed the risk of handcrafted systems being overtaken by model improvements with Norm’s early team in 2023. Knowing that risk did not let them avoid it: they still needed working software with the models available at the time. His subsequent position was to decide where to invest heavily and where to build conservatively, expecting some components to be replaced.
Coding assistants changed his assessment of how broadly agents could generalize. He had been skeptical that agents could extend far beyond predefined tasks. Watching systems write code, execute it and check the result gave him a concrete mechanism for handling unfamiliar requests: an agent could create a procedure for a new problem and run it on a computer. That shift strengthened his emphasis on making useful data available to models and measuring their work, while leaving room to adopt capabilities from model providers and other developers.
Scott’s approach gives different parts of the platform different treatment:
▶ Watch ↗20:45
Loading more talks and highlights…