← All speakers

Scott Worland

Conference affiliation: Norm AI

Scott Worland has led engineering as chief technology officer at Norm AI, building software for legal and regulatory work. His team’s experience exposed a useful distinction: stronger models could make carefully written workflow code obsolete, while the expert judgments collected to test that code remained valuable. Scott used those evaluations to guide a platform redesign around agents that could choose more of their own steps.

On this page

Norm develops technology alongside professional services businesses, including Norm Law and Norm Compliance. Professionals help shape the shared platform through their work. For Scott’s engineering team, that relationship provides both a way to define successful legal work and demanding constraints on how agents handle sensitive institutional information.

When the workflow became an obstacle

In early 2024, Norm’s engineers prescribed much of the route from source material to a finished work product. Their software selected the context models received, sequenced model calls, checked intermediate results for consistency and combined the outputs. Scott compared the architecture to a large MapReduce job. That structure helped the team produce accurate, consistent results for regulatory work.

To catch regressions, the team accumulated tens of thousands of expert-labeled examples. Those examples later let it test a different architecture on familiar cases: give a model tools, a goal and a way to check its work, then let it choose how to proceed. Scott described finding that much of the team’s elaborate sequencing code had become obsolete and could get in the way of better performance. The evaluations outlasted parts of the implementation they had been built to test.

Scott brought Norm’s engineering leadership to upstate New York to specify the platform they would build if they were starting again. They set two requirements: enable clients to delegate work to agents, and make the codebase suitable for AI coding tools to work alongside Norm’s developers. In his conversation with Blackstone CTO John Stecher, Scott described the redesign as work the team was continuing to execute.

What changed his view of agents

Scott had discussed the risk of handcrafted systems being overtaken by model improvements with Norm’s early team in 2023. Knowing that risk did not let them avoid it: they still needed working software with the models available at the time. His subsequent position was to decide where to invest heavily and where to build conservatively, expecting some components to be replaced.

Coding assistants changed his assessment of how broadly agents could generalize. He had been skeptical that agents could extend far beyond predefined tasks. Watching systems write code, execute it and check the result gave him a concrete mechanism for handling unfamiliar requests: an agent could create a procedure for a new problem and run it on a computer. That shift strengthened his emphasis on making useful data available to models and measuring their work, while leaving room to adopt capabilities from model providers and other developers.

Evaluations and protected foundations

Scott’s approach gives different parts of the platform different treatment:

  • Measure successful work with legal experts. Norm’s engineers work directly with attorneys to build benchmarks and evaluations. Scott favors scorecards that assess task completion across several dimensions, including standards that could also apply to a human doing the work. General improvements to the system should raise those scores, rather than progress depending entirely on a roadmap of separately programmed agent features.
  • Require human oversight of core infrastructure. Norm retains human oversight of the design, implementation, ownership and approval of changes to its core primitives. Software built above those foundations cannot change them without human review. Scott places data security ahead of delivery speed: an impressive demo may depend on access or capabilities the platform cannot yet safely provide.
  • Keep model-dependent machinery replaceable. Norm has invested in agent infrastructure such as harnesses, sandboxes and controls on outbound access. Scott also wants the platform to absorb improvements from frontier labs and third parties. The earlier workflow redesign made the tradeoff concrete: structure that helps today must remain open to reconsideration when evaluations show a better approach.

1 conference talk

Loading more talks and highlights…

References