AI Engineer World's Fair 2025
From Copilot to Colleague: Building Trustworthy Productivity Agents for High-Stakes Work
Loading slides…
Read the talk
From Copilot to Colleague: Building Trustworthy Productivity Agents for High-Stakes Work
Productive agents need more than good answers: they need domain tools, inspectable sources, evaluation that accounts for human variability, and agency matched to the risk of the work.
From a talk by Joel Hron
When helpful answers become consequential work
What changes when an assistant stops helping a professional make a decision and starts making judgments on that professional’s behalf? Early assistants were built around helpfulness, with accuracy and citations supporting that goal. The next expectation is more demanding: produce the work itself.
In law, tax, global trade, risk and fraud investigations, that shift changes the consequences of an error. A useful suggestion and a completed piece of professional work carry different expectations. Productivity requires trust in the output and in the process that produced it. This is the problem Joel Hron brings to agent design at Thomson Reuters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The assets behind a professional agent
Thomson Reuters approaches this problem with longstanding relationships in the professions it serves. Hron reports that its customers include 97% of the top 100 US law firms, 99% of the Fortune 100, and the top 100 US CPA firms. Those relationships create expectations about how professional software should behave, as well as access to the expertise needed to evaluate it.
Hron reports 4,500 domain experts and more than 1.5 terabytes of proprietary content underpinning the company’s software. He also tentatively describes Thomson Reuters as the world’s largest employer of lawyers. The relevant asset is the combination: a collection of professional source material and people who can judge whether a system uses it correctly.
Hron describes more than $3 billion in acquisitions over the preceding couple of years, an applied research lab with slightly more than 200 scientists and engineers, and annual capital spending above $200 million on AI product development. These are his descriptions of company scale and spending, rather than measures of agent performance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agency has four adjustable dimensions
Hron introduces the shift through his paraphrase of Y Combinator’s summer 2025 Requests for Startups: build law firms of agents rather than agentic tools for law firms. The distinction is between supporting someone who produces a result and taking responsibility for producing that result.
Agency is a spectrum, not a binary property. Exploratory work may benefit from giving a system more freedom. A workflow that demands precision and an expected sequence of operations may require less. The useful design question is how much agency each part of the experience should have, given the task and the user’s tolerance for risk.
Hron separates that decision into four dimensions:
- Autonomy: Move from a discrete instruction, such as summarizing a document, to a workflow in which the assistant plans, executes and replans in response to what it observes.
- Context: Move from the model’s parametric knowledge to retrieval-augmented generation, then to multiple sources. With both a controlled collection and the web available, the system must judge which source is appropriate in context. At the speculative upper end, it might modify the sources or their schemas to support future questions.
- Memory: Move from retrieving context for one request to sharing information across execution steps and retaining it across user sessions.
- Coordination: Move from an assistant completing a task on its own to delegating to tools and collaborating with other agents.
These dimensions can be adjusted separately. Giving an agent persistent memory does not, by itself, require giving it unrestricted control over the workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation has variability on both sides
Trust often comes from predictability: provide a familiar input and receive an expected outcome. Users bring that expectation to AI systems, and so do the internal subject-matter experts who evaluate them. Probabilistic behavior makes that expectation difficult to meet, but the evaluator is not a perfectly stable reference either.
Hron reports ten-plus-percent swings in assessed accuracy when the same trained legal experts reevaluate the same question-response data a week later. This is an internal observation about variation in human judgments, not a measured model regression; he does not specify whether the change is relative or in percentage points. If the reference judgment moves, an apparent improvement or decline becomes harder to interpret. Repeated evaluation is also expensive when weekly development cycles require time from lawyers or tax professionals.
Greater agency adds another evaluation problem: the final answer is only the end of a trajectory. Source attribution becomes harder to preserve, and a poor result may reflect drift earlier in the workflow. Finding where the agent went off course—and why—requires inspecting more than the output. Even the guardrails used to constrain the system require deep domain knowledge.
Thomson Reuters uses detailed rubrics alongside preference judgments, with different roles for each:
| Evaluation signal | Role |
|---|---|
| Detailed rubrics | Improve particular components |
| Preference judgments | Assess overall progress |
The rubrics make local improvement tractable. Preference provides a broader signal about whether the resulting experience is getting better or worse.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expose established domain logic as tools
Existing applications can constrain a new product, but they also contain carefully developed domain logic that users already expect. Early assistant development often started separately, leaving much of that logic and infrastructure behind. Agents create another option: decompose the application into capabilities the agent can call as tools.
A specialized application then becomes part of the agent’s working environment. Its domain logic need not be recreated in model-generated reasoning. The slide connects domain logic, user context and human expertise with agent tools, integrated experiences and alignment of AI judgments. What once looked like legacy baggage can become a foundation for the new system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build enough to observe the whole workflow
An isolated component can be the wrong unit of product learning. Hron describes teams overemphasizing the minimal part of an MVP and spending time optimizing what appeared to be the smallest valuable piece of the system. They could not yet see whether that piece was the actual obstacle to completing the work.
Running the whole workflow changed that picture. Some weaknesses needed engineering attention; others could be recovered from through the agent’s subsequent actions. Observe complete execution before deciding which component deserves optimization. The team’s shift was to build the whole workflow first and learn from its behavior, rather than treating improvement of an isolated component as sufficient evidence of product progress.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tax preparation: extract, calculate, validate, revisit
The tax demonstration starts with explicitly fake data and a familiar professional task: receive source documents, extract their contents, map the information into tax software and generate a return. Inputs include forms such as a W-2 or 1099. The demonstrated outcome is return preparation; it does not establish autonomous filing.
The workflow divides interpretation from specialized calculation:
- Extract document data. AI reads the values from the source forms.
- Map values to tax-engine fields. The system considers tax rules and conditions to determine whether a value applies and which line or field should receive it.
- Generate the return through the tax engine. The existing engine provides the calculation capability used by the agent.
- Inspect validation errors. A validation engine built into the tax engine checks the work and exposes errors.
- Return to the source documents. The agent looks for additional information when needed and resolves issues to finish the workflow.
The validation result is therefore an input to further work. The agent can use a domain system’s feedback to decide what information it needs next, rather than stopping after its first extraction and mapping pass.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Legal research: from tool calls to cited synthesis
The second demonstration follows legal research for litigation preparation. It applies deep research to the company’s proprietary content, using capabilities from an existing litigation-research product: search for documents, fetch them, compare citations across cases and validate citations within cases. The sources span case law, statutes, regulations, legal know-how, articles, blogs and licensed material.
The visible Research Trajectory Explorer exposes the work beneath the product interface. An expanded fetch_document entry shows its input and returned court-opinion text. This is an intermediate retrieval step: it makes part of the research path inspectable, rather than showing only a polished answer.
As research proceeds, the model writes notes about what it is finding and learning. At the end, it reconciles those notes into a final report. The report links to actual cases and statutes within the product, and visible flags indicate risks associated with the cited material. The result connects synthesis back to legal authorities that a professional can inspect.
Together, the tax and legal demonstrations show why application decomposition and complete workflows matter. In tax, the agent uses calculation and validation capabilities; in legal research, it uses retrieval and citation capabilities. In both cases, established software participates in producing the result rather than merely supplying background text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build differentiation around what the organization knows
The design strategy begins with the whole problem and adjusts agency to the risk of the use case and the user’s tolerance. Existing systems supply reusable capabilities, while subject-matter experts remain central to evaluating the work. These decisions connect product scope, architecture and human judgment: none can be settled solely by choosing a model.
Hron’s closing question is organizational as much as technical: which assets do you have that others do not? For Thomson Reuters, he points back to its domain experts and proprietary content. The opportunity is to turn those assets into distinctive product behavior—what the agent can do, which sources it can consult, and how its work can be judged.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Security posture remains a separate deployment question
The audience question brings the discussion to adoption in defense or finance. Invoking CISA and government requirements, the questioner asks about the architecture’s cybersecurity posture, including LLM firewalls, guardrails, automated vulnerability scanning and SCM security posture management. These are requests for concrete controls, beyond the correctness and usefulness of the agent’s output.
Hron’s response stays at the level of standards and supporting documentation. He points to online technical documentation and describes a focus on standards such as FedRAMP when working with government; the response does not establish a particular product’s authorization. He also says several products comply with a recent ISO standard, without naming the standard or products, and emphasizes adapting as requirements evolve. The exchange leaves the deployment topology and implementation of the requested security controls unspecified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A conversation with Joel Hron and David Wong about professional AI, domain expertise and customer needs.
The March 2025 announcement explains the AI management-system certification and names the covered Westlaw solutions.
Updates since the talk
A July 2025 introduction to tax agents that combine document processing, tax software and professional review.
A later account of CoCounsel Legal’s architecture, source access and evolution beyond predefined workflows.
Full transcript
Loading transcript…