AI Engineer Summit 2025
Challenges to Scaling Agents for Generative AI Products
Loading slides…
Read the talk
Challenges to Scaling Agents for Generative AI Products
Bloomberg’s research workflows show why scaling agents requires mandatory checks, resilient downstream consumers, and team boundaries that evolve with the product.
From a talk by Anju Kambadur
Before you start: Familiarity with APIs, machine-learning models, and basic data retrieval will help; no finance background is required.
From building a model to building on models
Should a company with years of AI investment build its own large language model or build products on models available elsewhere? Bloomberg initially chose to build. The decision came in late 2021; development occupied 2022, and the BloombergGPT paper followed in 2023. The effort taught the team how to organize datasets, evaluate models, and improve performance in particular areas.
ChatGPT’s arrival in late 2022, alongside rapid progress in open weights and open source, changed the landscape. Bloomberg continued related model work, but its product strategy shifted toward building on what was available. With many different use cases to serve, the question became how to turn those models into useful products.
That work crossed organizational boundaries. Anju Kambadur described an AI group reporting to Bloomberg’s global head of engineering and working closely with the data organization, product teams, and CTO function. At the time of the talk, Kambadur reported about 400 people across 50 teams in London, New York, Princeton, and Toronto.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Shipping with bounded autonomy
Bloomberg had spent roughly 12–16 months seriously building generative AI products, progressing from tools toward more agentic systems. The constraints here belong to that shipping experience: what must be solved to put a product into users’ hands with the technology available at the time. Kambadur remained optimistic that these problems would become easier; optimism did not remove the need to handle them before release.
Even the vocabulary needed agreement. Internally, people used tool and agent differently. Cognitive Architectures for Language Agents supplied a common reference. In the vocabulary used here, tools sit toward one end of a spectrum; agents sit toward the more autonomous end, with memory and the capacity to evolve. This is a working distinction for designing the systems, not a universal classification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The research workflow behind the agent
Finance contains many different professions, with overlapping but distinct information needs. Bloomberg both generates and accumulates the information those users work with: news, research, documents, slides, reference data, and market data, as well as access to websites. Kambadur described daily intake of 400 billion structured-data ticks, more than a billion unstructured messages, and millions of documents including news. This is information-ingestion scale, not agent throughput. The daily flow sits alongside more than 40 years of information history.
A research analyst makes the requirements concrete. An analyst might specialize in AI, semiconductors, technology, or electric vehicles, but staying informed spans several kinds of work:
- Unstructured information: Search, discover, and summarize relevant material.
- Structured information: Retrieve data and perform analytics.
- Communication: Gather information from colleagues and distribute findings.
- Model building: Normalize data, program, and construct models.
An assistant serving this workflow has to connect these activities without losing the meaning of the information passed between them.
The product requirements remain the same whether AI is involved or not: precision, comprehensiveness, speed, throughput, and availability. Contributor and client data must be protected, and the product must provide transparency. An agent’s autonomy does not relax the obligations of the product containing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A summary becomes a publishing system
One early application was earnings-call summarization, with work beginning in 2023. Public companies hold quarterly calls about their performance and outlook, typically combining executive presentations with analyst Q&A. During earnings season, many calls happen on the same day. An analyst cannot investigate every discussion with equal attention.
AI-generated transcripts provided the starting material. Bloomberg already knew many of the questions that mattered for a company and its sector, so the opportunity was to answer those questions from the call. An analyst could use the resulting summary to decide whether a deeper investigation was worthwhile. The apparently simple product still had to satisfy the same financial-information requirements as everything else Bloomberg shipped.
Out-of-the-box precision, accuracy, and factuality were not sufficient. The engineering work therefore included remediation workflows and circuit breakers. These summaries were published for everyone to read, rather than generated inside one person’s private chat with a transcript. An error in a shared publication has a wider impact. Continuous monitoring and remediation became part of improving the summaries’ accuracy, not an operational afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Checks the agent cannot skip
Substantial CI/CD supported this production work. The broader architecture was deliberately semi-agentic: some components operated autonomously, while others did not. Kambadur attributed that boundary to insufficient trust, at the time, in making the entire system autonomous.
Bloomberg does not offer financial advice, so a request asking whether someone should invest needs to be detected. Factuality also requires checks. These guardrails are coded as required steps; an agent does not get to decide whether to invoke them. The architecture places guardrails across the research workflow, establishing the setting for two scaling problems: how components can improve independently, and how teams should own those components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What changes when an API contains a model
The first scaling requirement is rapid improvement. An initial agent will need repeated revisions, and its callers need to survive those revisions. Numerical software offers a useful comparison. Kambadur recalled using generalized matrix multiplication as a graduate student: detailed documentation described inputs, errors, and execution expectations. A clear, dependable contract makes it easier to build robust software above it. The analogy is about predictable behavior, not a universal guarantee of elapsed runtime.
Model-backed APIs weaken that certainty. Even when developers characterize input and output distributions, a caller cannot know in advance whether a particular result will work for its application. Composing LLMs into agents creates more opportunities for one component’s error to affect another. Kambadur described these errors as compounding into fragile behavior; the point is a dependency problem, not a quantified error law.
Bloomberg’s 2009 news-sentiment product shows how bounded the earlier problem could be. Its task was to determine whether a piece of news was beneficial to a given company. The team knew which news wires it monitored, the language of the material, and the editorial conventions shaping those inputs. The sentiment output was approximately bounded from −1 to +1. Bloomberg had built the training data itself, and test sets held out in time and space helped assess deployment risk and support monitoring.
Even with those constraints, releases required communication outside the API. Downstream users received notice that a model version was changing so they could test applications that consumed the sentiment signal. That arrangement becomes difficult when agents should improve daily and downstream systems are also changing independently. A release cycle based entirely on batch regression tests and coordination with every consumer becomes an obstacle to iteration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five quarters of CPI, returned as months
Consider the request US CPI for the last five quarters, with Q denoting a quarter. The research agent interprets the question, determines which domain should handle it, and calls a data-retrieval tool with an NLP front end. This looks like a straightforward composition: understand the request, route it, and fetch the data.
In the example, one missed character caused the tool to retrieve monthly data instead of quarterly data. A capable analyst looking at the table could catch the mismatch. A downstream workflow that hid the table and presented only a final answer would make the same error much harder to detect. The failure is semantic: the tool returned data, but not data at the frequency the user requested.
The illustrated recovery path adds query rewriting, explicitly requesting US CPI YoY for the last five quarters, and shows a subsequent table with quarter-end dates. This gives the guardrail a concrete responsibility: check whether the returned data actually matches the request, instead of treating a successful retrieval as sufficient. The broader design principle is to expect upstream systems to be fragile and evolving, then perform safety checks in the consuming workflow.
Better on average does not mean better for a particular downstream consumer. Even within one organization, a new data-and-analytics API can improve overall while changing behavior that one caller depends on. Consumer-side checks make independent evolution more practical: each agent can improve without requiring every downstream caller to understand every change and sign off before promotion to beta or production. Resilience is therefore part of release speed, not merely a cost imposed on it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let product understanding shape team boundaries
The second scaling problem is organizational. Traditional machine-learning software has familiar component boundaries, and those boundaries tend to be reflected in team structure. Kambadur invoked reverse Conway’s law: organizational design and software design influence one another. Moving to a different technical stack and different products requires revisiting both.
How many agents should exist? What should each do? Should their responsibilities overlap? Keeping the current software stack or organizational chart can seem like the easiest starting point, but it answers these questions before the new product is understood. Bloomberg’s experience with reorganizations suggested that early work benefits from collapsing boundaries: give a team enough ownership across the stack to discover what needs building. Fast iteration requires easy sharing of code, data, and models.
As the product becomes better understood, the balance changes. Teams learn what an agent is useful for, where it works well, and where it does not. With more agents in use, shared capabilities become easier to identify and justify.
| Stage | Team emphasis | Engineering priority |
|---|---|---|
| Early discovery | Broad ownership of one product | Fast iteration and sharing |
| Understood agent portfolio | Selected shared capabilities | Performance, cost, testability, transparency |
The move toward horizontal capabilities follows accumulated product knowledge. It is not a requirement to begin with a fully decomposed architecture.
Guardrails are a concrete shared capability at Bloomberg. Having every team independently work out how to detect thinly disguised financial-advice requests would duplicate a common responsibility. Centralizing that capability makes sense, while the timing of other shared services depends on the organization. Breaking monolithic agents into smaller pieces and changing team ownership should happen together, when the boundaries are understood well enough to help.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The resulting research-agent architecture
The research-agent architecture shown at the end gives those boundaries specific responsibilities. One agent takes user and session context, understands the question, and determines what information is needed to answer it. That responsibility is separately factored and reflected in the organization. Answer generation is another distinct responsibility, with rigorous requirements for what constitutes a well-formed answer.
Mandatory guardrails run at multiple points in this workflow. Their invocation remains outside the agent’s discretion, preserving the semi-agentic design even as individual components gain autonomy. Beneath those components sits years of data-processing infrastructure, including the evolution from sparse indices toward dense and hybrid indices. The new agent boundaries build on that foundation while making room for components—and the teams responsible for them—to improve independently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Bloomberg's original finance language-model paper, covering training data, model development, and evaluation.
The CoALA framework organizes language agents through memory, actions, and decision-making.
Further reading
The original product announcement explains analyst-guided summaries and links back to supporting transcript passages.
A concrete matrix-multiplication contract specifying dimensions, transpose options, scalar inputs, and argument validation.
Full transcript
Loading transcript…