AI Engineer World's Fair 2026
The 100-Tool Agent Is a Trap: Scaling with Semantic Routers and JIT Context
Loading slides…
Read the talk
The 100-Tool Agent Is a Trap
Semantic routing and just-in-time context let an agent’s tool catalog grow without forcing the model to consider every schema on every request.
From a talk by Sohail Shaikh and Ankush Rastogi
Before you start: Familiarity with LLM tool calling, JSON schemas, embeddings, and basic nearest-neighbor retrieval will help you follow the implementation.
When every request carries every tool
Giving an agent every tool it might ever need looks harmless when the demo has ten tools. As the product grows, the same design can become slower, more expensive, and less accurate. That is the scaling problem Sohail Shaikh and Ankush Rastogi address: the catalog grows, but every request still carries all of it.
Both introduce themselves as working at Prosodica. Sohail, a data scientist focused on applied AI, NLP, conversational intelligence, and RAG, approaches the problem through model and routing behavior. Ankush, a senior data solutions engineer with more than a decade in AI, data engineering, and production systems, focuses on what survives real load, real users, and failures.
Consider an agent that can query a database, send email, check a calendar, look up an order, and call other APIs. The simplest implementation puts every function name, description, and JSON schema into every model request. The user may need only an order lookup, but the model receives the entire catalog.
This fat agent can appear workable at ten tools. As ten becomes thirty and then hundreds, similar functions compete for attention. The model may select the wrong function, confuse neighboring capabilities, invent tool names, or take longer to respond. An individual tool need not be broken: catalog-wide loading creates the problem. In Sohail’s example, 741 tools require approximately 127,000 tokens of descriptions and schemas before the user’s question is considered.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A larger choice set can make selection worse
Sohail reports the following tool-selection results for loading the full catalog. These are the speakers’ reported measurements; the available experimental details do not specify dataset versions, sample counts, scoring rules, or enough model and runtime conditions to reproduce the curves.
| Tools available | Full-catalog accuracy |
|---|---|
| 10 | Approximately 78% |
| About 100 | Approximately 40% |
| 741 | 13.6% |
At the largest size, that is roughly one correct selection in eight. The slide rounds the endpoint to 13%. Sohail reports that semantic routing stays above 83% tool-selection accuracy across the same catalog sizes.
The difference is the model’s immediate choice set: hundreds of schemas versus a small, relevant subset. Sohail proposes lost-in-the-middle behavior as one explanation. Models can use information at the beginning and end of long contexts more reliably than information buried in the middle. The original research studies question answering and key-value retrieval, so it supports the positional concern rather than proving the cause of these tool-selection results. A larger prompt can therefore cost more while making the decision harder.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paying to describe tools the request will never use
The same catalog creates recurring input cost. At the example workload of 100,000 requests per day, repeatedly sending approximately 127,000 tool tokens means billions of daily tokens just to describe capabilities. Sohail estimates that selecting three to five relevant schemas instead brings tool context to about 1,000 tokens, roughly a 99% reduction. These figures concern tool context, not the entire conversation or total application cost.
Longer prompts also require more processing before generation begins. Sohail reports that a 500-tool fat-agent path can exceed five seconds to first token. For an interactive product, that delay occurs before the user sees a response; routing aims to keep this prompt-processing burden bounded as the catalog grows.
The architecture changes where selection happens:
| Design | Before the model call | Model’s choice set |
|---|---|---|
| Full catalog | Load every schema | Every tool |
| Semantic routing | Inspect query; retrieve relevant tools | Usually three to five tools |
The full-catalog design also tends toward a monolith that is difficult to test, risky to update, and painful to debug. Routing separates retrieval from the model’s final tool choice and keeps the request context focused.
That separation is not automatically worth its overhead. Ankush’s practical heuristic is to load tools directly when there are fewer than twenty, and to consider routing more seriously beyond fifty tools in production. Those are starting points for an architecture decision, not hard limits on model capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve tools, then load their schemas
Semantic routing is RAG for tools. Instead of retrieving documents to help answer a question, retrieve tool descriptions to determine which capabilities the model should see.
- Give each tool a clear description: search flights, check calendar availability, or retrieve customer order status.
- When the catalog is created or updated, embed those descriptions and store the vectors in an index.
- At runtime, embed the user’s query with the same embedding model and search for the closest descriptions.
- Return the top
Ktools—often three or five—and inject only their schemas into the model call.
An existing RAG stack already supplies much of the required infrastructure: an embedding model, a vector index, and a retrieval path.
Semantic routing is the retrieval layer; just-in-time context injection is the context-management strategy. Full-catalog loading supplies everything before understanding the request. Just-in-time loading waits until the query is known and then supplies what that request needs. It applies the familiar principle behind lazy loading, just-in-time compilation, and on-demand resource loading to model context.
Ankush connects this principle to Anthropic’s Code execution with MCP. Its November 2025 illustration reduces token usage from 150,000 to 2,000, a 98.7% reduction, through on-demand interface discovery and code execution. That is a related context-management example, not a controlled benchmark of top-K semantic routing or a measured latency reduction. The shared principle is to discover and load tool interfaces when needed instead of placing the whole catalog in every prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the working set, not just the catalog
A fair comparison measures four outcomes: tool-selection accuracy, time to first token, input tokens per request, and estimated cost per 1,000 calls. Ankush describes using the Berkeley Function Calling Leaderboard, SkillsBench-style scenarios, and synthetic tool pools at catalog sizes of ten, fifty, one hundred, two hundred, and 741 tools. “SkillsBench-style” describes the scenarios; it does not identify a reproducible benchmark subset.
The stated comparison holds the queries, model, answer key, and tool catalog constant. Only the schemas visible to the model change: all tools in one mode, retrieved tools in the other. The retrieval sweep uses K = 3, 5, and 10. A smaller K reduces context and favors speed and cost; a larger K can recover edge cases by including tools that narrowly missed the smaller candidate set. Ankush recommends K = 5 as a starting point.
The paired charts revisit the accuracy decline and add first-token latency. The full-catalog curve slows as more schemas enter the prompt, while the routed curve stays approximately flat in the reported comparison. The latency chart is labeled GPT-4o, though that label alone does not specify the full experimental setup or whether routing overhead is included. The interpretation is architectural: the catalog can grow while the model’s working set stays small.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the retrieved schemas the model’s actual tool list
The implementation has three stages. Offline, store each tool’s name, description, and schema, then embed the description and associate its vector with the tool identity. ChromaDB, Pinecone, and Qdrant are the vector-store examples in the talk. On each request, embed the query and retrieve the nearest tool descriptions. Finally, fetch the selected schemas, pass them into the model call, and log the resulting tool choice.
Logging closes the improvement loop: a missed tool may call for a better description or a different K. Ankush characterizes the additional retrieval work as one embedding call and one vector search, exchanged for a much smaller model prompt.
The following Python implementation makes the selection boundary explicit. It uses cosine similarity over an in-memory index; embed supplies the application’s embedding function, and call_model supplies its model adapter. Each catalog entry’s schema is the complete tool definition expected by that adapter.
python
from math import sqrt
def cosine(a, b):
if len(a) != len(b):
raise ValueError("Embedding dimensions differ")
denominator = sqrt(sum(x * x for x in a)) * sqrt(
sum(y * y for y in b)
)
if denominator == 0:
raise ValueError("Embedding must be nonzero")
return sum(x * y for x, y in zip(a, b)) / denominator
def build_index(catalog, embed):
return [
(tool, embed(tool["description"]))
for tool in catalog
]
def route_and_call(query, index, embed, call_model, logger, k=5):
if k < 1:
raise ValueError("k must be positive")
query_vector = embed(query)
ranked = sorted(
index,
key=lambda item: cosine(query_vector, item[1]),
reverse=True,
)
selected = [tool for tool, _ in ranked[:k]]
logger.info(
"routed_tools=%s",
[tool["name"] for tool in selected],
)
return call_model(
messages=[{"role": "user", "content": query}],
tools=[tool["schema"] for tool in selected],
)
The decisive line is tools=[tool["schema"] for tool in selected]. Retrieving candidates accomplishes little if the subsequent model call still receives the full catalog. This call asks the model to choose among the retrieved tools; it does not execute a tool.
The same structure works with a vector database instead of the in-memory search. Sohail suggests starting with local Qdrant or another local store and moving to a managed service if needed. The architecture does not depend on one embedding model or database, and an existing RAG implementation can often supply the retrieval machinery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A flight request and a weather request get different tools
Ankush illustrates the mechanism with a 200-tool agent. A user asks, “Find me a flight to New York next Wednesday.” The router returns search flights, book flight, and calendar check. Those schemas become the model’s candidate set. Hotels, weather, email, SQL, and unrelated workflow tools would otherwise compete for attention despite having little to do with the request. Retrieval supplies candidates; the example does not establish that a flight was booked.
The next query is, “What is the weather in Paris right now?” This time the router returns get weather and get forecast. Flight and hotel tools are absent from the model call, not merely placed lower in a long list. Routing removes irrelevant choices as well as adding relevant ones. The catalog remains available to the application, but this request exposes only the selected schemas to the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect retrieval to evaluation and maintenance
The production checklist extends the retrieval function into an operational loop:
- Centralize the catalog. Store name, description, schema, owner, and version for each tool.
- Build the index. Embed descriptions and store their vectors.
- Wire the router. Embed each query, search for top-
Kmatches, and fetch their schemas. - Connect the agent loop. The model’s tool list must come from the router, not a hard-coded full catalog.
- Evaluate the candidate count. Test
K = 3,5, and10, then choose the smallest value meeting the accuracy target. - Monitor and refresh. Log retrieved tools, final tool calls, failures, and fallback usage. Re-embed tools when descriptions or schemas change.
Ankush presents this as a focused integration effort for most teams, rather than a six-month platform rewrite. The checklist describes the work required; it is not a demonstrated deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the existing retrieval ecosystem
The speakers return to Anthropic’s on-demand loading example and describe similar complaints in agent and SDK projects: sending every definition increases token usage, latency, and tool confusion. They also point to MCP-Zero, which explores discovery across thousands of tools and many servers. It is a research and retrieval-experiment reference, not evidence of a production deployment. Forum reports of confusion with only a few tools further qualify any fixed tool-count threshold. When adding capabilities makes an agent worse, investigate the selection architecture rather than assuming the prompt alone is at fault.
For experimentation, Sohail recommends Aurelio Labs’ Semantic Router, along with ToolBench and the Berkeley Function Calling Leaderboard as evaluation starting points. The Anthropic MCP writeup supplies practical context-management guidance. These are pieces to build on; adopting routing does not require inventing a new infrastructure ecosystem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover from misses without rebuilding the trap
Restricting the candidate set introduces a new failure mode: the router may omit the tool needed to finish the task. If the model cannot complete the request, Sohail suggests widening K, running a second retrieval pass, or routing to a broader tool group. Recovery should expand the available choices when needed.
Retrieval quality also depends on the catalog itself.
- Weak descriptions: Vague text produces weak matches. Describe intent, action, and key entities in the language users actually use.
- Rare tools: A specialized capability may never rank highly unless its description includes the relevant language. Monitor misses and rewrite the descriptions that consistently fail retrieval.
A router cannot compensate indefinitely for a catalog that does not explain what its tools do.
For ten or fifteen tools, static loading may still be the simplest adequate design. Routing becomes useful when prompt size, latency, or tool confusion creates a real production problem. The aim is to introduce a retrieval boundary where it helps, not to make every small agent more elaborate.
The operating discipline is straightforward: index descriptions, retrieve at runtime, and inject only the needed schemas. Start with K = 5, log decisions, evaluate against a real test set, and improve descriptions when the wrong tools are selected. Tool tokens consume both money and response time; the reason to manage them is to stop forcing the model to reason over irrelevant capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic explains on-demand tool discovery, code-based tool composition and keeping intermediate data outside model context.
Python library with examples for embedding-based routing, local execution and vector-store integrations.
Active tool-discovery research code and a dataset covering 2,797 tools from 308 MCP servers.
Function-calling evaluations with versioned methodology, benchmark data and implementation links.
An open platform for training and evaluating language models that use tools.
Project overview of a benchmark measuring how reusable skills affect agent performance on professional tasks.
Further reading
- Lost in the MiddlePaper
Research on how the position of relevant information affects question answering and retrieval in long contexts.
- RAG-MCPPaper
A retrieval-first MCP selection study reporting accuracy and prompt-token measurements on web-search tasks.
Full transcript
Loading transcript…