AI Engineer World's Fair 2026
Semantic Blindness: 500,000 Sensors Confused an LLM - Raahul Singh & Vanč Levstik, Phaidra
Loading slides…
Read the talk
Semantic Blindness: 500,000 Sensors Confused an LLM
Phaidra turns ambiguous infrastructure questions into structured retrieval plans, using compact hierarchy summaries and deterministic set operations to keep model work bounded.
From a talk by Raahul Singh and Vanč Levstik
Before you start: Familiarity with LLM context windows, embeddings, and basic set operations will help.
When equipment names stop fitting in context
Phaidra gave an LLM 500,000 sensor names, and it became confused. The task sounded straightforward: help a data-center operator identify a chiller running hot, analyze temperature distributions across data halls, or investigate GPU problems. But answering those questions starts with resolving exactly which equipment the user means. A question about chiller six targets one item; a question about GPUs in a data hall targets an entire group.
Equipment names do not follow a common industry convention. One customer may encode the data hall, rack, and GPU in a readable name; another may use an opaque identifier whose meaning depends on local knowledge. At demo scale, an LLM can inspect the complete list and infer a match. That approach makes the inventory itself part of every retrieval problem.
In Singh’s one-gigawatt example, a factory has more than 400,000 GPUs, plus power meters, chillers, and other supporting equipment. Passing all those names into a finite context window quickly becomes impractical. The production requirement also changes: finding a plausible match in one demonstration is insufficient when an omitted device can silently invalidate an operational answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Similarity does not establish identity
Embedding the equipment names for retrieval appears to remove the context-window bottleneck. But these identifiers carry meaning differently from ordinary prose. Two long names can differ by a single character that changes the physical device: chiller six is not chiller seven. In Phaidra’s experiments, semantic similarity did not preserve those distinctions reliably enough for accurate retrieval. A nearby vector is not necessarily the requested equipment, and finding several nearby vectors does not establish that every required device was found.
Direct enumeration created another failure mode. Consider a request to list roughly a hundred GPUs in aisle seven: the answer contains many almost identical token sequences. Singh reports that the model could stop before completing the list, attributing the behavior to repetition penalties or guardrails interpreting the output as a loop. That explanation is specific to the reported behavior, not a universal rule that repetitive output terminates every LLM. For this system, neither embedding-based name retrieval nor asking the model to produce the full inventory met the production requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Sharding the inventory preserves the wrong task
The next attempt divided the inventory into shards and searched them through parallel LLM calls. This reduced the amount of equipment each call had to inspect, but it retained the underlying task: ask a generative model to identify and enumerate exact members of a large collection. Phaidra observed both phantom equipment that did not exist and silent omissions of equipment that did. Missing a device can hide a developing operational problem, allow it to cascade, and erode the operator’s trust.
Model work needed to grow with the infrastructure’s structure, not its instance count. A data center contains data halls, aisles, rows, racks, and GPUs. Cooling infrastructure has its own groupings, including chiller rooms, pumps, and cooling towers. These arrangements resemble trees whose width grows much faster than their depth. Adding more GPUs usually adds instances of a known equipment type, not a new kind of relationship the model must learn. That difference creates an opportunity for sublinear growth in model work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Linearize the structure instead of listing the instances
The first architectural change is a linearizer: a compact textual representation of the systems graph that explains where equipment lives and how to reach it. Its purpose is to let the model map a vague request onto an equipment type or group without seeing every equipment node. Singh describes a one-gigawatt factory with more than a million such nodes. The useful context is the set of structural routes through that graph, rather than a separate description of every instance.
The illustrated routes to GPUs, chillers, and switches each traverse four layers. Expanding the inventory along those existing routes need not expand the description proportionally. Phaidra reports roughly equal summary sizes for systems with 64 and 460,000 GPUs. The summary gives the model enough information to understand the plant’s arrangement; the full inventory remains available to the retrieval backend.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask for a plan, not a list
The second change separates interpreting a request from executing a search. For a request to find all GPUs running hot in data hall eleven, the model does not need to read GPU names and decide which belong in the answer. It needs to specify how to construct the answer: select GPUs, restrict the scope to the data hall eleven subtree, and apply the relevant hot-running condition.
A compact representation of that plan is:
json
{
"equipmentType": "GPU",
"scope": {
"kind": "subtree",
"location": "data hall 11"
},
"filter": "running_hot"
}
Here, running_hot names a condition for the backend to resolve, not a temperature threshold invented by the model. Its meaning depends on the equipment and operational context. The structured output describes a proposed selection; it does not itself establish which GPUs satisfy it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resolve the plan with indexed sets
The third change moves retrieval into a backend with pre-indexed equipment subsets. These subsets capture locations and relationships between equipment: all GPUs in a data hall, equipment within a rack, or another relevant subtree. For the running example, the backend retrieves the GPU subset for data hall eleven, queries for GPUs that satisfy the hot-running condition, and intersects the two sets.
The intersection itself needs no model judgment. With equipment identifiers represented as strings, the core operation can be written directly:
typescript
function intersectEquipment(
scopedGpuIds: ReadonlySet<string>,
hotGpuIds: ReadonlySet<string>,
): Set<string> {
const result = new Set<string>();
for (const id of scopedGpuIds) {
if (hotGpuIds.has(id)) result.add(id);
}
return result;
}
const dataHall11GpuIds = new Set(["gpu-11-a", "gpu-11-b"]);
const hotGpuIds = new Set(["gpu-11-b", "gpu-12-a"]);
const result = intersectEquipment(dataHall11GpuIds, hotGpuIds);
// result contains only "gpu-11-b".
In this small example, gpu-11-a belongs to the requested hall but is not hot; gpu-12-a is hot but falls outside the requested scope. Exact membership testing excludes both. Singh describes set operations as providing perfect recall and accuracy. The guarantee applies to executing the operation over its supplied sets: correct intent interpretation, accurate indexes, and an appropriate definition of running hot remain separate requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn vague references into executable patterns
Some requests are too vague to resolve through an obvious equipment type and location alone. The fourth change gives the model a narrower search responsibility: infer a pattern from the naming convention or the data, then hand that pattern to the backend for execution. The model sees enough naming structure to infer what to look for without receiving the complete name list. The architecture does not depend on a particular pattern language.
This keeps model token exposure relatively stable as equipment instances multiply. Reading every name would make model work grow with inventory size; describing a pattern can remain compact. The distinction concerns the model’s workload and reported operating cost. Index construction and backend pattern execution still have to handle the underlying equipment data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bound the query pipeline
The pieces form a short retrieval pipeline:
- Interpret the user query. A planner LLM determines the requested item or group and produces a search plan or pattern.
- Resolve the selection. Deterministic code uses indexes, pattern execution, and set operations to identify the matching equipment.
- Return the result set. The resolved collection becomes the concrete equipment selection for the request.
The boundary between the planner and resolver is the central design decision: the model specifies the search, while code determines membership.
Phaidra describes this as a two- or three-step process instead of an agentic loop that repeatedly searches and reasons. Bounded orchestration complements the compact context: neither the number of equipment names nor an open-ended sequence of model calls needs to determine the model’s total workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test correctness and token cost separately
An elegant retrieval architecture still has to survive customer data, real load, and edge cases. Before exposing the new system to customers, Levstik’s team compared it with the old approaches using the same LLM and the same data, with three runs per case. The reported correctness results were:
| System size | Old approach | New approach |
|---|---|---|
| 64 GPUs | 80% correctness | 100% correctness |
| 460,000 GPUs | About 30% correctness | 100% correctness |
The new approach maintained that correctness across the tested scales, while the baseline degraded as the inventory grew.
Separately, Phaidra reports zero failures across 66 cases drawn from six real production systems. These are outcomes on the reported test cases, not a guarantee of error-free operation on arbitrary future queries.
The token comparison has two different denominators:
| Measurement | Old approach | New approach |
|---|---|---|
| One evaluation pass at one-gigawatt scale | 116 million tokens | 390,000 tokens |
| Per query at both 64 and 460,000 GPUs | Not stated | 9,000 tokens |
The evaluation-pass comparison represents approximately 300 times fewer tokens. The separate per-query result shows why the architecture matters as a customer grows: adding equipment did not increase the reported query token usage at those two scales. These figures describe token consumption, not a measured dollar price or a claim that backend computation is constant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep judgment in the model and exact work in code
Levstik connects this division of responsibilities to Karpathy’s Software 1.0 and Software 3.0 framing. Software 1.0 is deterministic code: predictable and exact, but less flexible. Software 3.0 is behavior elicited through prompts: adaptable enough to interpret a request about hot GPUs, but fuzzy. In the legacy-software trend he describes, prompts increasingly replace work previously expressed as code. For a new AI-native system, maturation can move in the other direction.
The practical boundary follows the work’s requirements:
- Model judgment: parse ambiguous requests, decide where and what to search, handle unfamiliar phrasing, and synthesize the final human-readable answer.
- Deterministic execution: represent hierarchies, graphs, and schemas; retrieve equipment in bulk; apply exact set logic; count results; and deduplicate near-identical names.
If a structure or rule can be written down, code can execute it reproducibly. A language model scanning a well-structured inventory token by token discards precisely the information that makes conventional retrieval effective. Use the model where interpretation is necessary, and give it exact tools wherever the rules are known.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Productionize by adding reliable tools
Phaidra began with an almost pure Software 3.0 prototype, putting everything into the context window because that was a fast way to discover which behavior was worth building. As the system encountered real scale, the team moved known engineering problems into deterministic code while retaining difficult judgment in the LLM. For the use cases that justify it, AI-native software can therefore start with prompts and mature by adding Software 1.0.
Those functions do not replace model judgment; they give it reliable results and structure to work with. The resolver supplies an equipment set instead of asking the model to guess its members. Each exact tool removes another mechanical uncertainty from the model’s task.
Leave the hard decisions with the LLM. Implement the parts it should not be guessing about, then hand that structure back to it. Levstik’s closing formulation, after correcting himself, is to demo with Software 3.0 and productionize by adding Software 1.0.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Phaidra discusses why physical relationships, large context, and repetitive sensor data complicate industrial AI.
API reference for response fields, termination reasons, and validation of generated function arguments.
Full transcript
Loading transcript…