AI Engineer World's Fair 2026
Stop Renting Your Cognitive Infrastructure
Loading slides…
Read the talk
Stop Renting Your Cognitive Infrastructure
Inference that looks cheap per call can become expensive at scale, while enterprise workloads introduce requirements that a lower token price cannot solve.
From a talk by Thiyagarajan Maruthavanan
When inexpensive calls become an expensive dependency
Thiyagarajan Maruthavanan opens with an unnamed major retailer that, he reports, spent nearly $200 million on Anthropic inference before building its own infrastructure. He follows with the Uber CTO’s budget-overrun account: in Maruthavanan’s telling, an annual token budget was exhausted in month four. These are accounts of inference spending, not measurements of what ownership would have saved.
The difficulty is that an inexpensive individual request gives little intuition for the accumulated bill. A monthly phone bill supplies a recurring number to anchor expectations. Prepaid inference credits encourage a different rhythm: load credits, consume them, replenish them. Maruthavanan compares that cycle to buying casino credits—each replenishment feels manageable until the total crosses the threshold you had in mind. His own application, Ultasuno, brought that problem home.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reverse music-prompt application meets scale
Suno turns a text prompt into music. Ultasuno asks the reverse question: given a song, what prompt could have generated it? The output is a plausible description for generation, rather than a recovery of some known original prompt. Maruthavanan built it for himself, enjoyed using it, and shared it with friends; distribution then extended beyond that initial circle. He reports hundreds of thousands of users and hundreds of thousands of dollars in inference spending.
Adoption explains why requests multiply, but it does not explain how much work each request needs. Maruthavanan identifies three places where applications waste inference:
- Context management: carrying more context than the current task requires.
- Input compression: sending material without reducing its token footprint.
- Agent loops: making repeated calls that consume tokens without enough useful progress.
His diagnosis is that the inference endpoint does not understand the shape of the consuming workload. It can serve the request it receives, but the application has the context needed to decide whether that request, its input, or another loop iteration is necessary. Workload efficiency starts before the endpoint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A stolen key creates a second spending problem
Not all consumption comes from legitimate users or inefficient agents. Maruthavanan says his API key was stolen three weeks before the talk, and attributes the unauthorized use to someone in China. He watched the cost rise through $7,000, $7,500, and $8,000 before his co-founder, who leads research and technology, stopped it at $10,000. His suggestion that it could have reached $100,000 describes a feared outcome, not an incurred expense.
This is a different failure mode from a successful application becoming expensive: possession of a credential allowed someone else to consume the endpoint against his account. He does not describe the containment procedure. The incident leads into a broader question about dependence on rented inference—and the alternative commonly offered in response, a token factory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a cloud token factory to local inference
A token factory changes where inference comes from. Instead of paying Anthropic or OpenAI for their models, a customer uses open-source models deployed by a neo cloud or inference provider, with capacity provisioned in tokens per second. The local version takes the same idea into a garage or basement: buy GPUs, assemble a machine, and run the models yourself. Maruthavanan encountered that approach through people promoting local inference on AI Twitter.
He bought DGX Sparks and moved Ultasuno off Anthropic onto that setup. The application initially worked well, but memory became a bottleneck. He then expanded beyond the application to agents for his research lab, describing growth from two agents to six, eight, and twelve. The setup was useful enough to support that experimentation.
The next step exposed a different standard. Three enterprises asked him to replicate the setup, and a local installation that worked for his own applications did not automatically settle their reliability requirements. Serving tokens locally and operating an enterprise inference service are different responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control, audit approval, and reproducibility
The three organizations were an investment fund, a hospital, and a tax practice. Each had encountered a constraint that cheaper inference alone would not remove.
| Organization | Workload or situation | Blocking requirement |
|---|---|---|
| Investment fund | Investment analyst on a neo cloud | Control over rate limits |
| Hospital | Working use case undergoing audit | Acceptable vendor dependencies |
| Tax practice | Generated tax recommendations | Ability to recreate a recommendation |
The fund did not want an outside provider to dictate its consumption limits. The hospital had a working use case, but an audit red-lined a third-party vendor dependency and prevented it from proceeding. For the tax practice, producing a recommendation was insufficient: it needed to recreate that recommendation later. Maruthavanan attributes that obstacle to insufficient access to model internals; the example establishes a reproducibility requirement, not that access to internals alone guarantees reproducibility.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When does renting stop fitting the workload?
Maruthavanan places the decision boundary at product-market fit. Before demand is established, renting lets a founder test whether a use case deserves to exist. After product-market fit, he argues for building owned infrastructure. He applies the same reasoning to an enterprise that has already budgeted a project: in his framing, committing that budget assumes that the use case has sufficient demand to justify an enduring deployment. This is his proposed ownership threshold, rather than a calculated cost break-even point.
His analogy is moving to a new city. An Airbnb or rented home lets you experience the city and its neighborhoods before committing to a house. Eventually, however, the temporary arrangement no longer fits the life being built around it: “You cannot raise a family in an Airbnb.” That was the conclusion he reached for his applications, agents, and the infrastructure needed to scale them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ownership and token optimization operate at different layers
Maruthavanan calls his infrastructure effort jusInfer. Building it also reinforced that optimization is available before ownership: input cost, token management, and context management remain relevant while renting or leasing inference. The local deployment described here is his historical account; the current jusInfer site describes provider routing and prepaid billing, rather than documenting that local architecture.
He packaged some of those optimization experiences into the open-source project jusTokenMax, positioning it as an alternative to Headroom. Although he calls the comparator “Headroom from Netflix,” contemporary reporting identifies it as Tejas Chopra’s project, created while he was a Netflix engineer, not an official Netflix project.
Maruthavanan claims that jusTokenMax outperforms Headroom on many parameters, but supplies no numerical results or benchmark conditions in the talk. The project’s published results report input-token reductions without a Headroom baseline or an answer-quality comparison, so they do not establish that comparative claim. The useful architectural distinction remains: reducing the work sent to inference and choosing who operates inference are separate decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose against your workload, not a market prediction
Maruthavanan also introduces his book, Peak Inference: Infraeconomics of AI Inference, as a treatment of when to build inference infrastructure. He describes an AI market whose rules change every three to six months, leaving buyers surrounded by competing predictions.
He associates Jensen Huang with a future of token factories, Satya Nadella with local, unmetered intelligence, and neo clouds and inference providers with the expectation that endpoint providers will capture the value. These are the market narratives as he presents them. Their disagreement leaves the application builder with the practical task of deciding which arrangement fits the workload. His own path—from an application to agents, then scaling and infrastructure—led him toward ownership.
“Rent to learn, own to earn.” That is Maruthavanan’s closing rule, paired with a qualification: listeners must come to their own answers. Renting supported his exploration; established usage, accumulating costs, and enterprise requirements changed what he needed from the infrastructure beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source toolkit for reducing coding-agent context through file, log, structured-data and attachment compression, with installation instructions.
Context-compression software for tool outputs, files and retrieved material, with library, proxy and MCP integrations.
The project's current public overview of workload-aware inference routing, supported integrations and billing.
Further reading
- jusTokenMax benchmark resultsDocumentation
Project-reported token counts and assumptions for PDF conversion, logs, structured data and incremental file reads.
Full transcript
Loading transcript…