AI Engineer World's Fair 2026

Stop Renting Your Cognitive Infrastructure

Loading slides…

Read the talk

Stop Renting Your Cognitive Infrastructure

Inference that looks cheap per call can become expensive at scale, while enterprise workloads introduce requirements that a lower token price cannot solve.

From a talk by Thiyagarajan Maruthavanan

When inexpensive calls become an expensive dependency

Thiyagarajan Maruthavanan opens with an unnamed major retailer that, he reports, spent nearly $200 million on Anthropic inference before building its own infrastructure. He follows with the Uber CTO’s budget-overrun account: in Maruthavanan’s telling, an annual token budget was exhausted in month four. These are accounts of inference spending, not measurements of what ownership would have saved.

Black slide displaying “$200M” and “One enterprise. On the cost of inference. Then they built their own.” Speaker visible at right.Open full source frame
One enterprise’s $200M inference cost anchors the opening problem.

The difficulty is that an inexpensive individual request gives little intuition for the accumulated bill. A monthly phone bill supplies a recurring number to anchor expectations. Prepaid inference credits encourage a different rhythm: load credits, consume them, replenish them. Maruthavanan compares that cycle to buying casino credits—each replenishment feels manageable until the total crosses the threshold you had in mind. His own application, Ultasuno, brought that problem home.

0:000:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

A reverse music-prompt application meets scale

Suno turns a text prompt into music. Ultasuno asks the reverse question: given a song, what prompt could have generated it? The output is a plausible description for generation, rather than a recovery of some known original prompt. Maruthavanan built it for himself, enjoyed using it, and shared it with friends; distribution then extended beyond that initial circle. He reports hundreds of thousands of users and hundreds of thousands of dollars in inference spending.

Adoption explains why requests multiply, but it does not explain how much work each request needs. Maruthavanan identifies three places where applications waste inference:

  • Context management: carrying more context than the current task requires.
  • Input compression: sending material without reducing its token footprint.
  • Agent loops: making repeated calls that consume tokens without enough useful progress.

His diagnosis is that the inference endpoint does not understand the shape of the consuming workload. It can serve the request it receives, but the application has the context needed to decide whether that request, its input, or another loop iteration is necessary. Workload efficiency starts before the endpoint.

1:071:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:07 · section reference included

A stolen key creates a second spending problem

Not all consumption comes from legitimate users or inefficient agents. Maruthavanan says his API key was stolen three weeks before the talk, and attributes the unauthorized use to someone in China. He watched the cost rise through $7,000, $7,500, and $8,000 before his co-founder, who leads research and technology, stopped it at $10,000. His suggestion that it could have reached $100,000 describes a feared outcome, not an incurred expense.

This is a different failure mode from a successful application becoming expensive: possession of a credential allowed someone else to consume the endpoint against his account. He does not describe the containment procedure. The incident leads into a broader question about dependence on rented inference—and the alternative commonly offered in response, a token factory.

2:082:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:08 · section reference included

From a cloud token factory to local inference

A token factory changes where inference comes from. Instead of paying Anthropic or OpenAI for their models, a customer uses open-source models deployed by a neo cloud or inference provider, with capacity provisioned in tokens per second. The local version takes the same idea into a garage or basement: buy GPUs, assemble a machine, and run the models yourself. Maruthavanan encountered that approach through people promoting local inference on AI Twitter.

He bought DGX Sparks and moved Ultasuno off Anthropic onto that setup. The application initially worked well, but memory became a bottleneck. He then expanded beyond the application to agents for his research lab, describing growth from two agents to six, eight, and twelve. The setup was useful enough to support that experimentation.

White slide with the heading “Local Inference Memory was the bottleneck,” a compact computer pictured above right, and the speaker beside the slide.Open full source frame
Local inference: memory was the bottleneck.

The next step exposed a different standard. Three enterprises asked him to replicate the setup, and a local installation that worked for his own applications did not automatically settle their reliability requirements. Serving tokens locally and operating an enterprise inference service are different responsibilities.

2:352:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:35 · section reference included

Control, audit approval, and reproducibility

The three organizations were an investment fund, a hospital, and a tax practice. Each had encountered a constraint that cheaper inference alone would not remove.

OrganizationWorkload or situationBlocking requirement
Investment fundInvestment analyst on a neo cloudControl over rate limits
HospitalWorking use case undergoing auditAcceptable vendor dependencies
Tax practiceGenerated tax recommendationsAbility to recreate a recommendation

The fund did not want an outside provider to dictate its consumption limits. The hospital had a working use case, but an audit red-lined a third-party vendor dependency and prevented it from proceeding. For the tax practice, producing a recommendation was insufficient: it needed to recreate that recommendation later. Maruthavanan attributes that obstacle to insufficient access to model internals; the example establishes a reproducibility requirement, not that access to internals alone guarantees reproducibility.

3:474:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:47 · section reference included

When does renting stop fitting the workload?

Maruthavanan places the decision boundary at product-market fit. Before demand is established, renting lets a founder test whether a use case deserves to exist. After product-market fit, he argues for building owned infrastructure. He applies the same reasoning to an enterprise that has already budgeted a project: in his framing, committing that budget assumes that the use case has sufficient demand to justify an enduring deployment. This is his proposed ownership threshold, rather than a calculated cost break-even point.

Black slide listing Pre-PMF: keep renting; Post-PMF: start building; Enterprise: build your own inference infrastructure. Speaker visible at right.Open full source frame
“Where do you sit?” separates pre-PMF, post-PMF, and enterprise infrastructure choices.

His analogy is moving to a new city. An Airbnb or rented home lets you experience the city and its neighborhoods before committing to a house. Eventually, however, the temporary arrangement no longer fits the life being built around it: “You cannot raise a family in an Airbnb.” That was the conclusion he reached for his applications, agents, and the infrastructure needed to scale them.

4:525:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:52 · section reference included

Ownership and token optimization operate at different layers

Maruthavanan calls his infrastructure effort jusInfer. Building it also reinforced that optimization is available before ownership: input cost, token management, and context management remain relevant while renting or leasing inference. The local deployment described here is his historical account; the current jusInfer site describes provider routing and prepaid billing, rather than documenting that local architecture.

He packaged some of those optimization experiences into the open-source project jusTokenMax, positioning it as an alternative to Headroom. Although he calls the comparator “Headroom from Netflix,” contemporary reporting identifies it as Tejas Chopra’s project, created while he was a Netflix engineer, not an official Netflix project.

Maruthavanan claims that jusTokenMax outperforms Headroom on many parameters, but supplies no numerical results or benchmark conditions in the talk. The project’s published results report input-token reductions without a Headroom baseline or an answer-quality comparison, so they do not establish that comparative claim. The useful architectural distinction remains: reducing the work sent to inference and choosing who operates inference are separate decisions.

5:536:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:53 · section reference included

Choose against your workload, not a market prediction

Maruthavanan also introduces his book, Peak Inference: Infraeconomics of AI Inference, as a treatment of when to build inference infrastructure. He describes an AI market whose rules change every three to six months, leaving buyers surrounded by competing predictions.

He associates Jensen Huang with a future of token factories, Satya Nadella with local, unmetered intelligence, and neo clouds and inference providers with the expectation that endpoint providers will capture the value. These are the market narratives as he presents them. Their disagreement leaves the application builder with the practical task of deciding which arrangement fits the workload. His own path—from an application to agents, then scaling and infrastructure—led him toward ownership.

Black slide contrasting Jensen/Nvidia, Satya/Microsoft, and Lin Qiao/Fireworks, with a closing line advocating ownership for enterprises and the speaker at right.Open full source frame
“Everyone sells one story. Find your own answer.”

“Rent to learn, own to earn.” That is Maruthavanan’s closing rule, paired with a qualification: listeners must come to their own answers. Renting supported his exploration; established usage, accumulating costs, and enterprise requirements changed what he needed from the infrastructure beneath it.

6:316:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:31 · section reference included

Resources

From the talk

  • Open-source toolkit for reducing coding-agent context through file, log, structured-data and attachment compression, with installation instructions.

  • Context-compression software for tool outputs, files and retrieved material, with library, proxy and MCP integrations.

  • The project's current public overview of workload-aware inference routing, supported integrations and billing.

  • Project-reported token counts and assumptions for PDF conversion, logs, structured data and incremental file reads.

Transcript download
Full transcript

Loading transcript…