AI Engineer NYC 2026 Online Talks
I Spent Two Years Building the Median — Hariprasad Sudharshan
Slide chapters
20 slide and demo groups · Timestamps are approximate.
Read the talk
I Spent Two Years Building the Median
Hariprasad Sudharshan’s solo-built calculator worked—and resembled calculators other people had built independently. His response was to change what he budgeted for: less attention to coding workflows, more to a specific customer problem, distribution and a difference worth building.
From a talk by Hariprasad Sudharshan
At a glance
Ideas worth remembering
Separate model capability, the harness that applies it and the product customers receive. Comparing coding products does not isolate a model’s contribution.
A useful application can still be easy for others to reproduce. Internal utility, learning and commercial differentiation require different success criteria.
Before building, search for existing solutions and name the distinctive contribution. Put that contribution in the roadmap and review the effort devoted to it.
Evaluate demos by asking who already solves the problem, what the team uniquely contributes and how the application will reach users.
Free model weights leave inference infrastructure, operating expertise and workload utilization to account for. The revised calculator makes that enterprise decision its focus.
The software underneath the city still has to work
Imagine New York stopping for the evening: no flights leaving JFK, a subway stranded at its next station, failed tap payments, elevators losing track of their floors. Hariprasad Sudharshan opens with that thought experiment to make an ordinarily invisible dependency visible. Everyday life relies on software that follows rules and behaves reliably. A product person with almost two decades of software experience, he begins his account of AI-assisted development with the systems people already trust.
In his simplified picture of traditional software development, engineers supply judgment: what to build, which tradeoffs to accept and what a good product feels like. Their tools execute deterministic rules. Those tools also help build the next generation of tools, allowing the industry to accumulate capability. Differences between engineers—their experience, taste and priorities—therefore become differences between products.
AI changes one part of this picture. Engineers can now rent model capability through APIs, along with much of the surrounding development machinery. Small teams gain access to capabilities they could not previously assemble by hiring. Yet the harness, tools and integrations around the model still contain ordinary software, and the delivered application still needs to behave reliably. Sudharshan’s forecast is that probabilistic models will accelerate the production of deterministic software and make previously unaffordable or neglected applications practical. That is his framing of the opportunity, rather than a claim that every AI application has deterministic behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A model leaderboard does not describe the whole product
The next comparison comes from sprinting. Elite athletes can be close enough that a tiny difference determines the winner, while fans remain deeply attached to their favorites. Sudharshan recognizes the same emotional investment in model discussions. A model’s lead can move with the next release; making that temporary lead the basis of a product roadmap gives a team an advantage it does not control.
Even the numbers require care. The example ratings are 1,503 for Anthropic and 1,416 for Mistral, a gap of 87 points. These are relative ratings derived from comparisons in a particular evaluation environment. They do not mean one system possesses 87 more units of intelligence. The useful question is what the evaluation measures and whether that difference matters for the application.
There is a second source of confusion: comparing Claude Code, Cursor and Codex does not isolate a model. Such comparisons mix several things:
- Model: The underlying capability—the athlete or engine in Sudharshan’s analogy.
- Harness: The tools, context, instructions, orchestration, memory, workflows and MCP tool calls that determine how that capability is used.
- Product: The result a customer receives and the reason they care about it.
A better experience may come from the harness as well as the model. Some offerings couple these parts tightly; others let a team combine them more freely.
This separation leads to the product question. If competitors can rent the same engine and much of the same harness, what remains distinctive? The capability can be excellent while being equally available to the next team. Access alone does not explain why a customer should choose one application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The median can be useful and easy to reproduce
Sudharshan calls the readily available answer the “median.” This is his name for a shared starting point, rather than a measured statistical median of software quality. Point similar tools at a familiar problem and different builders can arrive at similar solutions. As model releases improve, that starting point moves upward: more of the work becomes available before a team has contributed a distinctive idea.
The trap is that the result may already be good enough to ship. A builder sees a working product; a customer sees another option among several similar products. When that happens, distribution—reaching customers and getting the application adopted—can become the decisive advantage. Independent convergence is enough to create this situation. Nobody needs to copy anyone.
That does not make the software worthless. Sudharshan pushes back on calling it all “AI slop.” Two weeks before the talk, he shipped an iOS app despite having never expected to work in that development lane. Other builders may want revenue, a useful internal tool, or simply a website they can proudly show a colleague. Those are different goals, and each can be satisfied without creating a defensible commercial product.
The distinction is between usefulness and differentiation. A coding agent can produce a useful finished artifact that solves a team’s problem. Competing in an open market introduces another requirement: a reason to choose this implementation. Sudharshan reaches back to search engines for the analogy. A crawler and an index establish basic participation; a distinctive way to solve search well can change the competitive position. His own public building and contributions to LlamaIndex supplied plenty of learning, but did not spare him this distinction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The build-first reflex skips the search
When building was costly, searching for an existing solution was often the cheaper first move. AI-assisted development reverses that feeling. Opening an agent can seem easier than reading through existing projects and deciding what to adopt. Sudharshan calls the resulting habit the “build-first reflex”: “Open a problem, open an agent.”
His example is a retrieval-augmented generation, or RAG, system. A user supplies a PDF and asks questions through a backend language model; the implementation involves chunking and a vector database. Getting those pieces working feels like a substantial win, especially on a first attempt. In his illustrative timeline, scaffolding appears in two minutes, a rough working demo in two hours, and a supported product in four weeks. Only afterward does the builder discover mature open-source alternatives. Speed has shortened implementation while leaving the search decision undone.
Why did the agent build another RAG system instead of considering an existing one? In the exchange Sudharshan describes, the agent’s explanation is simple: it had been asked to build. A different request could have led to a different course. The response exposes the missing decision. Successful execution of a construction request does not answer whether construction was necessary. Meanwhile, small working increments supply enough satisfaction to keep the builder moving before that question gets asked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two years of building meet an already crowded lane
The calculator brings the argument home. It was one of many applications Sudharshan built during two intense years of solo work. He skipped social occasions, movie nights and time outside, describing the attraction as something like an addictive game of chess. AI platforms let him turn ideas into applications without the large team and architects he would previously have expected to need. That visible progress made the effort feel justified.
The soft launch supplied a different comparison. People in his circle had seen similar calculators elsewhere. Looking at other open and closed implementations, he found results that sometimes seemed the same, beyond merely resembling one another. The useful comparison is what those calculators actually offered: his interpretation was that familiar problems and shared development tools had led independent builders toward the same answer. The surprise was discovering how little two years of effort alone said about distinctiveness.
The same pattern appeared across his projects: a promising idea led to immediate building, then a crowded lane. Sometimes competitors were already there; sometimes he returned after a month away to find many similar products. His analogy is having the same developers on your team and your competitor’s team. The experience gave him practical intuition, but also exhaustion. Building had taught him what could be produced quickly; the next task was learning where to spend effort that would survive comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put the distinctive part in the budget
The correction is a “novelty budget.” Before starting, name one thing that makes the project distinctive, put it in the roadmap, and review it as deliberately as spending. PageRank serves as Sudharshan’s example of a specific contribution that could distinguish one search engine from others. The budget is a way to reserve attention for such a contribution; it is not a numerical formula for buying originality.
For the calculator, this changed how time was allocated. The first two years included architecture, agent design, prompt chains, skills and experiments with new coding workflows. The rework concentrated on differentiation. Tool fluency had produced results, but those results often resembled what others could obtain. If a project has no identifiable novelty, Sudharshan’s advice is to accept it as a learning exercise or use an existing solution. Both are legitimate outcomes; neither requires pretending that implementation effort created an advantage.
The roadmap advice contains several distinct decisions:
- Reuse solved work. A product may already have an open-source twin. Adopting or forking it can avoid rebuilding the common foundation.
- Choose an opinion. Decide how this product should solve its particular problem instead of accepting probable, broadly familiar answers as its identity.
- Match effort to purpose. A prompt-to-production application can be entirely appropriate for internal consumption.
- Treat “done” as a starting point for differentiation. When an agent finishes the baseline implementation, inspect what still needs human judgment to make the product distinctive.
Sudharshan compresses this into a deliberately sharp description: coding agents provide the “last great design for free.” He expects them to scaffold established approaches well and assigns the search for the next important design to the builder. This is his working stance toward current tools, rather than proof that models cannot produce novel designs. Its practical force is to stop treating an agent’s completion message as evidence that the product question has been settled.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three questions recalibrate the demo
Leaders face a related problem. Their intuition may have formed when a particular deliverable required a team and a quarter of work. A new engineer can now show something similar after much less effort. The demo is real, but its old production cost is a poor guide to its present value. Sudharshan recommends hands-on building to recalibrate that intuition, then three questions when evaluating what a team presents.
- Who has already done this? Force the search that the build-first reflex skipped. A functioning demo may reproduce a mature existing solution.
- What do we have that the next team does not? Make the distinctive contribution explicit. A better prompt alone may still produce an answer available to competitors.
- Where will this application live? Ask how the application will reach users and become useful in practice. Without a distribution model, a demo can be an expensive way to discover that nobody has a path to adopting it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The revised calculator asks what free weights cost to run
Applying those questions changed the calculator’s subject. The new question was what stands between an open model and a system that actually generates tokens—and what an enterprise must pay for that system. “Open weights are not open inferences” is the distinction. Access to model weights leaves a substantial operating problem to solve.
Sudharshan describes ten intervening layers, naming concerns including precision, KV cache, batching, execution parallelism, MLOps and site reliability engineering. These put work between possession of a model and dependable token generation. Some require specialist expertise. The recording names these concerns without explaining a complete ten-layer architecture, so the lesson is the scope of the operating burden rather than a reproducible deployment recipe.
Utilization receives special attention. He calls it structural: an enterprise cannot simply purchase a utilization level in the way it purchases infrastructure. That makes the organization’s actual workload consequential to the decision. A comparison based only on access to free weights misses the question of how the resulting serving system will be used.
What should the revised calculator let an enterprise compare? It holds the model weights constant and asks about the cost of running them inside the enterprise versus running them in a neocloud and buying the generated tokens. Sudharshan reports that none of the 244 calculators he examined addressed this question, and that enterprises responded positively to the revision. Those are his reported search and feedback results; they do not establish a validated cost model or a disclosed business outcome. The practical invitation is to enter a real workload and see which option the calculator favors. He describes the application as free and open source.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the education; change where judgment goes
The ending preserves both sides of the experience. Readily generated products can be useful, and strong distribution may sustain them in competition. Sudharshan continues to ship them. His recommendation for a product seeking a stronger position is to pick one problem and pursue it with the intent of solving it exceptionally well. That intent directs effort; it does not guarantee the ambition will be achieved.
The two years were education, and he considers them worth the tokens. He also returns to the city underneath the opening thought experiment: in his view, AI makes reliable software faster to write while the world continues to depend on it. The part of the work he now wants to protect is judgment about what deserves to be different. His closing phrase gives that allocation its edge: “novelty is scarce and the median is free.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Develops the model–harness distinction through concrete optimization examples: changing prompts, agent programs and repository skills using execution feedback.
Read the complete timestamped transcript
19 slide groups · 1 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate.
- 0:02
If I tell you that New York City tonight suddenly stops working, you tell me what could be possible reasons for no flights leaving JFK. The subway suddenly stops at the next station and stays there. None of your Tap Pay is working. The elevators you get in, they don't know which floor they are. The phones are not ringing. The Wall Street and the entire world comes to a
- 0:31
standstill. What could be the reason?
The engineer and the next toolchain
Engineers use judgment and toolchains to build deterministic software; new tools feed back in, as assemblers built C and C built Unix. Hari says software drives the $125 trillion world economy.
Text in image
$125T The modern world runs on it. Every dollar of world GDP stops without it. NAVIGATION · BANKING · HEALTHCARE · APPLIANCES · GRID · SPACE · IMF WEO 2026 HIRED · JUDGMENT The engineer judgement varies: taste, scars, the last bug shipped. Uniqueness OWNED · DETERMINISTIC The toolchain COMPILERS · EDITORS · FRAMEWORKS · CI · CLOUD Rules all the way down. When a rule needs a judgment, it waits for the engineer. Every step of modern software is built brick by brick OWNED · DETERMINISTIC The software Output 1. Rule-based, predictable, the foundation. $1.4T sold as software, $28T of digital economy, everything else downstream. drives OWNED · DETERMINISTIC The next toolchain Output 2. Assembler built C, C built Unix, Unix built everything. Each one fed back in. feeds back in
- 0:38
Software. Unless otherwise, there is a software issue that affects the entire globe, there is nothing that can stop this modern world. This is how much the city of New York and our entire globe is actually driven by unseen software. Not the AI kind of software that we mostly talk about. It's the traditional, deterministic, rule-based, reliable software that has brought
- 1:08
the modern world to us. It is handcrafted for over seventy years by humans. I'm Hari. I'm the product person who has built software for almost, like, two decades now, and I have this one genuine theory.
- 1:29
If at all there is a new industry in the human history, it must be software industry. Because you take navigation, agriculture, construction, and even astronomy, we had this in different forms and shapes earlier. Now that it may be more sophisticated, but they're nothing new to us. Today, from getting to work to watching this video, from nuclear reactor to all the way to space station, that drives
- 2:00
the hundred and twenty-five trillion dollar economy, the one common factor is software. So it would be catastrophic if for some reason software stops, just like the Y2K. And our modern world owes a lot to this kind of a software, the deterministic kind, where you trust it with your pacemaker and when you fly on your seven four seven.
- 2:27
So if you look into the product here, on the left side, you have the engineers.
- 2:36
The engineers are hired, and they are the source of judgment for every product out there with their unique taste. And I'll go to an extent and call their belief systems and value systems are reflective in the software that they shipped. The tool chain owned deterministic rules all the way. The software that runs the world and the tool chains next to it in
- 3:06
a cross-feedback loop where the assemblers build C, C built Unix, Unix built everything, and then we built Kafka, ZooKeeper, all the way up to Kubernetes that runs the world today.
- 3:21
The only judgment in this picture is human judgment, and we all know how varied we are, and that variance is where every product edge originated.
- 3:36
The block here is a high-level representation of the entire software industry and the world economy. Who has the IP, where is the continuous loop, and how that loop where every brick on the tool chain helps us scale new heights. Whether the software is deterministic or probabilistic, that's a question. And hold this picture. Let's watch one card change, which is, yes, in the age of
- 4:06
AI, the engineers today are assisted or driven, or sometimes they claim it's completely autonomous by models that I would call rented.
- 4:19
The good thing is that it creates a level playing field with everyone having equal access to the bleeding-edge models and harnesses. Teams who cannot afford earlier to the talent pools of Google and Meta, who handcrafted this modern software, can now afford intelligence through APIs, where it claims to have solved the software problem. Let's see whether it really solved or what's happening. If you actually pay
- 4:49
attention and look under the hood, everything around the model, the harness, the agents, the tools, the MCP are still the same old deterministic software. People who are building this would definitely know that the-- these are not AI products. These are the same old deterministic software built through maybe the models. And what did not change and stay put
- 5:19
is what's the outcome? At the end of the wor- day, the world requires software that does the same thing reliably every time. So the final artifact is a deterministic software that drives the world economy as we know it. If that is the case, then what is the role of AI here? Its role is to, one, to
- 5:49
accelerate the development of deterministic software at the lowest cost possible
- 5:57
and write deterministic software that never existed before. Because of budget, not an interesting problem, hard problems. So those are the two use cases I see gonna play out in the next five to 10 years. So the bottom line is, we will be using a probabilistic gear to crank out a reliable deterministic software by
- 6:27
augmenting or somebody will go to an extent, say, replacing human judgment. Is it possible? Where are we today? We will see in this talk.
The sprint analogy
French sprinter Jimmy Vicaut, whose personal best was 9.86 seconds, finished eighth in the 2015 men's 100 metres in 10.00. Usain Bolt won in 9.79 seconds; Justin Gatlin was second in 9.80.
Text in image
Does anyone know him? Photo: Erik van Leeuwen (bron: Wikipedia), GFDL
Show 2 builds
- 6:39
The sprint analogy — recorded presentation build 1
Text in image
Does anyone know him? Photo: Erik van Leeuwen (bron: Wikipedia), GFDL
- 6:48
The sprint analogy — recorded presentation build 2
Text in image
Jimmy Vicaut LANE 1 · 10.00 · 8TH · PB 9.86 Usain Bolt LANE 5 · 9.79 · 1ST Justin Gatlin LANE 7 · 9.80 · 2ND · WORLD LEADER 9.74 Photos: Erik van Leeuwen (bron: Wikipedia), GFDL · Brunel University / Flickr, CC BY 2.0 · Erik van Leeuwen (bron: Wikipedia), GFDL
- 6:39
- 6:41
Who knows this person? I'm a big fan boy of him. He's a superhuman, and he's French.
- 6:51
Okay, now I see you will connect dots because you know the couple of mens there, and he's part of the elite subten club. Two of them are relatively well-known, but all three are superhumans, aren't they?
Nine runners, narrow margins
In the 2015 men’s 100 metres, nine runners finished within 0.27 seconds; Bolt beat Gatlin by 0.01 seconds. The speaker suggests the order could change on another night.
Text in image
Nine superhumans. Twenty-seven hundredths of a second. 100 METRES MEN WIND: -0.5 23 AUG 2015 21:15 RESULT POS | BIB | COUNTRY | ATHLETE | MARK | REACTION TIME 1 | 620 | JAM | Usain BOLT | 9.79 SB | 0.159 2 | 1017 | USA | Justin GATLIN | 9.80 | 0.165 3 | 997 | USA | Trayvon BROMELL | 9.92 | 0.135 3 | 309 | CAN | Andre DE GRASSE | 9.92 PB | 0.136 5 | 1054 | USA | Michael RODGERS | 9.94 | 0.139 6 | 1018 | USA | Tyson GAY | 10.00 | 0.128 7 | 634 | JAM | Asafa POWELL | 10.00 | 0.144 8 | 500 | FRA | Jimmy VICAUT | 10.00 | 0.170 9 | 338 | CHN | Bingtian SU | 10.06 | 0.175 Official result, worldathletics.org · 15th IAAF World Championships, Beijing 2015 Interface text: SEIKO Photo Finish Official Startlist Official Result
- 7:11
So in twenty fifteen, nine souls competed, and the difference between them is only twenty-seven hundredths of a second. And Gatlin lost the gold by one-hundredth of a second. Can you believe it? It's not comprehensible to human mind. The scientific measurements give that, okay, that is how Usain Bolt won over Gatlin that night. On another night, the order could be completely different. I
- 7:41
still run those hundred meters, and I take ten-- thirteen seconds, not ten. How about you? That's the difference between us and these extraordinary athletes competing at the edge of human capability. But as a fan boy, I will argue my favorites like I just did earlier. But I also speak to professionals in this game. And in their opinion, the gap is so tiny that the
- 8:11
previous best or who holds the world record does not matter when they get onto the field. On a given day, anybody can beat anyone.
Six labs, narrow Elo gaps
The slide says Arena’s relative model ratings put the top four 97 Elo points apart in March 2025 and under 25 in March 2026; the top six were within 29 by September 2026.
Text in image
Six labs. A few dozen Elo points apart. Performance of top models on the Arena by select providers Source: Arena, 2026 | Chart: 2026 AI Index report Y-axis: Arena score (Elo-like) Y-axis ticks: 1,000; 1,100; 1,200; 1,300; 1,400; 1,500 Timeline: 2023-May; 2023-Sep; 2024-Jan; 2024-May; 2024-Sep; 2025-Jan; 2025-May; 2025-Sep; 2026-Jan 1,503, Anthropic 1,495, xAI 1,494, Google 1,481, OpenAI 1,449, Alibaba [illegible], DeepSeek 1,416, Mistral AI 1,335, Meta A year earlier the top four spanned 97 points. By March 2026, fewer than 25. By September, the top six sit inside 29. Stanford HAI, 2026 AI Index, Fig. 2.1.4 · arena.ai, Sept 2026
- 8:23
I'm gonna apply the same yardstick on models to drive a couple of points. Six labs, the recent study came out from Stanford. A few dozen Elo points here and there, and the leaderboard is changing with every release cadence. And the popular question this week, which model are you using, and how smarter did you feel by using that model? Isn't that the great dinner table argument these days?
- 8:54
But I can tell you one thing, it is not necessarily a good boardroom argument. Why? Because often the emotional side of us, like the fanboy moment, peaks into meeting rooms. After all, we are all humans. I am not saying don't compare models. All I'm saying is, don't make a small moving frontier gap into your product roadmap as an advantage. Let's lean a little bit deeper here because there are some numbers to unpack.
- 9:25
Anthropic, fifteen hundred and three, Mistral, fourteen sixteen. Eighty-seven points here and there. It does not mean Anthropic is eighty-seven points smarter than Mistral. Actually, the scores, the way we should read them, is relative rating derived from model comparison. It actually tells us how these models rank relative to one another in that evaluation environment. It's not the
- 9:55
absolute intelligence scale. But here is the elephant in the room. In all these conversations across multiple teams I have seen, multiple organizations, while we think we are comparing models, what I've observed is many times we're not even comparing models. We mistakenly compare the harness or the product for models. The Claude Code versus Cursor or Codex isn't a model comparison. It's closer to comparing Jamaica
- 10:25
versus USA. You're comparing an athlete and the training system around the athlete. So the model is the athlete or the engine. It provides the underlying capability. The harness determines how this engine is used, the tools, context, instructions, orchestration, memory, workflows, the MCP tool callings. That's the harness around it. The product determines why anyone should care, because that's the end
- 10:55
product. Today, you and I can rent the engine and very much all the harnesses that come with engine. Some come tightly coupled, some are loosely coupled.
- 11:09
So if everybody has the same source like we saw in the previous picture, where is the differentiation?
- 11:18
The one that used to come, which made these different companies thrive in different areas. The source or the first principles of that is from the diversity and the variance of their engineer minds. Given the model is rented, increasingly, what we are seeing is that once the capability is available to everyone, then what are we staring at?
Same inputs, similar answers
Hariprasad calls the similar answers produced by the same model, problem and tools the median. Model releases move that median forward, but competitors can ship products that are hard to tell apart.
Text in image
same model × same problem × same tools = similar answer The Median The model does not care who is typing. Point it at the same problem with the same tools and it gives the next person what it gave you: good enough to ship, hard to tell apart.
- 11:46
The model and the harness do not care who is typing and from which part of the world, or whether you are a E six, E seven, or a E one engineer. Point at the same problem, same tools. It gives the next person what it gave you. And I want to call this answer as a median. And this median keeps changing with every release cadence, as we saw in the diagram there. The previous
- 12:15
problem was solved, get into these LLMs, and then incrementally, that's what these releases come one after another. Just like software cadence. I'm super simplifying here, but in a nutshell, that's what keeps the median moving up front, which is a good thing.
- 12:35
So at the same time, it's also a kind of a trap because it is good enough to ship, but it is hard to tell apart that what the next person is gonna be shipped. Because the next person or the competitor could have also shipped something similar or exactly the same. This could not have happened before twenty twenty-three.
- 13:04
The median product was expensive. You needed a team, time, stack. Today, that solved problem is free, and the problem that is free is free for everyone. And you go out to the customer like how everybody goes out. And when this problem of so many products which are similar flooding the market, then
- 13:34
the distribution becomes the moat, not your product. So what is that moat that you can create in this new environment? I'm gonna quickly go through this GitHub's own number to reiterate the point
How fast the median gets built
The slide says the median is built 230 times a minute. Of 1.1M+ public repos importing an LLM SDK, 694K were created in Sep 2024–Aug 2025, up 178% year over year.
Text in image
The median is being built 230 times a minute. 1.1M+ public repos that import an LLM SDK 694K of them created in the last 12 months, up 178% year on year 4.3M AI-related repositories, nearly double since 2023 80% of new GitHub developers use Copilot in their first week Source: GitHub Octoverse 2025 (Sep 2024 – Aug 2025 data)
- 13:50
that this model and harness are producing enormous amount of new software. And to me, these are really good software,
- 14:03
but the outcomes are similar. This is not a crowd of copycats. Definitely not. It's a crowd of people who have arrived at the same answer because the source or the cookie cutter is the same.
- 14:17
And the word that our industry choose for what they built is AI slop. The software AI slop. I want to take that word away from you and give you a new meaning to it. Let me be fair first. If you have been developing for a long time and you stumbled upon this AI-assisted development, it felt like magic, isn't it?
Median work versus AI slop
Hariprasad calls much agent-built software the median rather than AI slop: a good problem often yields a useful, finished artifact, but similar outputs should not be counted as differentiation.
Text in image
It's not slop. It's the median. SLOP “AI slop” in this context is low-quality AI-generated software development. However, there is limited empirical research this phenomenon. THE MEDIAN A good problem pointed at a coding agent. Useful, finished, similar. It gets shipped.
- 14:42
Because we were taking a black box and pointing something to it, you would have definitely felt at some point in time, oh, it did really well. Two weeks ago, I shipped an iOS app. I've been a developer, but I've never dreamt of developing an app because remember we had lanes, embedded developer, app developer.
- 15:00
AI today is making everyone a developer. To me, that's a good thing, and I mean it. The word slop is negative, and they use it without any empirical analysis. Here is why I feel that those hundred thousand of repos and proliferation of AI-assisted development is bringing new people into the field who's never developed. The success metrics for these new products cannot be looked at from the lens of those experts.
- 15:30
I would like to define the success of those million AI apps as long as they meet the success of their creators, because who defines what a successful product looks like? For us, it could be something that brings a revenue. For someone else, it could be something that they can show it next to their colleagues. That, "Hey, see, I've built this wonderful website." That's a success too. And I've seen millions of people actually done that with Lovable and Replit, which is cool. They are not AI
- 16:00
slops.
- 16:06
What you get when you point a good problem at a coding agent?
- 16:12
Most times than not, today, it's useful and a finished artifact. Maybe similar because, you know, the same cutter went into the same dough. But most software developed this route is falling into what I call the median. I am not telling you to stop developing or shipping it. It's perfectly fine for internal consumption. The one caveat here is that stop counting this as a differentiating factor.
Build novelty on free foundations
Hariprasad argues that free software foundations need novelty to differentiate: Google in 1998 combined a crawler and index with PageRank.
Text in image
a crawler and an index + PageRank = Google, 1998 The median plus one idea is every product you admire. Take the median for free. Builds the moat on top. NOVELTY your one idea MEDIAN commoditized capabilities FREE
- 16:46
But this median is lacking one thing. We need to also know the positives as well as the disadvantages. It's not a new problem. Back in '90s, everybody was trying to solve the search problem, and most of them had the basic crawler and an index. And whoever is gonna solve that problem really well or who has that novelty element is gonna rise up because that's a differentiating factor.
- 17:16
So today I say the median of most software that you can think of building is actually free. But if you want to compete in the open market, you need that element of novelty.
Building in the open
Hariprasad says he built dozens of apps in public and contributed to LlamaIndex. His GitHub charts show 116 contributions in 2017 and 376 in 2026; an attempt to solve a real-world problem ended up at the median.
Text in image
Building in open since 2017. 116 contributions in 2017 Timeline: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Day labels: Mon Wed Fri Legend: Less More Learn how we count contributions 376 contributions in 2026 Timeline: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Day labels: Mon Wed Fri Legend: Less More Learn how we count contributions github.com/brettleehari Interface text: Contribution settings
- 17:32
So some credibility before I go further because I've spoke almost for a long time. I did not start building when the models got better. I have built more than dozens of applications in public and contributed to LlamaIndex and other repos. The last three years have been quite exciting, and that brought me to this topic, and tiring at the same time. I want to show you this because the project I'm going to show you are most hobbyist first attempts. I tried to solve a
- 18:02
real-world problem, and I ended up with a median. I never expected that.
The build-first reflex
A coding agent starts building a PDF RAG system. Hariprasad describes how a demo in two hours and a launch in four weeks can come before discovering similar tools, because building feels easier than searching.
Text in image
The Build-First Reflex 01 Get a problem 02 Open the coding agent 03 Demo in two hours 04 Launch in four weeks 05 Search. Find the twins. Claude Code v2.1.286 Opus 5.5 · Claude Max [illegible] Updated to latest. Got 56 features, 368 bugfixes, and 169 other changes. code.claude.com/docs/en/changelog for details Get to finished work sooner with Opus 5.5. Switch anytime with /model. can you build RAG system over PDFs ? Ran 1 shell command Skill(claude-api) Successfully loaded skill · 1 tool allowed Allowed by auto mode classifier Read 1 file, ran 3 shell commands Now let me verify the SDK types I plan to use (document blocks with text source, citation types, stream events) against the installed package. Ran 2 shell commands Write(pdfrag/ingest.py) Wrote 80 lines to pdfrag/ingest.py 1 """PDF -> page-aware text chunks.""" 3 from __future__ import annotations 5 import hashlib 6 import re 7 from dataclasses import asdict, dataclass 8 from pathlib import Path 10 from pypdf import PdfReader +70 lines Building became cheaper than searching. The agent will happily build the 400th RAG-over-PDFs tool. It does not know about the other 399. That is your job.
- 18:10
So I want to coin a few terms here. The build-first reflex. When we all got excited with this AI and things, we stopped doing something which is back in the days when building was costly and difficult, I used to search more than I built. Today, it's flipped. You find a problem at a coding agent is more easier than going and searching and reading through those verbose whatever comes out and making a decision. We don't want to make decisions
- 18:40
anymore, and this is getting the new normal in our craft. So what do we do? Let's ask the coding agent to build a RAG system.
- 18:52
And I call this the build-first reflex.
- 18:55
Open a problem, open an agent. Yeah, you get the scaffoldings in the first two minutes, and I'm sure if you've been used to this kind of development, you would do a demo in two hours, working demo, where you give it to someone with rough edges, it works, and you could actually launch within four weeks a real good product with support and everything. But only then you find out that, okay, I have built the four hundredth or four thousandth RAG system. I took RAG as an
- 19:25
example here.
An agent admits median RAG
Asked to build a PDF RAG system, the agent started coding without asking what problem it needed to solve. Challenged about existing open-source options, it called its result a textbook design.
Text in image
THE MEDIAN PROMPT › can you build RAG system over PDFs ? MY QUESTION, ONE TURN LATER › why did you build the nth RAG system when more mature, feature-rich open source systems are available? The solution appears to be a median RAG system. THE AGENT, VERBATIM You're right on both points. I read "can you build" as "go build it" and started coding without asking what problem you're solving. "Median RAG": yes. It's the textbook design. Claude Code · Opus 5.5 · 1 October 2026 · same session as the previous slide
- 19:28
So I pointed the same problem, and this is what we got. And I asked, "Hey, why did you build the Nth RAG system when more mature feature-rich open source systems are available with thousands of stars and maturity? We could have just leveraged it. We should not have built it. Three years ago, we wouldn't have built it."
- 19:50
So it accepted that, hey, okay, I'm the culprit here. You said, "Can you build?" I went ahead and built. If you had told me this, then I would have done something different. The median RAG is what actually it built, which is a solved problem. But if you are building it for the first time, it's gonna be exciting for you because you're gonna feel that I gave a PDF, I pointing it to a backend LLM, and I'm asking the questions, the chunking
- 20:20
and all those factors which I understand or do not understand, it's gonna solve it for me, the vector DB, and I got a solution. Small wins produce a lot of dopamine, and I fall for the same things as you are. And this is the, uh, verbose response. I refined that on the previous slide.
Inspecting the generated RAG design
The agent’s PDF pipeline combines local embeddings and keyword search for cited answers. It flags missing answer-quality evaluation, weak parsing and no reranker; its tests only check that the pipeline runs.
Text in image
can you build RAG system over PDFs ? How it works 1. Ingest (pdfrag/ingest.py): pypdf pulls out the text page by page. It splits the text into chunks of about 1,200 characters at sentence breaks, overlapping by about 200 characters. Each chunk keeps its file name and page number. 2. Index (pdfrag/index.py): embeddings are made on your machine with bge-small through fastembed, so you don't need a second API key. The index is saved in .pdfrag/. Each PDF is tracked by a hash of its contents, so re-indexing only processes new or changed files. 3. Retrieve: two searches run together, meaning-based (embeddings) and keyword-based (BM25), and their rankings are merged. The keyword search finds exact terms like part numbers and acronyms that embeddings often miss. 4. Answer (pdfrag/answer.py): the top chunks go to claude-opus-5-5 with citations turned on. The answer streams in with [1]-style markers, followed by a source list that quotes the cited text with its file and page. Claude is told to say so when the excerpts don't answer the question, instead of guessing. Prompt caching and refusal handling are on. What I checked: - Tests: chunk sizes and overlap, page tracking, that retrieval returns the right page, and incremental re-indexing (unchanged files skipped, deleted files dropped). One test uses a fake Claude stream to check that citation numbers stay correct across turns of a chat. - Real command line on a sample PDF: indexing, re-indexing and search all worked. With no key you get a clear error message, not a crash. A fake key got a 401 back from the API, so requests do reach it. Choices I made for you: - Fallback model: if Claude's safety filters decline a question, the API retries it on another model instead of returning a refusal. Remove the fallbacks lines in answer.py if you don't want that. - Effort: set to medium to keep cost down. Pass --effort high for questions that need reasoning across many excerpts. Limits: scanned PDFs have no text to extract. index warns you about them; run OCR on them first. Tables and multi-column pages come out as flattened text. If your PDFs are heavy on tables, a layout-aware text extractor would be the next upgrade. To try it, put your PDFs in pdfs/, set ANTHROPIC_API_KEY, and run index then ask. why did you build nth rag system when there are more matured feature rich open source systems are available to solve our problem. Question-2: The solution appears to be a median rag system. You're right on both points. 1. Why I built another one I read "can you build" as "go build it" and started coding without asking what problem you're solving. That's the real mistake. Whether to build or adopt depends on things I never asked about: how many PDFs, what kind, who uses it, and how accurate it has to be. If I had asked, the answer would probably have been one of these: - No RAG at all. If your corpus is small (under a few hundred pages per question, roughly), Claude can read whole PDFs directly, up to 600 pages and 1M tokens of context. It sees the actual pages, including tables, charts and scans, which my text-only pipeline throws away. With prompt caching, repeat questions are cheap. For many "chat with my PDFs" cases this beats any chunking pipeline. - An existing system: - RAGFlow: layout-aware parsing, OCR and tables, with a UI. - Onyx (formerly Danswer): team/enterprise search with permissions and connectors. - PaperQA2: scientific papers, with strong citation grounding. - Kotaemon / AnythingLLM / Open WebUI: quick self-hosted document chat. - A framework plus a strong parser: LlamaIndex or Haystack for the pipeline, with Docling, Marker or MinerU to parse the PDFs. Choose this if you need custom behaviour but don't want to own the plumbing. Building from scratch only makes sense if you need something these can't give you, and I didn't establish that. 2. "Median RAG": yes It's the textbook design: extract the text, split it into fixed-size chunks, embed them with a small model, run hybrid search, take the top few, and generate an answer. The specific weaknesses: 1. No evaluation. This is the biggest gap. I can't tell you whether it answers well, because nothing measures it. My tests only check that the plumbing runs. 2. Weak parsing. pypdf flattens tables, mangles multi-column layouts, and skips figures and scans. In most PDF RAG systems, most failures start here. 3. Chunks with no context. A chunk like "Revenue grew 12%" doesn't say which company or which year. Contextual retrieval, where each chunk gets a short description of where it sits in its document, measurably cuts retrieval failures. 4. No reranker. A cross-encoder reranker is cheap and consistently improves the top results. 5. One search pass. Questions that compare documents or need several steps fail. An agentic loop, where Claude issues its own searches and reads neighbouring pages or whole documents when needed, handles these much better. 6. Toy storage. A brute-force numpy array and a JSON file won't scale past small collections [illegible] filtering. Interface text: Churned for 3m 39s · done 3:59 PM Ran 1 shell command, wrote 1 memory 1 new message (click) mostly financial reports with tables, around 500 PDFs ⏵⏵ auto mode on (shift+tab to cycle) · ← for agents
20:36Demo milestone
- 20:42
Now, I'll stop talking about those trivial problems and show one of my applications. This one project is one of the representatives of the first two years where I developed a lot of apps.
Tokens times price
Hariprasad spent his free time building a model-cost calculator with Claude Code and Codex without first checking whether one already existed.
Text in image
Tokens × price. One of mine. I built the median for a problem statement. Then I worked to beat it. How much does it cost to run a model? With Claude Code and Codex I threw every free hour at it. I was addicted to building a calculator. Never checked whether it existed.
- 20:58
Like you all know, my build-first reflex kicks in. I did this a lot of times to coin this term. I built it by sacrificing going out to play. I skipped many socials and movie nights, and I was building like crazy. It was like playing a game of chess where you're addicted to it. And I had every AI platform you could imagine. Max plan, sometimes
- 21:28
free.
Token calculators already exist
After launching his calculator, Hariprasad learned of similar tools. A GitHub search dated Sept 29, 2026 found 244 repos matching “LLM COST CALCULATOR”; the slide also reports 11+ live calculators. He says none copied another.
Text in image
Everyone has a token calculator. 244 REPOS MATCH "LLM COST CALCULATOR" 11+ LIVE CALCULATORS A FEW OF THE TWINS · STARS · CREATED juxhinr/bindwidth — 113 ★ — Aug 2026 orkunkinay/openai_cost_calculator — 15 ★ — Mar 2025 glyphrun/framework-cost-calculator — 10 ★ — Oct 2025 Prajwalsrinvas/token-tariff — 7 ★ — May 2024 Sachinkry/inference-cost-calc — 5 ★ — Jan 2026 dpereirapaz/open-vs-closed-llm-tco — 2 ★ — Jun 2026 The newest shipped in August 2026. Nobody copied anybody. Same cutter, same dough, same shape. GitHub search API, Sept 29, 2026. Queries and full results in the repo.
- 21:32
After all that effort, I finally soft launched. I'm a bit of a perfectionist, so until then, I was bombastic because I could see my idea become something which is real in no time, where as a product manager, it would have taken me a big team, architects. I was able to do it all alone.
- 21:59
I was bombastic until I found out that I haven't calibrated to reality yet. The truth is, my circle told me that they have seen similar calculators elsewhere.
- 22:15
And before anyone thinks I stole from them or they from me, the surprising thing is that nobody copied from anybody. It's the same thing. It's a copycat work. When I analyzed some of those open and, uh, closed calculators, I found it same sometimes, not even similar. And how could people across could arrive at similar solution? Even if you look at the LLM routing,
- 22:46
everywhere is converging like those models did. Even solutions are converging
- 22:52
and competing for a tiny space. And it's like having same developers in your team and your competitors also have the same developers.
- 23:06
So you can build intuition only through building, uh, and I'm sure you're able to-- those builders would agree with me. We now have something to, uh, recalibrate and find a successful path. So after two years where I spent almost all time, I was tired because I spent multiple-- Any exciting idea I had, I went ahead and built with a build-first reflex. And almost every time I ended up with a crowded lane where there are
- 23:36
already two or three people, or I would be the first, and I would take a month off and come back, there would be sixty similar products. And whoever has a better distribution is obviously going to, uh, win that battle.
Spend your novelty budget
Hariprasad says each project gets about one unit of true novelty. His architecture and agent work often produced similar outputs, so he refocused his calculator on differentiation. He advises naming the novel element before building.
Text in image
Novelty Budget You get about one unit of true novelty per project. For two years I spent mine on the architecture, the agent design, the prompt chain, skills.
- 23:51
So I would like to coin another new term. Okay, the build first, then what? In the AI software development life cycle, a novelty budget. So if you can get at least one unit of true novelty, like how that PageRank was for Google, you can cross this median at least for a, a good amount of time. You pick a problem and you wanna be the best Solution for the problem in the world, then you have a good chance of
- 24:21
beating the crowd. For the first two years, I spent, you know, my days and nights on architecture, agent design, prompt chains, skills, and somebody would share, "Hey, this is my Claude new workflow. That's my Claude new workflow." I did all those things. And most of the times than not, trust me, you would end up with similar outputs, if not the same. So I had to rework my calculator that I showed you, and this time I spent almost all of the time
- 24:52
on how to differentiate and almost none on flaunting my Claude Code or Codex skills, which I used to do earlier, because that felt like magic. The median from model isn't opinionated. It is converged best practices. So I picked a lane, and I treated it like a budget. After two years of burning several million tokens on many max plans, I feel everything else is free now, and the highest return
- 25:22
of investment is the novelty budget. So before the build starts, name the one thing that is novel. If you don't have one, then you can take the median because there are a lot of people who have built the solution for the problem that you have, and I'm sure of it. You tell me a problem for which there are no solutions, I'll go and find it for you. And then put this novelty plan as part of your roadmap and your project planning. I
- 25:52
do this now and review it the way you actually review spending, like a project.
- 26:03
Honestly, if you cannot name the novelty in your project, then you should concede that it's a learning exercise. That's fine.
Four ways past the median
To move beyond typical AI output, bring your own opinion, search for existing work, treat the agent’s “done” as a starting point, and invent the next design. For internal tools, he says typical output may suffice.
Text in image
Four ways past the median. 1. Be highly opinionated. We are humans. The median has no opinion. 2. Prompt → production is the median path. Search so you don’t rebuild what exists. Decide what’s wrong with it. Then build. 3. When the AI says “done,” you’re at the start line. That word means you’ve received the median for free. 4. AI gives you the last great design for free. Your job is the next one. An Apple-grade UX in an afternoon, handed to everyone. Inventing the next one is the real work if you want to ship. For internal use happily use the median Apple -grade UX for free.
- 26:19
Uh, the failures, the learnings, and the mechanics to beat medians for two years, I came up with these four moves, and these are not, mm, side project rules. I want you to run them as your roadmap for the next quarter's features because your products will definitely have a twin, and sometimes it might be open source. Why take all the effort in building it? Leverage it, fork it. So one of the things is be highly opinionated.
- 26:49
The AI assistants, most of the times, are gonna give you probable answers, be it anything.
- 26:56
And those things are gonna be same for almost everybody. It might be similar, but that's in the spectrum of already solved. In the world of medians, an opinion is the cheapest novelty you could own, and go for it because the model cannot supply it. Two, the prompt to production is a median path, and there is absolutely nothing wrong with it. If you feel that it's an
- 27:26
internal consumption you want to build and your team has to use it, go for it. I'm all for it. And then the third, this is the most fascinating things. When AI says done, then it means that to me, okay, AI has given its median product. I have to start working from here. That's the signal from any of your coding agents because your work begins there. And remember this,
- 27:57
these coding agents so far has given the last great design for free. It's not gonna inve-invent the next Kubernetes or ZooKeeper or Kafka or whatever those building blocks are. It's gonna do a scaffolding, and it does a really good job in multiple places, but it's, it's, it's gonna be the last greatest design for free. And what's your job? If you want to thrive, you need to find the next one.
Three questions before building
Before building, search for existing solutions, identify a novel advantage such as data or distribution, and name a channel where users already are. A demo can be an expensive way to discover you lack distribution.
Text in image
Three questions to your team before you prompt a coding agent. 1. Who has already done this? “has anyone.” Who. Twenty minutes on GitHub, Hugging Face, Product Hunt. Found three? Fork the best one. The median is free why you build it. 2. What do I have that they do not? “A better prompt” is the median in a costume. The answer must be along the novel lines such as data, distribution or framing. None of the three? It is a learning project. Honestly call it that. 3. Where will this live? Name the channel where the user already is, before the first line. Cannot name it? You are building a demo, the expensive way to learn you have no distribution.
- 28:30
And here are some methods I found useful. I've shared this method with multiple leaders across organizations, and I've got good feedback on them. If you are leading and don't build now with this AI, this is exactly for you. You'll be seeing a bunch of demos. I'm sure you're bombarded with them every week. And it looks like a big deal because I've seen the instincts of leadership who are not calibrated
- 29:00
because back when they were building their intuition, the deliverable that they are seeing took them a quarter or more time and a team. Today, a new engineer shows up and claims to have built the same that would have taken a team and a quarter, uh, to build it.
- 29:22
So it's one of the hardest time to be on deciding where to put your eggs on,
- 29:29
especially when you're not immersed in this tech. Because nobody can recalibrate by reading or watching YouTube or LinkedIn posts. You recalibrate by building. You build intuition by building. So the playbook is simple. Three questions. It's not gonna solve your problems, but these are good handy questions to ask. One, who has already done this? This question forces the search because the world of exciting AI, we need to tame the
- 30:00
build first reflex powered by AI. Two,
- 30:06
what do we have that the next team or our competitors or another person in the team don't have? This forces novelty as a first-class citizen into the minds of the team members.
- 30:22
Because a better prompt, somebody would say, "It's just gonna throw you a median problem, median solution." And third,
- 30:32
where will this application live?
- 30:36
In the age of exciting AI, forcing outcome-based thinking when everyone is high on building is a way to show them the demo is the expensive way to find out if you have no distribution model. And I applied all this rigorously in the last one year, and you may be asking, what happened to the one project that I was talking to you?
Ten layers between models and tokens
The calculator traces ten layers from open weights to billable tokens. Its analysis puts fully loaded self-hosting at 1.1× to 3.1× the cost of renting, with fleet and utilisation costs added back.
Text in image
The calculator I built twice. Between an open model and a token there are ten layers. IN THE DOWNLOAD 1 · WEIGHTS A file you download MODEL EXECUTION 2 · Precision: quantise, calibrate — sets fleet size 3 · KV cache: paging, reuse, eviction — sets concurrency 4 · Batching: continuous, chunked prefill — ~100× on cost 5 · Execution: kernels, graphs, speculative — ~2× per release 6 · Parallelism: TP / PP / EP + collectives — makes it fit PLATFORM & ECONOMICS 7 · Fleet: replicas, autoscaling — idle is yours 8 · Reliability: drain, failover, rolling upgrade — [illegible] uptime 9 · Utilisation: pooled, uncorrelated demand — not replicable 10 · Surface: API, streaming, quotas, metering — a quarter of work OUTPUT BILLABLE TOKEN priced per 1M Engineering. Can be hired, bought, or adopted from open source. Structural. A property of your tenants, not your competence. WHERE THE COST ACTUALLY IS Bare compute — the silicon alone, at full utilisation: 0.07× – 0.50× cheaper than renting, for every model tested Fully loaded — the same fleet, with layers 7 and 9 put back: 1.1× – 3.1× the premium is idle capacity and people, not hardware And no single break-even. Same model, same hardware, same token volume — workload shape alone moves the answer 1.5× to 3.8×, further than model choice does. Sudharshan, H. (2026). From Free Weights to Reliable Tokens: the ten layers of open-model inference. Working paper, Figure 1. CC BY 4.0. tokencalci.onrender.com
- 31:04
I asked these questions. Okay, if I have to be providing that value which is above median, then what stands between an open model and a token-generating system? And what cost does an enterprise have to pay for it? Because open weights are at rage, but open weights are not open inferences.
- 31:31
And I found that the hard way because there are ten layers that sit between an open weight and a system that generates these tokens. We take the labs for granted for providing us. There is MLOps, there is SRE on a different scale altogether that the world has not seen. So precision KV cache, uh, batching, execution parallelism, and there are different slice and dices of them.
- 32:01
So some of them, you have to hire experts, and I don't know where these experts come from because this is not a tech or a skill that the industry already has.
- 32:14
And some of them, layer nine, is utilization. It is structural. You cannot buy it.
- 32:23
So unless, like I said, you take one problem and want it to be the best at solving that particular problem, you're not gonna cross the median just by using these AI tools. And I searched, and I found that none of the two hundred and forty-four calculator that I was showing you earlier did this. And I changed the entire application into kind of
- 32:52
what the identical weights cost to run inside your enterprise versus running it in the neocloud and taking tokens. That is what this calculator gives you because if you have open models and want to go and build it, then what's the difference? And received great feedback from a lot of enterprises who want to actually go that path because
- 33:22
when people say open models, they're excited. For what?
A calculator as a worked example
For Llama 3.3 70B, the calculator estimates $3,600/month to rent versus $6,236/month to self-host at 200,000 daily requests, each with 2,000 input and 500 output tokens, during business hours.
Text in image
Open weights are not open inference. When a lab publishes weights, the conversation moves from frontier to open as though the hard part had shipped. What shipped is the weights. Turning them into tokens at scale — continuous batching, quantisation that doesn’t cost quality, attention kernels, keeping thousands of GPUs hot across uncorrelated demand — is a separate discipline. Frontier labs and neoclouds do that work, and almost nobody prices it. This is a calculator for that gap: what an open model costs to run yourself, against what a neocloud charges for the identical weights. RIGHT NOW, FOR LLAMA 3.3 70B Free — to download the weights $0.240 — per 1M tokens from a neocloud $0.416 — per 1M tokens to run it yourself With nothing idle and nobody paid, your own GPUs reach $0.037/1M — below the neocloud price. For a model this sparse the serving gap is closeable; what you lose it on is utilisation and staffing. And the premium buys exactly one thing: a walled garden — a physical guarantee instead of a contractual one. Zero Data Retention already covers “never stored, never trained on” at no premium. Paying a large premium for a property a contract already gave you is the most expensive unexamined default in this decision. At 200000 requests a day of 2000 in / 500 out, business hours, serving Llama 3.3 70B: RENT IT FROM A NEOCLOUD: $3,600/mo $0.240 per 1M tokens RUN IT YOURSELF: $6,236/mo 2× H100 · fixed, 24×7 Rent it. Self-hosting costs about 1.7× more at 40% duty on 15.0B tokens a month. tokencalci.onrender.com Live prices. Open source. Free, like the median. Interface text: opentoken BETA Estimate The chain Paper Live prices · [illegible] Hariprasad Sudharshan
- 33:28
Organizations considering deploying open weight models. The weights are free, but you may want to check. That's the QR code. So scan it, put it into your workload, and see which way it goes for you. It's open source and free.
Novelty is scarce
Sudharshan says AI makes deterministic software faster to write and supplies the median for free, leaving novelty to builders. He argues that prompt-based median products have little durability in open competition.
Text in image
Novelty is scarce. The median is free. The world still runs on deterministic software, and it always will. The age of AI means we write more of it, faster, for problems that never had a team. The median comes from the model now, and it is free. The novelty still comes from you. That is the only part left worth squeezing your brain for. Hariprasad Sudharshan brettleehari.github.io/Hari.me github.com/brettleehari Which of your projects has a twin? Tell me. I promise it has one.
- 33:49
So here is where it all lands and ends. With the power of the LLMs and the harness, the last three years, building has got exponentially easier. But the changes and the dynamics of software development
- 34:11
still hasn't changed at the end. The prompt-based median products are useful. I ship them too. They're not slops. They have little durability when they go to open competition, and they may survive if you have a huge distribution moat. So the one attitude, if you want to try thrive in this field is, again, I'm, I may be repeating this, but take that one problem and pursue it with the intent of being the
- 34:41
best at it in the entire world. Yeah, we may not be able to, uh, crack it, but that intent will definitely put you above median.
- 34:54
The only question left is, can your AI-assisted products thrive? I spent two years on the median, but that was education. It was worth every token I burnt it. I asked two questions at the start. What happens to the software? To me, the software that is running this hundred trillion dollar economy, nothing happens to it. The world still runs on deterministic software, at least for the foreseeable future, and it might always be.
- 35:24
The age of AI means that we can write it more faster, but it is not going to take out the deterministic software.
- 35:38
So I would like to wrap up by saying that
- 35:44
spend everything you have on one thing,
- 35:50
which is the novelty. Squeeze every brain, every ounce of your brain on novelty because novelty is scarce and the median is free. It's up to you to choose. Thank you.