AI Engineer World's Fair 2024
Breaking AI’s 1 Gigahertz Barrier
Loading slides…
Read the talk
Breaking AI’s 1 Gigahertz Barrier
Faster inference can change more than response time: it can make research, multimodal interfaces and continuously running agents practical parts of everyday computing.
From a talk by Sunny Madra
When speed changes the computer
What changes when a computer crosses a speed threshold? Sunny Madra opens with Intel’s 1999 gigahertz milestone, roughly twenty-five years before this talk. The milestone was a technology demonstration, not a commercial processor launch. The slide’s Intel news release gives the analogy a concrete starting point: a memorable number that invites a larger question about what faster hardware makes possible.
Madra connects that milestone to the later shift toward multicore processors. In his historical comparison, microprocessor speeds improved by three orders of magnitude in about two decades. He then asks what a comparable trajectory would mean for LLMs, invoking Jensen Huang’s view that AI innovation is moving beyond the Moore’s-law curve. The useful question is architectural: when does a faster component change how the whole system is designed?
For a nearer-term example, Madra reports that Groq increased the speed of Llama 3 8B by over 50% between April and June 2024. Groq’s April launch account provides contemporary context, but the talk supplies neither absolute throughput endpoints nor workload conditions for this improvement. Faster inference matters because it expands the amount of work an application can perform before the user has to wait.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A pizza trip instead of a hundred tabs
Input processing is one half of that opportunity. Madra sketches an input rate of 10,000 tokens per second and mentions roughly one-third of a second of processing. He does not specify the input length, so those figures do not establish an end-to-end task time. His point is that a model can ingest and analyze information much faster than a person can read through it.
The concrete example is Globe.Engineer. Give it a task such as planning a trip to New York to try the best pizza, and it breaks the request into several research streams. Madra describes it working against the live Internet: finding flights, taxi options, hotels and food, then assembling an itinerary. The displayed result gathers those branches into one interface, with a navigation tree, travel text and flight cards.
Madra estimates that the Globe.Engineer trip-planning example completes in perhaps less than five seconds. Doing the same research himself might involve tens or even hundreds of browser tabs, each representing a separate line of inquiry. The interface consolidates that work because the model can both process incoming material quickly and generate the resulting plan quickly. Faster output alone would leave the research bottleneck intact; the example depends on both sides of token processing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Industrializing digital work
Once an application can absorb those research streams, the LLM begins to look less like an isolated feature and more like a computing core—or even an operating system. That would change how people program computers and what they expect computers to analyze. Madra frames this as an expansion of human capabilities, setting aside the question of AGI to focus on a different computing paradigm.
His analogy is the move from craft production to industrial production. He groups carmaking, farming and clothing together: a few cars made in a day versus hundreds or thousands, food produced for a village versus national distribution, and a sweater made in a day or a week versus clothing manufactured at scale. These are broad illustrations of increasing productive capacity, rather than a precise chronology of the Industrial Revolution.
For computing, Madra borrows a framework from Paul Maritz, whose career included Microsoft, VMware and Pivotal, where the two met. Earlier eras changed the representation, connectivity or location of work:
| Computing era | What changed | Familiar expression |
|---|---|---|
| Digitization | Paper processes became digital | Files, folders, inbox, outbox |
| Internet | Digital processes became connected | Networked information and services |
| Cloud and mobile | Work moved to new infrastructure and devices | Cloud scale and phone access |
| AI | Digital production can become automated at scale | Generating and analyzing artifacts |
The paper metaphors survive in operating systems because the first transformation largely reproduced existing office processes. Connectivity followed, then what Madra describes as roughly fifteen years of cloud and mobile form-factor changes. AI introduces the possibility of scaling the production of digital work itself.
Presentation artwork makes the distinction tangible. Madra’s illustrative comparison contrasts a designer making one or two artifacts a day, eighteen to twenty-four months earlier, with Midjourney producing a thousand in a minute. Those are his scale illustrations, not established production measurements. The shift he is pointing to is from commissioning individual artifacts to generating many candidates on demand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An LLM at the core
How fast would inference need to become before it could occupy that central role? Madra proposes a hypothetical decision time of 0.1 milliseconds, equivalent to 10,000 sequential decisions per second:
This is a thought experiment about complex decisions, not a demonstrated inference result. At that scale, repeated model calls could become part of ordinary software execution rather than conspicuous pauses in an interaction.
The resulting change would reach how software is built, how it runs and how it scales. Present-day latency makes it easy to assume that an LLM must remain outside the computer’s inner loop. Imagining CPU-like speed growth removes that assumption. Madra credits Andrej Karpathy for a diagram placing an LLM at the center of connections to video, audio, browsers, other LLMs, code interpreters and file systems. The model becomes a coordinator across capabilities that currently have separate interfaces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Personal interfaces need more than fast speech
The first visible change would be a move from responses arriving around reading speed to decisions that feel instantaneous. Returning to the Globe example, Madra contrasts a few seconds of automated research with an afternoon, an evening or several evenings of personal effort. The benefit is not just reading the same answer sooner; it is completing a larger task within the time previously spent waiting for a small part of it.
Personalization adds another requirement. OpenAI’s early memory features provide Madra’s starting point: a system can remember the names of a user’s pets, children or spouse. He cites Bill Gurley and Brad Gerstner’s podcast discussions of personalization as a major frontier. For those details to support a seamless experience, the system must retrieve and use them quickly enough that personal context does not become another source of friction.
The interface then moves beyond keyboards, pointing devices and mobile touch. At the time of the talk, GPT-4o voice demonstrations showed Madra the possibilities of mixed interaction, although he did not regard the full experience as achieved. His shorthand is XRX: any type of input, reasoning, and any type of output. Input and output need not use the same modality.
Consider booking a haircut. A user can ask aloud which appointments are available, but a spoken list is difficult to hold in memory:
| Part of the interaction | Useful modality | Content |
|---|---|---|
| Request | Voice | Ask for available haircut appointments |
| Response | Text | 9 AM, 11 AM, 3:30, 5:30 |
Displaying the options lets the user compare them without remembering a sequence of spoken times. No appointment has been booked at this point; the system has only presented availability. A natural interface chooses the output that helps the user act, rather than simply mirroring the input.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend the speed on more reasoning
Complex task scheduling requires more than a fluid interface. Madra expected increased work on advanced assistants from model providers in the second half of 2024. He connects that prospect to evaluation: LLMs were commonly assessed through a single attempt, and he sees performance constraints as part of the reason. A system that has time for only one response has fewer opportunities to reconsider its work.
Faster inference creates room for repeated attempts and collaboration. Madra refers to unnamed papers in which repeated reasoning or multiple agents let models with fewer parameters compete with larger models. The talk does not identify the papers, models or evaluation conditions, so it does not establish a particular benchmark comparison. The mechanism is nevertheless clear: additional inference can be spent on solving the problem instead of merely returning the first answer faster.
He treats Apple AI’s on-device and off-device interaction as an early example of systems cooperating across compute locations, while anticipating more sophisticated collaboration. That leads into predictive analytics: instead of waiting for a person to ask a question or trigger an action, an agent could keep working in the background.
That continuous operation depends on an economic assumption as much as a latency assumption. Madra imagines compute cycles becoming nearly free, explicitly describing a future condition rather than the situation at the talk. The same constraint applies to context: a large context window does not make processing its contents free. Lower costs could make both persistent agents and richer context practical, allowing a system to notice changes before a user asks about them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Custom stories and continuous analysis
Creative tools offer a personal version of this expanded capacity. Madra enjoys asking an LLM to write a Seinfeld episode around modern events. What interests him is the model’s assignment of situations to characters: it can identify who would experience the awkward or comic part of a scenario. Extending that customization from written episodes into multimedia is the next possibility he raises.
Business analysis provides a more operational example. Before Groq acquired Definitive Intelligence, Madra’s team worked beyond text-to-SQL on Pioneer, an automated data-science agent developed with his colleague Rick. Pioneer was intended to work on a problem almost indefinitely. Its starting point was a business KPI and the data used to understand it.
Ordinarily, people compare incoming data with KPIs, investigate changes, and turn the analysis into spreadsheets or PowerPoint presentations for management. Pioneer’s proposed loop makes the investigation continuous:
- Define the KPI the business wants to understand.
- Examine new data as it arrives.
- Ask additional questions prompted by the findings.
- Investigate those questions and continue the analysis.
The distinguishing feature is the ongoing inquiry, rather than a single translation from a natural-language question into SQL.
Madra recalls applying Pioneer to a dataset of workers and their performance reviews. It surfaced a relationship involving age, the type of review received and subsequent productivity decline. He qualifies the recollection and invites Rick to correct it; the talk gives no ages, effect sizes or causal evidence. The anecdote illustrates the kind of unexpected correlation a continuing investigation might surface, not a validated rule for employment decisions.
The same idea extends to dynamic optimization. Drawing on experience at Ford after its acquisition of Autonomic, Madra points to vehicle production and shipping: sophisticated supply-chain software can still leave substantial inefficiencies. Faster, continuing analysis might help those systems adapt. He presents this as an opportunity for former Ford colleagues, not a deployed optimization result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spare compute for work that can wait
Not every useful agent needs an immediate response. In discussing edge and decentralized AI, Madra names hyperspace.ai as a project that makes unused GPU capacity available to others, drawing comparisons with SETI@home and Render. The historical domain should not be treated as current documentation for that network: today it presents an AI tools service, and its connection to the described project is unresolved.
The workload distinction matters more than the project name. Tasks without strict real-time requirements can use distributed spare capacity even when interactive tasks need predictable latency. Madra connects this possibility to improving throughput and latency in existing systems, then raises power consumption as another reason to consider distributing the work. He suggests a potentially useful arrangement, without quantifying energy savings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A convincing voice still needs authentication
Speed also matters on the defensive side. Madra describes a colleague receiving a phishing call from someone who sounded formal and possessed extensive personal information. AI, he argues, can help scammers construct deeper, more convincing narratives than the scripts associated with older call-center scams. That creates a need for protective systems operating on the recipient’s side of the interaction.
The colleague’s decisive test was to request a formal message through the HSBC app. The caller could not provide one. That shifted the interaction from judging the plausibility of a story to asking for confirmation through a trusted channel. As voice cloning improves and more personal information becomes available online, Madra expects protective assistance to need very low latency: it must help while the suspicious conversation is still happening.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tutoring around the learner’s interests
Education brings speed, cost and personalization together. Madra describes cheaper, more widely available tokens as a priority at Groq, then invokes Sal Khan’s discussion of personalized tutoring and the two-sigma result. The relevant historical finding in Bloom’s two-sigma paper is an average achievement difference of about two standard deviations between human tutoring with formative testing and corrective feedback and conventional classroom instruction. The cited studies involved probability and cartography in grades 4, 5 and 8 over eleven instructional periods across three weeks. That is narrower than a guaranteed improvement for every student, and it is not a measured result for AI tutors.
The application opportunity is to make individual attention affordable and adapt explanations to a learner’s interests. Madra describes a homeschooling service that frames arithmetic around unicorns and ponies for a child who likes them. Addition, subtraction and multiplication retain their mathematical content, but the surrounding story becomes personally engaging. His playful example combines three ponies and two unicorns in a multiplication prompt. The useful design principle is to customize the context in which a concept is taught, so the learner has a reason to stay with it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The cost of making software work together
The final application is less visible than a voice assistant or personalized tutor: enterprise integration. Madra argues that most enterprise software deployment and maintenance spending is tied to interconnectivity, interoperability and compatibility. He offers no spending breakdown, but identifies a familiar source of work: getting systems to cooperate and keeping them working together.
Fast, inexpensive AI could reduce that burden. The closing prospect is economic as well as technical: if more interpretation and coordination can happen cheaply enough, organizations may spend less effort maintaining the connections between their tools. Crossing the speed barrier would matter here because it changes which work software can afford to do continuously.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
An April 2024 account of Llama 3 availability on Groq, with contemporary throughput and latency figures.
The 1984 paper describing human tutoring results and the search for group instruction methods with comparable effectiveness.
A layered approach to combining model responses, with benchmark results, latency tradeoffs and links to an implementation.
Full transcript
Loading transcript…