AI Engineer World's Fair 2024

Breaking AI’s 1 Gigahertz Barrier

Sunny Madra· Groq20:11

Loading slides…

Read the talk

Breaking AI’s 1 Gigahertz Barrier

Faster inference can change more than response time: it can make research, multimodal interfaces and continuously running agents practical parts of everyday computing.

From a talk by Sunny Madra

When speed changes the computer

What changes when a computer crosses a speed threshold? Sunny Madra opens with Intel’s 1999 gigahertz milestone, roughly twenty-five years before this talk. The milestone was a technology demonstration, not a commercial processor launch. The slide’s Intel news release gives the analogy a concrete starting point: a memorable number that invites a larger question about what faster hardware makes possible.

Slide titled “25 years ago” with an Intel News Release headed “Intel Breaks Gigahertz Barrier.”Open full source frame
Intel’s gigahertz milestone introduces the talk’s speed theme.

Madra connects that milestone to the later shift toward multicore processors. In his historical comparison, microprocessor speeds improved by three orders of magnitude in about two decades. He then asks what a comparable trajectory would mean for LLMs, invoking Jensen Huang’s view that AI innovation is moving beyond the Moore’s-law curve. The useful question is architectural: when does a faster component change how the whole system is designed?

For a nearer-term example, Madra reports that Groq increased the speed of Llama 3 8B by over 50% between April and June 2024. Groq’s April launch account provides contemporary context, but the talk supplies neither absolute throughput endpoints nor workload conditions for this improvement. Faster inference matters because it expands the amount of work an application can perform before the user has to wait.

0:000:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

A pizza trip instead of a hundred tabs

Input processing is one half of that opportunity. Madra sketches an input rate of 10,000 tokens per second and mentions roughly one-third of a second of processing. He does not specify the input length, so those figures do not establish an end-to-end task time. His point is that a model can ingest and analyze information much faster than a person can read through it.

The concrete example is Globe.Engineer. Give it a task such as planning a trip to New York to try the best pizza, and it breaks the request into several research streams. Madra describes it working against the live Internet: finding flights, taxi options, hotels and food, then assembling an itinerary. The displayed result gathers those branches into one interface, with a navigation tree, travel text and flight cards.

Slide with a Globe interface screenshot showing a navigation tree, travel text, and flight cards with maps and an airplane image.Open full source frame
A travel-planning result on the “State of the Art Today” slide.

Madra estimates that the Globe.Engineer trip-planning example completes in perhaps less than five seconds. Doing the same research himself might involve tens or even hundreds of browser tabs, each representing a separate line of inquiry. The interface consolidates that work because the model can both process incoming material quickly and generate the resulting plan quickly. Faster output alone would leave the research bottleneck intact; the example depends on both sides of token processing.

2:052:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:05 · section reference included

Industrializing digital work

Once an application can absorb those research streams, the LLM begins to look less like an isolated feature and more like a computing core—or even an operating system. That would change how people program computers and what they expect computers to analyze. Madra frames this as an expansion of human capabilities, setting aside the question of AGI to focus on a different computing paradigm.

His analogy is the move from craft production to industrial production. He groups carmaking, farming and clothing together: a few cars made in a day versus hundreds or thousands, food produced for a village versus national distribution, and a sweater made in a day or a week versus clothing manufactured at scale. These are broad illustrations of increasing productive capacity, rather than a precise chronology of the Industrial Revolution.

For computing, Madra borrows a framework from Paul Maritz, whose career included Microsoft, VMware and Pivotal, where the two met. Earlier eras changed the representation, connectivity or location of work:

Computing eraWhat changedFamiliar expression
DigitizationPaper processes became digitalFiles, folders, inbox, outbox
InternetDigital processes became connectedNetworked information and services
Cloud and mobileWork moved to new infrastructure and devicesCloud scale and phone access
AIDigital production can become automated at scaleGenerating and analyzing artifacts

The paper metaphors survive in operating systems because the first transformation largely reproduced existing office processes. Connectivity followed, then what Madra describes as roughly fifteen years of cloud and mobile form-factor changes. AI introduces the possibility of scaling the production of digital work itself.

Presentation artwork makes the distinction tangible. Madra’s illustrative comparison contrasts a designer making one or two artifacts a day, eighteen to twenty-four months earlier, with Midjourney producing a thousand in a minute. Those are his scale illustrations, not established production measurements. The shift he is pointing to is from commissioning individual artifacts to generating many candidates on demand.

3:494:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:49 · section reference included

An LLM at the core

How fast would inference need to become before it could occupy that central role? Madra proposes a hypothetical decision time of 0.1 milliseconds, equivalent to 10,000 sequential decisions per second:

tdecision=0.1 ms=10−4 srdecision=1tdecision=10,000 decisions/s.\begin{aligned} t_{\text{decision}} &= 0.1\,\text{ms} = 10^{-4}\,\text{s} \\ r_{\text{decision}} &= \frac{1}{t_{\text{decision}}} = 10{,}000\,\text{decisions/s}. \end{aligned}

This is a thought experiment about complex decisions, not a demonstrated inference result. At that scale, repeated model calls could become part of ordinary software execution rather than conspicuous pauses in an interaction.

The resulting change would reach how software is built, how it runs and how it scales. Present-day latency makes it easy to assume that an LLM must remain outside the computer’s inner loop. Imagining CPU-like speed growth removes that assumption. Madra credits Andrej Karpathy for a diagram placing an LLM at the center of connections to video, audio, browsers, other LLMs, code interpreters and file systems. The model becomes a coordinator across capabilities that currently have separate interfaces.

6:476:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:47 · section reference included

Personal interfaces need more than fast speech

The first visible change would be a move from responses arriving around reading speed to decisions that feel instantaneous. Returning to the Globe example, Madra contrasts a few seconds of automated research with an afternoon, an evening or several evenings of personal effort. The benefit is not just reading the same answer sooner; it is completing a larger task within the time previously spent waiting for a small part of it.

Personalization adds another requirement. OpenAI’s early memory features provide Madra’s starting point: a system can remember the names of a user’s pets, children or spouse. He cites Bill Gurley and Brad Gerstner’s podcast discussions of personalization as a major frontier. For those details to support a seamless experience, the system must retrieve and use them quickly enough that personal context does not become another source of friction.

The interface then moves beyond keyboards, pointing devices and mobile touch. At the time of the talk, GPT-4o voice demonstrations showed Madra the possibilities of mixed interaction, although he did not regard the full experience as achieved. His shorthand is XRX: any type of input, reasoning, and any type of output. Input and output need not use the same modality.

Consider booking a haircut. A user can ask aloud which appointments are available, but a spoken list is difficult to hold in memory:

Part of the interactionUseful modalityContent
RequestVoiceAsk for available haircut appointments
ResponseText9 AM, 11 AM, 3:30, 5:30

Displaying the options lets the user compare them without remembering a sequence of spoken times. No appointment has been booked at this point; the system has only presented availability. A natural interface chooses the output that helps the user act, rather than simply mirroring the input.

8:028:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:02 · section reference included

Spend the speed on more reasoning

Complex task scheduling requires more than a fluid interface. Madra expected increased work on advanced assistants from model providers in the second half of 2024. He connects that prospect to evaluation: LLMs were commonly assessed through a single attempt, and he sees performance constraints as part of the reason. A system that has time for only one response has fewer opportunities to reconsider its work.

Faster inference creates room for repeated attempts and collaboration. Madra refers to unnamed papers in which repeated reasoning or multiple agents let models with fewer parameters compete with larger models. The talk does not identify the papers, models or evaluation conditions, so it does not establish a particular benchmark comparison. The mechanism is nevertheless clear: additional inference can be spent on solving the problem instead of merely returning the first answer faster.

He treats Apple AI’s on-device and off-device interaction as an early example of systems cooperating across compute locations, while anticipating more sophisticated collaboration. That leads into predictive analytics: instead of waiting for a person to ask a question or trigger an action, an agent could keep working in the background.

That continuous operation depends on an economic assumption as much as a latency assumption. Madra imagines compute cycles becoming nearly free, explicitly describing a future condition rather than the situation at the talk. The same constraint applies to context: a large context window does not make processing its contents free. Lower costs could make both persistent agents and richer context practical, allowing a system to notice changes before a user asks about them.

10:3410:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:34 · section reference included

Custom stories and continuous analysis

Creative tools offer a personal version of this expanded capacity. Madra enjoys asking an LLM to write a Seinfeld episode around modern events. What interests him is the model’s assignment of situations to characters: it can identify who would experience the awkward or comic part of a scenario. Extending that customization from written episodes into multimedia is the next possibility he raises.

Business analysis provides a more operational example. Before Groq acquired Definitive Intelligence, Madra’s team worked beyond text-to-SQL on Pioneer, an automated data-science agent developed with his colleague Rick. Pioneer was intended to work on a problem almost indefinitely. Its starting point was a business KPI and the data used to understand it.

Ordinarily, people compare incoming data with KPIs, investigate changes, and turn the analysis into spreadsheets or PowerPoint presentations for management. Pioneer’s proposed loop makes the investigation continuous:

  1. Define the KPI the business wants to understand.
  2. Examine new data as it arrives.
  3. Ask additional questions prompted by the findings.
  4. Investigate those questions and continue the analysis.

The distinguishing feature is the ongoing inquiry, rather than a single translation from a natural-language question into SQL.

Madra recalls applying Pioneer to a dataset of workers and their performance reviews. It surfaced a relationship involving age, the type of review received and subsequent productivity decline. He qualifies the recollection and invites Rick to correct it; the talk gives no ages, effect sizes or causal evidence. The anecdote illustrates the kind of unexpected correlation a continuing investigation might surface, not a validated rule for employment decisions.

The same idea extends to dynamic optimization. Drawing on experience at Ford after its acquisition of Autonomic, Madra points to vehicle production and shipping: sophisticated supply-chain software can still leave substantial inefficiencies. Faster, continuing analysis might help those systems adapt. He presents this as an opportunity for former Ford colleagues, not a deployed optimization result.

12:3612:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:36 · section reference included

Spare compute for work that can wait

Not every useful agent needs an immediate response. In discussing edge and decentralized AI, Madra names hyperspace.ai as a project that makes unused GPU capacity available to others, drawing comparisons with SETI@home and Render. The historical domain should not be treated as current documentation for that network: today it presents an AI tools service, and its connection to the described project is unresolved.

The workload distinction matters more than the project name. Tasks without strict real-time requirements can use distributed spare capacity even when interactive tasks need predictable latency. Madra connects this possibility to improving throughput and latency in existing systems, then raises power consumption as another reason to consider distributing the work. He suggests a potentially useful arrangement, without quantifying energy savings.

15:4916:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:49 · section reference included

A convincing voice still needs authentication

Speed also matters on the defensive side. Madra describes a colleague receiving a phishing call from someone who sounded formal and possessed extensive personal information. AI, he argues, can help scammers construct deeper, more convincing narratives than the scripts associated with older call-center scams. That creates a need for protective systems operating on the recipient’s side of the interaction.

The colleague’s decisive test was to request a formal message through the HSBC app. The caller could not provide one. That shifted the interaction from judging the plausibility of a story to asking for confirmation through a trusted channel. As voice cloning improves and more personal information becomes available online, Madra expects protective assistance to need very low latency: it must help while the suspicious conversation is still happening.

16:3917:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:39 · section reference included

Tutoring around the learner’s interests

Education brings speed, cost and personalization together. Madra describes cheaper, more widely available tokens as a priority at Groq, then invokes Sal Khan’s discussion of personalized tutoring and the two-sigma result. The relevant historical finding in Bloom’s two-sigma paper is an average achievement difference of about two standard deviations between human tutoring with formative testing and corrective feedback and conventional classroom instruction. The cited studies involved probability and cartography in grades 4, 5 and 8 over eleven instructional periods across three weeks. That is narrower than a guaranteed improvement for every student, and it is not a measured result for AI tutors.

The application opportunity is to make individual attention affordable and adapt explanations to a learner’s interests. Madra describes a homeschooling service that frames arithmetic around unicorns and ponies for a child who likes them. Addition, subtraction and multiplication retain their mathematical content, but the surrounding story becomes personally engaging. His playful example combines three ponies and two unicorns in a multiplication prompt. The useful design principle is to customize the context in which a concept is taught, so the learner has a reason to stay with it.

18:0318:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:03 · section reference included

The cost of making software work together

The final application is less visible than a voice assistant or personalized tutor: enterprise integration. Madra argues that most enterprise software deployment and maintenance spending is tied to interconnectivity, interoperability and compatibility. He offers no spending breakdown, but identifies a familiar source of work: getting systems to cooperate and keeping them working together.

Fast, inexpensive AI could reduce that burden. The closing prospect is economic as well as technical: if more interpretation and coordination can happen cheaply enough, organizations may spend less effort maintaining the connections between their tools. Crossing the speed barrier would matter here because it changes which work software can afford to do continuously.

19:2119:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:21 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…