AI Engineer World's Fair 2024
State Space Models for Realtime Multimodal Intelligence
Loading slides…
Read the talk
State Space Models for Real-Time Multimodal Intelligence
Voice, video and sensor streams keep arriving while a model works. Karan Goel explains how compressing that information into recurrent state changes the architecture of continuous inference.
From a talk by Karan Goel
When intelligence must keep up with a stream
A model can spend a long time reasoning through a difficult mathematics or physics problem. Incoming audio or sensor data presents a different problem: the stream continues while the model works. The system must keep interpreting information or generating a response as inputs arrive. This is the distinction between batch intelligence and streaming intelligence.
A familiar batch interaction is to call a model in the cloud, wait several seconds and receive an answer. Streaming applications repeatedly ask for responses at low latency. That distinction concerns the interaction pattern, not whether computation happens in a data center: a cloud service can also serve a stream. The requirement is to keep pace with an ongoing exchange rather than finish one isolated request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice, devices and robots impose practical constraints
Conversational voice makes responsiveness immediately useful. You should be able to speak to an assistant, have it understand you and ask it to perform tasks. An assistant that remains available on a phone or laptop adds another constraint: it must operate efficiently enough to stay available without consuming excessive power.
Generated game worlds extend the idea beyond conversation. Imagine playing a game whose world is generated as you interact with it, much as a GPU renders graphics in real time. Robotics brings these demands together in one device that must interpret its surroundings and act. Making intelligence faster and cheaper determines where these experiences can run.
The conversational examples are deliberately mundane: handling customer support, answering health-insurance questions or calling a vendor to arrange shipment pickup. These coordination tasks consume human time even when the underlying problem is straightforward. Automating them could leave people available for difficult or consequential cases, including an especially frustrated customer. A robot faces the corresponding physical challenge: ingest audio, video and sensor data together, then respond quickly enough for the interaction to remain useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build compression into the model
Karan Goel connects Cartesia’s work on real-time foundation models to a question from his doctoral research: can a model compress information as it arrives, making streaming part of its basic operation? State-space models, or SSMs, pursue this direction. Mamba is a more recent selective-state-space model in that research lineage; it is not a name for every SSM or every model Cartesia builds.
At Cartesia, the research question becomes how to improve these architectures and push their capabilities. The motivation is to explore designs beyond the standard transformer recipe. If the application requires continuous processing, compression and efficient memory use should influence the architecture from the beginning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A day of footage is not a day of useful events
Long context matters because much practical data arrives in long sequences. Text already concentrates information: a few sentences can communicate a great deal. Audio and video can contain much more redundant material. A security camera running for a day illustrates the difference between the volume of incoming data and the amount worth remembering.
The architectural question is what to retain. Keeping the entire history available lets a model consult it later. Compressing the stream instead asks the model to carry forward information useful for understanding and future decisions. The camera example motivates task-relevant memory, not a lossless recording hidden inside a smaller representation.
The talk estimates annual human sensory exposure at roughly one billion text tokens, ten billion audio tokens and one trillion video tokens, excluding other sensory inputs. These are back-of-the-envelope quantities without a specified tokenizer or sampling convention. Their purpose is to evoke the scale of simultaneous streams handled by a compact biological system, which continues operating as its energy intake varies.
For comparison, the talk places leading model contexts of the period at approximately ten million to one hundred million tokens. This is a historical order-of-magnitude assertion, not a comparison of named benchmarks. The larger ambition is a long-lived system that can understand information across extended periods. Human memories reaching back thirty years provide the motivating analogy: useful information remains available without consciously replaying every intervening experience.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What grows with context length
The computational problem becomes sharper as multimodal sequences grow. With model dimensions fixed, dense attention performs quadratic attention work over a full sequence.
It helps to distinguish three costs:
- Full-sequence attention: the aggregate attention work is quadratic in sequence length.
- Cached next-token attention: the new position attends over a growing prefix; it does not require recomputing every previous token’s representation.
- Retained history: the key/value cache grows as positions are added.
For fixed model dimensions and dense attention over the retained prefix, cached next-token attention work and KV-cache storage grow linearly with prefix length. Quadratic aggregate work does not mean every individual decoding step is quadratic.
More compute can support harder tasks, but it does not remove that dependence on sequence length. Audio and video intensify the problem because they can supply far more positions to process than a compact text exchange.
The talk considers hypothetical inference demand increases of a thousandfold or a hundred-thousandfold. These scenarios show why architectural costs matter beyond a single request: an expensive operation becomes harder to deploy everywhere when it must run continuously at enormous volume. Transformers have nevertheless provided a successful recipe for scaling models, so the aim is to improve efficiency while preserving capability.
For a fixed recurrent model, an SSM can process a sequence with work linear in its length while keeping recurrent state size independent of that length. Compression happens as information arrives, creating opportunities for lower memory use and lower-power execution. Actual speed and energy use still depend on model size, hardware and implementation; bounded recurrent state does not mean the entire system consumes constant resources.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retain the history or update a state
The talk makes the difference concrete with the prefix the quick brown fox and the next word jumped. An attention-based model consults representations of the preceding positions to produce the continuation. It then adds the new position to its context and repeats. With caching, earlier keys and values remain available while the new position contributes additional entries.
A recurrent SSM carries a different kind of memory forward. Each incoming token updates an internal state; subsequent processing uses that state without needing to retain the raw token sequence inside it. The talk’s compressed-file analogy describes a summary, not literal ZIP encoding or a guarantee that every earlier detail can be recovered.
The illustration isolates this memory choice with a toy scalar recurrence and inputs 1, 2 and 3. Its values are teaching choices, not Mamba or Cartesia parameters. The input strip remains visible so you can follow the example; the retained-memory panels distinguish the carried state from the growing schematic attention cache. They compare what remains available, not execution speed.
One state, updated as inputs arrive
Follow a toy recurrence through 1, 2 and 3. Its scalar state illustrates compression, not Cartesia's model parameters.
- x11next input
- x22waiting
- x33waiting
Start with h0 = 0. Process the first input to update it.
Recurrence retains
One scalar. The next update uses this value.
Attention cache retains
Earlier keys and values stay as positions are added.
This repeated update is recurrence: the previous state participates in constructing the next one. The talk describes renewed interest in that approach as an alternative to retaining an expanding history for later consultation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compression changes which details remain available
Looking ahead from the talk, the forecast is that multimodal workloads will make recurrent architectures more relevant over the following three to five years. Efficiency alone, however, does not answer the question of quality. Compression gives up direct access to some past detail in exchange for a more compact representation.
Return to the same twenty-four-hour camera stream. Summarizing it as it arrives may make relevant events easier to use than repeatedly consulting the entire recording. But whether that helps depends on what the summary preserves and what questions come later. The talk proposes that compression is especially helpful for long contexts, where redundancy is substantial, and less helpful for short ones. This is a task-dependent intuition illustrated by the camera example, rather than a reported camera benchmark or a guarantee across tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice generation brings the architecture back to latency
The closing application is Cartesia’s recently released text-to-speech model, the original Sonic. Voice generation is an initial step toward bringing more multimodal data into a single model, with SSMs supporting training and inference. The reported benefit is responsiveness: the playground is described as returning generated voice almost immediately from a data center.
Work on running these models on Macs and other devices aims to bring that experience to local hardware efficiently and at low power. That remains distinct from the cloud playground described in the talk. Time expires before a live demonstration, leaving the audience with the invitation to try play.cartesia.ai and provide feedback. No on-device demonstration is completed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Selective state spaces and the original linear-time sequence-modeling argument.
Further reading
Goel's coauthored precursor research on efficiently modeling long sequences.
- The original Sonic launchArticle
The contemporaneous voice-model release behind the talk's closing invitation to try Cartesia.
Full transcript
Loading transcript…