# State Space Models for Realtime Multimodal Intelligence

Karan Goel, Cartesia

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/U9DPRZ0lSIQ-state-space-models-realtime-multimodal) · [Video](https://www.youtube.com/watch?v=U9DPRZ0lSIQ)

## State Space Models for Real-Time Multimodal Intelligence

Voice, video and sensor streams keep arriving while a model works. Karan Goel explains how compressing that information into recurrent state changes the architecture of continuous inference.

### When intelligence must keep up with a stream

[0:00](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=0s)

A model can spend a long time reasoning through a difficult mathematics or physics problem. Incoming audio or sensor data presents a different problem: the stream continues while the model works. The system must keep interpreting information or generating a response as inputs arrive. This is the distinction between batch intelligence and streaming intelligence.

A familiar batch interaction is to call a model in the cloud, wait several seconds and receive an answer. Streaming applications repeatedly ask for responses at low latency. That distinction concerns the interaction pattern, not whether computation happens in a data center: a cloud service can also serve a stream. The requirement is to keep pace with an ongoing exchange rather than finish one isolated request.

### Voice, devices and robots impose practical constraints

[1:28](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=88s)

Conversational voice makes responsiveness immediately useful. You should be able to speak to an assistant, have it understand you and ask it to perform tasks. An assistant that remains available on a phone or laptop adds another constraint: it must operate efficiently enough to stay available without consuming excessive power.

Generated game worlds extend the idea beyond conversation. Imagine playing a game whose world is generated as you interact with it, much as a GPU renders graphics in real time. Robotics brings these demands together in one device that must interpret its surroundings and act. Making intelligence faster and cheaper determines where these experiences can run.

The conversational examples are deliberately mundane: handling customer support, answering health-insurance questions or calling a vendor to arrange shipment pickup. These coordination tasks consume human time even when the underlying problem is straightforward. Automating them could leave people available for difficult or consequential cases, including an especially frustrated customer. A robot faces the corresponding physical challenge: ingest audio, video and sensor data together, then respond quickly enough for the interaction to remain useful.

### Build compression into the model

[3:56](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=236s)

Karan Goel connects Cartesia’s work on real-time foundation models to a question from his doctoral research: can a model compress information as it arrives, making streaming part of its basic operation? State-space models, or SSMs, pursue this direction. **[Mamba](https://arxiv.org/abs/2312.00752)** is a more recent selective-state-space model in that research lineage; it is not a name for every SSM or every model Cartesia builds.

At Cartesia, the research question becomes how to improve these architectures and push their capabilities. The motivation is to explore designs beyond the standard transformer recipe. If the application requires continuous processing, compression and efficient memory use should influence the architecture from the beginning.

### A day of footage is not a day of useful events

[5:26](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=326s)

Long context matters because much practical data arrives in long sequences. Text already concentrates information: a few sentences can communicate a great deal. Audio and video can contain much more redundant material. A security camera running for a day illustrates the difference between the volume of incoming data and the amount worth remembering.

The architectural question is what to retain. Keeping the entire history available lets a model consult it later. Compressing the stream instead asks the model to carry forward information useful for understanding and future decisions. The camera example motivates task-relevant memory, not a lossless recording hidden inside a smaller representation.

The talk estimates annual human sensory exposure at roughly one billion text tokens, ten billion audio tokens and one trillion video tokens, excluding other sensory inputs. These are back-of-the-envelope quantities without a specified tokenizer or sampling convention. Their purpose is to evoke the scale of simultaneous streams handled by a compact biological system, which continues operating as its energy intake varies.

For comparison, the talk places leading model contexts of the period at approximately ten million to one hundred million tokens. This is a historical order-of-magnitude assertion, not a comparison of named benchmarks. The larger ambition is a long-lived system that can understand information across extended periods. Human memories reaching back thirty years provide the motivating analogy: useful information remains available without consciously replaying every intervening experience.

![Goel’s slide puts books, audio, biology and interactive systems side by side as long-sequence workloads.](https://ai.engineer/api/talk-essay-media/734b53a99c2f199b963050bc2b84ea4dae90d8f438663778d8f2cb009ac58271)

Goel’s slide puts books, audio, biology and interactive systems side by side as long-sequence workloads.

### What grows with context length

[8:13](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=493s)

The computational problem becomes sharper as multimodal sequences grow. With model dimensions fixed, dense attention performs quadratic attention work over a full sequence.

It helps to distinguish three costs:

- **Full-sequence attention:** the aggregate attention work is quadratic in sequence length.
- **Cached next-token attention:** the new position attends over a growing prefix; it does not require recomputing every previous token’s representation.
- **Retained history:** the key/value cache grows as positions are added.

For fixed model dimensions and dense attention over the retained prefix, cached next-token attention work and KV-cache storage grow linearly with prefix length. Quadratic aggregate work does not mean every individual decoding step is quadratic.

More compute can support harder tasks, but it does not remove that dependence on sequence length. Audio and video intensify the problem because they can supply far more positions to process than a compact text exchange.

The talk considers hypothetical inference demand increases of a thousandfold or a hundred-thousandfold. These scenarios show why architectural costs matter beyond a single request: an expensive operation becomes harder to deploy everywhere when it must run continuously at enormous volume. Transformers have nevertheless provided a successful recipe for scaling models, so the aim is to improve efficiency while preserving capability.

For a fixed recurrent model, an SSM can process a sequence with work linear in its length while keeping recurrent state size independent of that length. Compression happens as information arrives, creating opportunities for lower memory use and lower-power execution. Actual speed and energy use still depend on model size, hardware and implementation; bounded recurrent state does not mean the entire system consumes constant resources.

### Retain the history or update a state

[10:45](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=645s)

The talk makes the difference concrete with the prefix `the quick brown fox` and the next word `jumped`. An attention-based model consults representations of the preceding positions to produce the continuation. It then adds the new position to its context and repeats. With caching, earlier keys and values remain available while the new position contributes additional entries.

A recurrent SSM carries a different kind of memory forward. Each incoming token updates an internal state; subsequent processing uses that state without needing to retain the raw token sequence inside it. The talk’s compressed-file analogy describes a summary, not literal ZIP encoding or a guarantee that every earlier detail can be recovered.

![The talk’s fox example contrasts attention to past tokens with an updated, compressed recurrent state.](https://ai.engineer/api/talk-essay-media/452e9f2930d1db3bd0f848bb96c4894e35ec62107f0b1da28cdfaab83d4674b7)

The talk’s fox example contrasts attention to past tokens with an updated, compressed recurrent state.

The illustration isolates this memory choice with a toy scalar recurrence and inputs `1`, `2` and `3`. Its values are teaching choices, not Mamba or Cartesia parameters. The input strip remains visible so you can follow the example; the retained-memory panels distinguish the carried state from the growing schematic attention cache. They compare what remains available, not execution speed.

Updated recurrent state versus retained token history
A recurrence keeps a summary, not a lossless copy of the prefix. The cache sketch compares retention, not speed.
- Previous recurrent state — The summary carried forward from the preceding input.
- New stream input — A newly arriving input participates in the next state update.
- Updated state retained — The recurrent summary is updated instead of retaining the entire raw input sequence.
- Past-position state retained — The attention comparison retains information associated with earlier positions as the sequence grows.
- Previous recurrent state → Updated state retained (Update summary)
- New stream input → Updated state retained (Incorporate input)
- New stream input → Past-position state retained (Retain another position)

This repeated update is recurrence: the previous state participates in constructing the next one. The talk describes renewed interest in that approach as an alternative to retaining an expanding history for later consultation.

### Compression changes which details remain available

[12:01](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=721s)

Looking ahead from the talk, the forecast is that multimodal workloads will make recurrent architectures more relevant over the following three to five years. Efficiency alone, however, does not answer the question of quality. Compression gives up direct access to some past detail in exchange for a more compact representation.

Return to the same twenty-four-hour camera stream. Summarizing it as it arrives may make relevant events easier to use than repeatedly consulting the entire recording. But whether that helps depends on what the summary preserves and what questions come later. The talk proposes that compression is especially helpful for long contexts, where redundancy is substantial, and less helpful for short ones. This is a task-dependent intuition illustrated by the camera example, rather than a reported camera benchmark or a guarantee across tasks.

### Voice generation brings the architecture back to latency

[13:03](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=783s)

The closing application is Cartesia’s recently released text-to-speech model, the original **[Sonic](https://www.cartesia.ai/blog/sonic)**. Voice generation is an initial step toward bringing more multimodal data into a single model, with SSMs supporting training and inference. The reported benefit is responsiveness: the playground is described as returning generated voice almost immediately from a data center.

Work on running these models on Macs and other devices aims to bring that experience to local hardware efficiently and at low power. That remains distinct from the cloud playground described in the talk. Time expires before a live demonstration, leaving the audience with the invitation to try `play.cartesia.ai` and provide feedback. No on-device demonstration is completed.

## Resources

- [Mamba](https://arxiv.org/abs/2312.00752) — Selective state spaces and the original linear-time sequence-modeling argument.
- [Structured state spaces with S4](https://arxiv.org/abs/2111.00396) — Goel's coauthored precursor research on efficiently modeling long sequences.
- [The original Sonic launch](https://www.cartesia.ai/blog/sonic) — The contemporaneous voice-model release behind the talk's closing invitation to try Cartesia.

## Slides

### Batch and real-time streaming intelligence

The speaker contrasts long-running reasoning on hard problems with real-time systems that generate video or audio, or process sensor streams.

[0:18](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=18s) [Foundation models are described as serving batch workloads, while interactive applications require real-time streaming intelligence; a Road Runner image illustrates speed.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ca6ef36afc8de9beb3853bf2853ff467d6c1485daedafd40c2c1d2c2bf41200.webp)

Foundation models are built for batch workloads that require slow intelligence.

Interactive applications require real-time streaming intelligence.

[illegible]artesia
June 2024

Interface text:
AIE
Microsoft
smol ai
AWS
World's Fair
DATASTAX
Galileo

### Applications of real-time intelligence

The speaker envisions low-power, always-on assistants and games generated in real time, with robotics combining these capabilities on a single device. Many applications, he predicts, will run on-device.

[1:27](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=87s) [Slide lists conversational interfaces, always-on assistants, world generation in games and VR, AI across devices, and robotics, noting that many applications will run on-device.](https://assets.ai.engineer/sites/aiecode2025/sha256/6233977426647e8672984c5d99659fbd1772de73426f8cb4aa297470c566b772.webp)

Real-time intelligence will transform a wide range of interactive applications.

Conversational Interfaces
Always-on Assistants
World Generation in Games & VR
AI Across Devices
Robotics

Many of these applications will run on-device.

artesia
June 2024

Interface text:
AIE
Microsoft
smol ai
aws
World’s Fair
DATASTAX
Galileo

[2:41](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=161s) [Overview of conversational interfaces, always-on assistants, game and VR world generation, AI across devices, and robotics; many applications will run on-device.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8b77798be0b1d2553f7d9ca187f4f0202bf11eca74a7dac1aaadbddca0d50b9.webp)

Real-time intelligence will transform a wide range of interactive applications.

Conversational Interfaces
Always-on Assistants
World Generation in Games & VR
AI Across Devices
Robotics

Many of these applications will run on-device.

artesia
June 2024

Interface text:
AIE
aws
Microsoft
World’s Fair
DATASTAX
Galileo

### Example 1: Conversational interfaces

Real-time agents could handle support, insurance questions and shipment coordination, freeing people to tackle harder customer problems.

[2:46](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=166s) [Examples of AI agents providing customer support, answering health insurance inquiries, and calling a supplier or vendor to pick up a shipment.](https://assets.ai.engineer/sites/aiecode2025/sha256/5416e6d3bfa8f458833bc1e0c0dfad0f4eabe35147e04b439d9cd140d620c532.webp)

Example 1

Conversational Interfaces

Real-time intelligence will power AI agents to
• provide customer support for a problem
• answer inquiries about health insurance
• call supplier / vendor to pick up shipment
• ...

artesia
June 2024

Interface text:
AIE
Microsoft
AI Engineer World's Fair
aws
Google Cloud
neo4j
DATASTAX
Galileo

### Example 2: Robotics

Real-time intelligence could let robots ingest audio, video and sensor data, respond instantly, and output actions and information.

[3:30](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=210s) [Real-time intelligence enables robots to ingest audio, video and sensor data, respond instantly, and output actions and communicate information.](https://assets.ai.engineer/sites/aiecode2025/sha256/7672c9093b6bf8a817217aa20560d8e220e4318f297de4c616a8def9b89be756.webp)

Example 2

Robotics

Real-time intelligence will power robots to
• ingest audio, video and sensor data
• respond instantly to new information
• output action and communicate information
• ...

[illegible]artesia
June 2024

Interface text:
AIE
Microsoft
aws
AI Engineer World's Fair
DATASTAX
Galileo

### Cartesia’s founding team

Cartesia’s team is building real-time models on its Stanford work in state space technology. The speaker describes a goal of compressing incoming information to support streaming systems.

[4:01](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=241s) [Cartesia introduces its founding team and its work on real-time state space models.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a370e1937e86936739c3ae6fe5a334114ca72b3c65f873f8d1f60e9484a7f7d.webp)

Cartesia is building real-time foundation models.

Karan Goel
Stanford

Prof. Albert Gu
Carnegie Mellon Univ.

Brandon Yang
Stanford, Google

Arjun Desai
Stanford, Apple

Prof. Chris Ré
Stanford

Our founding team laid the foundations for efficient, long-context real-time models with state space model technology at Stanford.

We’re building an amazing team to develop this technology.

AI Engineer World’s Fair | Cartesia
June 2024

### Efficiently modeling long context

Practical data can form very long sequences: one minute of audio has about 1M samples at 16 kHz. The speaker argues that compression helps retain what is necessary to remember or understand.

[5:28](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=328s) [Examples of long sequences in text, audio and biology, alongside the need for long memory in robotics and agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/88b5dca184a74acce6ada38ac5af38fdca800f25cd57b67b203797f1d4c893a4.webp)

Efficiently modeling long context is a huge problem

Practical data comes as long sequences

Text (books are 100k tokens),
Audio (1 minute ~1M samples at 16khz),
Biology (Chromosome ~ 250M, Full DNA ~ billions)
Robotics, conversational systems, agents all need long memory!

*Adapted from Chris’s NeurIPS slides.
[illegible]artesia
June 2024

Interface text:
AIE
Microsoft
AWS
AI Engineer World’s Fair
Google Cloud
DATASTAX
Galileo
smol ai

### The challenge of multimodal AI

The speaker estimates that humans process roughly 1B text, 10B audio and 1T video tokens per year. The June 2024 slide compares this with O(10)M tokens processed by the best models in a datacenter.

[6:31](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=391s) [Slide compares estimated annual human text, audio and video processing with model context capacity.](https://assets.ai.engineer/sites/aiecode2025/sha256/23883208b1a118156ac8e19796be84c42b1c9a5d1e2142835da552628dd5d14c.webp)

The future of multimodal AI is still challenging

Humans are very good at processing multimodal information.

In 1 year we process and understand
• O(1)B text tokens
• O(10)B audio tokens
• O(1)T video tokens

The best models today process O(10)M tokens in a datacenter.

[illegible]artesia
June 2024

Interface text: AIE; Microsoft; AI Engineer World’s Fair; aws; Google Cloud; DATASTAX; Galileo

### The big problem with models today

The slide links Transformers’ quadratic context scaling to slow inference and poor scaling for large multimodal contexts, motivating smaller, faster architectures.

[9:25](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=565s) [Transformer diagram and limitations of quadratic context scaling, followed by a call for small, fast, capable models.](https://assets.ai.engineer/sites/aiecode2025/sha256/224430ddaf7a7126d951b4d65ee3b6b584ee5c22be52b7d1dc411bbe43642812.webp)

The big problem with models today

Add & Norm
Feed Forward
Add & Norm
Multi-Head Attention
V  K  Q

Foundation models today are built on Transformers & optimized for the data-center.

Quadratic scaling in context length.
- lots of parameters
- slow inference for large contexts
- scale poorly with huge multimodal context

We need new architectures that enable small, fast and capable models.

AI Engineer World’s Fair | Cartesia
June 2024

### We need new model architectures

The proposed architectures compress incoming information and scale more linearly with context length, aiming for low-power operation, low memory use and larger contexts.

[9:47](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=587s) [State-space diagram and equations beside proposed benefits of linear context scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2ec76db020745e83339afb3e33b37d3d89345402f79ec961300d6971dd6ee26.webp)

We need new model architectures

Diagram labels: u, x, y, A
ẋ = Ax + Bu
y = Cx + Du

New fundamentally efficient architectures will enable smaller, faster & more capable models.

Linear scaling in context length.
- low power on-device implementation
- constant toks/s & memory for any context
- scale to massive multimodal contexts
- high throughput in the data center

AI Engineer World’s Fair | Cartesia
June 2024

### State space models (SSMs)

The team presents SSMs as stateful models with linear scaling in sequence length. The equations show how input affects the evolving state and output.

[10:19](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=619s) [Introduction to state space models, with a research citation, scaling claims, equations, and a quotation.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc9f6f3b1ad891f7c8ee7fe6bc5713af08a3d25a440ea6738e5e4fefb41b8695.webp)

State space models (SSMs)

An entirely new foundation model architecture and stack developed by our team.

Efficiently Modeling Sequences with Structured State Spaces.
Albert Gu, Karan Goel, Chris Re, ICLR 2022.
Best paper honorable mention.

State Space Models (SSMs) are stateful models with linear scaling in sequence length.
Enjoy long context, low memory footprint and fast inference.

x′(t) = Ax(t) + Bu(t)
y(t) = Cx(t) + Du(t)

“The work around SSMs ... probably is the next transformer”
- Jensen Huang

[illegible]artesia
June 2024

Interface text: AIE; Microsoft; smol ai

[10:29](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=629s) [Overview of state space models, their linear scaling, state equations, and a quotation attributed to Jensen Huang.](https://assets.ai.engineer/sites/aiecode2025/sha256/f823ff93f161eb92aecd5b24d02e466920f3ac59ebf3c1ebbed8b17e5a698b07.webp)

State space models (SSMs)

An entirely new foundation model architecture and stack developed by our team.

Efficiently Modeling Sequences with Structured State Spaces.
Albert Gu, Karan Goel, Chris Re, ICLR 2022.
Best paper honorable mention.

State Space Models (SSMs) are stateful models with linear scaling in sequence length.
Enjoy long context, low memory footprint and fast inference.

x′(t) = Ax(t) + Bu(t)
y(t) = Cx(t) + Du(t)

“The work around SSMs ... probably is the next transformer”
- Jensen Huang

[illegible]artesia
June 2024

Interface text: AIE

### Working intuition

To generate “jumped,” a transformer attends to the preceding tokens. An SSM instead updates a compressed state as tokens stream in, then uses that state for future generation.

[10:46](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=646s) [Diagrams contrast transformer attention to past tokens with an SSM that compresses tokens into a state.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb883c942f376a5976aee9e27b1545be62cb7206aeec24f077fa8047165dd547.webp)

Working intuition

Transformers generate quadratically by attending to every past token

The | quick | brown | fox | jumped | over

In SSMs, tokens stream in, update the model’s state and are discarded

The quick brown fox → [state] → jumped | over | the | lazy | dog

State is highly compressed, enabling generation in linear time

[illegible]artesia
June 2024

Interface text: AIE, Microsoft, smol ai, AWS, World’s Fair, Google Cloud, DataStax, Galileo

[11:14](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=674s) [Diagrams compare attention to past tokens with streaming tokens through a compressed SSM state.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d117da56c1caec107fdef87c7e36e160cddcd72e9c68ad72daaa650515adbd4.webp)

Working intuition

Transformers generate quadratically by attending to every past token

The | quick | brown | fox | jumped | over

In SSMs, tokens stream in, update the model’s state and are discarded

The quick brown fox → [state] → jumped | over | the | lazy | dog

State is highly compressed, enabling generation in linear time

[illegible]artesia
June 2024

Interface text: AIE, AWS, Microsoft, World’s Fair, Google Cloud, DataStax, Galileo

### SSM development and adoption

A timeline traces SSM work from HiPPO in 2020 through Mamba to multiple offshoots. The speaker says adoption has picked up in recent months.

[11:43](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=703s) [A timeline runs from HiPPO and S4 through Mamba and multiple offshoots.](https://assets.ai.engineer/sites/aiecode2025/sha256/c52e078a5a96c51a697211e4bad5e8d4c3a7bc43e69f9898f0d863451d08068a.webp)

SSMs are starting to be adopted in the last few months

LRA

S4ND
Kernel K(t)
C
[1.1, −.2, 2.9, 0.5]
1/Δ
S4
S4

U-Mamba
Caduceus
MambaDNA
Cobra
Jamba
MambaByte
MoE-Mamba
...

Academic work from us

Timeline: HiPPO, LSSL, S4, S4D, S4ND, Hyena, Mamba, Multiple offshoots
2020 — Today

AI Engineer World’s Fair | Cartesia
*Images from (HiPPO, S4, S4ND, Mamba)
June 2024

### SSM scaling laws

On The Pile at sequence length 8192, perplexity falls as FLOPs increase across the plotted models. Mamba tracks among the lowest curves alongside Transformer++.

[12:17](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=737s) [A log-scale chart compares perplexity against FLOPs for Mamba and several other models.](https://assets.ai.engineer/sites/aiecode2025/sha256/62adb11fd16f6f056384cd80823d6249891d912de02f75b40eeec2e95aacb5d1.webp)

SSMs have the right scaling laws – don’t sacrifice quality

Scaling Laws on The Pile (Sequence Length 8192)

Y-axis: Perplexity (log scale)
Y-axis ticks: 2 × 10¹, 10¹, 6 × 10⁰
X-axis: FLOPs (log scale)
X-axis ticks: 10¹⁹, 10²⁰

Legend: Hyena, RWKV, Transformer, RetNet, H3++, Transformer++, Mamba

*From the Mamba paper by Albert & Tri.

AI Engineer World’s Fair | Cartesia
June 2024

### Cartesia Sonic voice generation

Cartesia presents Sonic as an SSM-based text-to-speech model with 135 ms model latency and zero-shot voice cloning from 10 seconds of recorded speech. The speaker ran out of time before demonstrating the playground.

[12:59](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=779s) [A slide lists claimed latency, throughput, voice quality, cloning and control features of Cartesia Sonic.](https://assets.ai.engineer/sites/aiecode2025/sha256/fdd98dfb3de5737c1f5a27bad6e28898827814de1a09fe51644f873dd8931098.webp)

Cartesia Sonic: A Turbo Model for Voice Generation

Conversational Interfaces
https://play.cartesia.ai

✓ Blazing fast
135 ms model latency

✓ High throughput
Highly concurrent, low-cost inference with our first-of-its-kind state space model inference stack

✓ Ultra-realistic
Human, emotional, expressive text-to-speech models built on an entirely new state space model architecture

✓ Supports zero-shot voice cloning
Match prosody, inflection, and vocal characteristics with only 10 seconds of recorded speech

✓ Controllable  ALPHA
Adjust pitch, speed, emotion, pronunciation and speed

AI Engineer World’s Fair | Cartesia
June 2024

## Full transcript

[0:00](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=0s) [on-hold music] Maybe to set the stage a little bit, um, the last four or five years of AI have basically been really focused on this idea of batch intelligence, which is sort of, um, pretty core to this, uh,

[0:25](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=25s) idea of, uh, building, like, an AI system that can reason, uh, for long periods of time on a problem and then, uh, solve it. So you can think about, like, math problems or, you know, physics problems that are hard.

[0:36](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=36s) Um, there's a lot of applications where actually what you need are systems that are streaming. So they're real-time, they, uh, work instantly. So imagine, um, generating video, audio, or, um, doing, like, understanding applications on, um, sensor streams, et cetera.

[0:51](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=51s) So, um, so it sort of bifurcates where there's these two different types of, uh, applications, similar to how there's, you know, generally this idea of having, uh, batch workloads and streaming workloads.

[1:01](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=61s) And so, um, a lot of what we've seen over the last few years has really been focused on, uh, batch APIs where you call a model in the cloud, it takes a few seconds, and then you get a, a pretty good response back.

[1:12](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=72s) Um, and now we're seeing some shift towards more real-time applications where you constantly will be querying a model and, um, h- asking it to return responses at low latency and then, uh, using that to sort of interpret, uh, or generate information.

[1:28](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=88s) Um, and I think this, you know, this area is really exciting because it's going to be transformative to a lot of interesting applications that, um, have so far actually not necessarily been, uh, the main focus for, uh, a lot of what we've seen, uh, over the last few years.

[1:43](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=103s) So conversational voice, uh, is an example of this where you should be able to interact with a system and then talk to it, um, a-and it should be able to understand you and, and do all kinds of, uh, tasks on your behalf.

[1:54](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=114s) Uh, this is similar to having assistants, um, that are on device and run kind of really efficiently at low power, um, at, you know, all times, um, regardless of whether you're on a phone or, or a laptop.

[2:06](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=126s) And then, um, things like world generation where, like, you can imagine actually playing a game that is generated in real time similar to, um, how the graphics are rendered, um, on GPUs.

[2:15](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=135s) Um, and, um, all of this, you know, should be able to happen in real time, on low power, on your phone, uh, on your MacBook, et cetera. Um, robotics is another great example where it sort of culminates with all of these coming together on a, on a, a single, um, uh, device that is, uh, trying to kind

[2:35](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=155s) of interpret everything in the world. And so I think this is sort of the ex-exciting intersection, which is like, how do we make intelligence faster and cheaper so that we can put it, um, everywhere, basically.

[2:47](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=167s) Um, and a couple examples that are really powerful. Real-time intelligence for conversational interfaces, uh, is gonna be really interesting because you would be able to have a agent that can provide customer support for a problem, answer questions about health insurance, uh, you know, call your vendor to pick up a shipment.

[3:04](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=184s) A-all these coordination tasks that, uh, generally are, um, annoying to do should be really automated and, uh, real-time intelligent agents should be doing them. Um, and then humans can spend their time solving sort of harder problems that are, uh, more interesting and, uh, in customer support, that could be dealing with, uh, you know, the tail customers that

[3:22](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=202s) are, uh, much more important because they're pissed off or they're, uh, uh, uh, more important because they have, uh, you know, more customer value, et cetera. And similarly in robotics, there's this idea of, like, ingesting similar to humans, like audio, video, sensor data, and then responding instantly to a lot of these, um, uh, pieces of information.

[3:40](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=220s) So I think this is sort of the, the world we, we should be living in where all of these intelligent models run super fast, they solve all these different problems and, uh, they're able to really kind of, um, power these new experiences that are interactive at their core.

[3:56](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=236s) So this is where we come in. We're building, uh, these real-time foundation models. So, um, some of what I'll talk about is, uh, the work we've done in really building kind of new ideas for how you can create, uh, deep learning models.

[4:10](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=250s) So, um, I did my PhD before this. I was working with a lot of these folks for my PhD. Chris was our PhD advisor. Um, and, um, we were really focused on this idea that you should be able to have a model that can compress information as it comes into the model and, um, use that to really

[4:26](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=266s) kind of build powerful systems that are streaming at their core. Um, and I'll talk a little bit about this, but that's really the technology that we've been working with for the last four or five years.

[4:36](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=276s) We've been developing in academia, and some of you might have heard of things like Mamba, which is sort of a, a more recent iteration of this technology. You know, we-- I did my PhD working on some of the early iterations that nobody uses anymore, but, um, are sort of the precursors to a lot of the modern stuff

[4:51](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=291s) that is now, uh, more widely used. And, uh, and now what we're doing at Cartesia is basically taking this and trying to understand how we can improve it. Mmm, how do we push the boundaries on, uh, what architectures can do?

[5:02](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=302s) And, um, and I think it's an interesting question because, you know, we should not settle for having a, uh, one way of doing things. I think that's sort of a, a poor way to kind of, um, think about the future.

[5:13](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=313s) So our approach is sort of like, let's think about new ways of actually designing models that aren't necessarily built on, let's say, the transformer architecture and the standard recipe for, uh, deep learning that's, uh, you know, prevalent today.

[5:26](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=326s) And I think it boils down to this question of, like, efficiently modeling long context is a huge problem because, you know, a lot of practical data is really long sequence data.

[5:36](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=336s) I think text is maybe the least interesting, uh, long sequence data because text is actually fairly, uh, compressed already, right? Like, you have a lot of information that is, uh, embedded in two minutes of, uh, or, uh, two sentences of text.

[5:49](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=349s) But there's all these other domains where, you know, audio, video, et cetera, where there's so much information. Um, you know, imagine looking at a security camera for a day.

[5:57](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=357s) Like, you would probably have just so much information coming into the system and just, um, very little of that would be useful. So compression is kind of really fundamental to intelligence because we're able to do this where we can look at all this information and then sort of compress it down to whatever's necessary to remember or understand.

[6:15](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=375s) Um, and I think so far what we've seen is that the AI systems that we built-

[6:18](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=378s) Have not necessarily exhibited that same behavior. So they're really kind of built not on the principles of compression, but more on this idea of retrieval, like keeping all the context around and then using it to reason over all the information that you've seen.

[6:30](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=390s) So I think, um, our kind of point of view is that multimodal AI will remain challenging as, as long as you're sort of, uh, working in this paradigm. Because, uh, if you try to think about what humans do in a year, um, you're basically processing understanding about a billion text tokens, ten billion audio tokens.

[6:48](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=408s) These are, you know, back of the envelope calculations that I did. And about a trillion video tokens, probably underestimates how much video we process, and not including all the other sensory information that you're processing, and you're doing it simultaneously, and you're doing it on a computer that fits in your brain.

[7:03](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=423s) Um, and you, you know, sometimes don't eat and drink and, you know, you're still functioning fine, so you know, you can have variable amounts of power in the system.

[7:12](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=432s) Um, so I think the idea that like intelligence is solved is sort of very far from the truth because humans just are an extremely amazing machine that, uh, does something very extraordinary, um, in a very compressed way that, um, our AI models can't do.

[7:27](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=447s) So I think that's sort of our, uh, you know, sort of, uh, the reason we get up in the morning is we think about this and we're like, yeah, we're very far away from where we should be.

[7:35](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=455s) Um, and the best models today are in the, you know, ten million, a hundred million sort of token range. So, uh, that's really good. A lot of progress has been made, but really this is sort of what we aspire to is how do you kind of build these machines that are long-lived that can actually understand information over

[7:51](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=471s) very long periods of time? And I think the cool thing is like, as a human, you can remember things that happened thirty years ago with very little effort. You don't need to do RAG or retrieval or anything.

[8:00](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=480s) You just, you know, you remember it. It's just it in your brain, and then you figure it out, basically. So I think that's kind of an extraordinary capability that we should be able to put into our, uh, AI models as well.

[8:13](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=493s) And so, uh, some of the big problems with models today are, uh, you know, they're built on transformers really optimized for data center. I think, um, we see this with like a lot of the work we did, which was on subquadratic models.

[8:23](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=503s) So quadratic scaling in context length really just means that, um, you know, the amount of, uh, computation you have to do to process long amounts of, uh, context is very large.

[8:33](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=513s) And so right now, the sort of predominant approach is to throw compute at that problem and then hope that that would scale. Um, obviously, compute is a very important piece of the puzzle because you do need more computation to be able to do more difficult things.

[8:45](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=525s) But, um, this type of approach, because the quadratic scaling actually has poor scaling with, you know, very large multimodal context. And text context tend to be shorter. Multimodal context will get larger because you have just way more tokens and information that's going into the system.

[8:59](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=539s) So that's gonna be a big challenge for these models, especially how do you do this inference efficiently so you're not, you know, burning down the data centers to, you know, do, uh, a fairly limited amount of inference.

[9:09](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=549s) Like, you have to imagine that we're doing a thousand times or, um, you know, a hundred thousand times more inference. And then, um, if the models are scaling the same way, it's gonna be really, really, really expensive.

[9:19](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=559s) So you're not gonna be able to permeate all these applications that I talked about very easily. Um, and so, you know, that's sort of a big challenge, I would say.

[9:27](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=567s) A-and so, you know, again, our hypothesis is you need new architectures, and that's kind of where we spend our time, and we wanna make these models more efficient, faster, more capable, while being able to handle all these long context problems.

[9:37](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=577s) Um, this is a slide about, you know, transformers being somewhat inefficient at handling this, um, but obviously, um, a very good recipe for scaling, uh, uh, these models out.

[9:48](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=588s) And so, you know, some of the work that we've been doing is new, fundamentally efficient architectures. So they have compression at their core. So they sort of-- The way they operate, uh, I'll, I have a slide on this just to give you kind of a, a, a quick illustration, but, um, they really scale more linearly in context

[10:04](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=604s) length. So you should be able to have, uh, because of this, like, more low power implementations of these models, um, you can compress information as it comes into the system, you have low memory usage, um, and you can actually scale to much more massive context because of that.

[10:21](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=621s) Um, and this is all the work around SSMs. I just threw this, uh, nice slide, which I thought was cool. Um, Jensen had an interesting quote about SSMs in one of his Wired articles that I like to keep talking about.

[10:31](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=631s) But, uh, uh, but I think it's a cool technology that has a lot of potential and sort of that's where we're, um, spending a lot of our time. And if you folks are interested in reading more, there's lots of videos on YouTube and lots of sort of resources that try to make this more accessible to understand and,

[10:45](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=645s) and kind of get into some of the details. Um, but, you know, the working intuition is basically, uh, transformers are generating quadratically by attending to every past token of information.

[10:54](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=654s) So as tokens come into the system, you're kind-- sort of keeping them around and then looking at all the past tokens. So if you wanna generate the word jumped, uh, from the quick brown fox, you would actually look at the entire context, try to understand what the next word should be, and then, um, generate it, push it

[11:09](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=669s) into the context, do it again. Uh, with SSMs, you just have a streaming system. So you, you have a to-- uh, token stream in, uh, they update an internal memory for the model, and then the token gets thrown away.

[11:19](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=679s) So that actually really simplifies the system, and that's why it's such a core sort of streaming interface, because you're just not keeping all this memory around about what happened in the past.

[11:27](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=687s) You're compressing it into some sort of zipped file, uh, state inside the model that's gonna be used to do a future generation.

[11:36](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=696s) And so this is sort of taking this idea of, uh, taking advantage of this idea of recurrence, which is sort of core to how even humans, uh, do a lot of their reasoning.

[11:45](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=705s) And, you know, last few months, a lot of these models have been getting adopted. So it's great that, you know, a lot of folks are now excited about the, uh, this, you know, alternate way of doing things that is much more, uh, sort of are oriented around this idea of recurrence rather than, um, retrieval.

[12:01](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=721s) Um, and so I think, like we'll see a lot more activity in this, especially with multimodal data becoming more important. And, uh, you know, a lot of the challenges of multimodal data around efficiency will mean that I think that these models will have more of a role to play in the next three to five years, um, as

[12:15](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=735s) we also do our work in, uh, scaling them up and making them more interesting. A lot of people ask me about quality. Uh, I only have a few minutes, so I'll go through this, uh, the rest of the slide super fast.

[12:24](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=744s) But Um, you know, SSMs generally have the right, uh, quality. Obviously, there's a trade-off between compression and keeping all of the information around. Uh, but actually, like, compression can be helpful.

[12:36](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=756s) So if you imagine the security camera example, if you're watching twenty-four hours of footage, actually compressing all of that information on the fly would help you solve tasks and a- answer questions better, uh, rather than looking at all twenty-four hours every time.

[12:48](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=768s) So I think that's sort of the rule of thumb to think about, which is compression's super helpful for a large context, uh, not as helpful for, um, short context.

[12:55](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=775s) And so we see that quality actually is very good for long context problems and, and multimodal problems. Let me talk quickly about some of the work we've been doing.

[13:03](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=783s) So we've been starting to work on sort of multimodal data, and we did a release a few weeks ago for a, uh, voice generation model. So this is sort of text-to-speech and sort of in line with some of the work we're doing to bring more multimodal, uh, data into a single model, um, and use SSMs to power

[13:19](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=799s) the inference and the training and, and so on. So this is a model you can actually play with. I'll try to show you a demo. But, um, one of the things we're proudest of with this model is that we really shrunk the latency down, so when you play with it on the playground, you get instant voice back

[13:32](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=812s) generated from the data center. And there's some cool work we're doing to actually run these models on Mac, um, and other devices so that you can basically have the same experience as you have in the data center, but on any device and, and do that efficiently and at low power.

[13:45](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=825s) How much time do I have? You have four seconds. Okay. We're, we're out of time, but I was also almost done. So, um, go to the website play.cartesia.ai. I unfortunately couldn't walk through the demo, but, um, play with it and, um, send us feedback.

[13:58](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=838s) This is my email in case you wanna, uh, send me a note. Uh, would love to hear feedback and anything that you folks, uh, find interesting. Thank you. [clapping] [upbeat music]
