AI Engineer World's Fair 2025

The End of Awkward AI Transcriptions

Travis BartleyMyungjong KimByungjoongJaehan16:24

Loading slides…

Read the talk

The End of Awkward AI Transcriptions

Reliable speech recognition depends on matching encoders, decoders, speaker models, and customization tools to the audio and deployment constraints of each application.

From a talk by Travis Bartley, Myungjong Kim, Byungjoong and Jaehan

Before you start: Basic familiarity with neural networks and inference is useful; ASR means automatic speech recognition.

What must a speech system handle?

Will a recognition system work as well on a noisy telephone call as it does on clean speech? Will it recognize medical terminology, handle a speaker switching languages, and still respond quickly enough for a conversation? These are the opening engineering questions behind NVIDIA Riva, an enterprise platform covering speech recognition, synthesis, and translation. Low latency and efficient inference matter throughout, including on embedded devices.

The requirements fall into four connected categories:

  • Robustness: Noise, telephony audio quality, and environmental interference change what reaches the recognizer.
  • Coverage: Medical, entertainment, and call-center applications need different vocabulary and data. Language coverage also includes dialects and code switching, not just a list of supported languages.
  • Personalization: Target-speaker recognition, uncommon-word boosting, and finite-state text normalization let customers influence whose speech is recognized and how the result appears.
  • Deployment: Speed competes with accuracy, while broad task coverage can compete with efficiency.

The model choice follows the application’s constraints. A versatile model is useful only if its resource requirements and response time fit the product.

The Developer’s Challenge slide shows a photo of two people beside four categories of speech application requirements and trade-offs.Open full source frame
Speech application trade-offs: robustness, coverage, personalization, and deployment.
0:320:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:32 · section reference included

Choose the decoder for the workload

CTC remains useful because its non-autoregressive decoding supports fast inference. The streaming approach described here runs recognition over chunks of audio. When that approach does not deliver the desired accuracy, RNN-T and NVIDIA’s TDT variant provide autoregressive streaming: they combine an audio encoder’s representations with an internal language-modeling component.

When streaming is less important, an attention encoder-decoder can prioritize accuracy and accommodate several tasks. Whisper is a familiar reference point for this architecture. A Transformer decoder can use prompt changes to select speech recognition, translation, timestamp prediction, or language identification.

Decoder approachDecoding behaviorEmphasis in the talk
CTCNon-autoregressiveFast recognition over audio chunks
RNN-T / TDTAutoregressive transducerStreaming with language context
Attention encoder-decoderAttention-based sequence generationAccuracy and multiple tasks

These are workload choices, not a universal accuracy ranking. The relevant question is whether the application needs immediate partial results, richer task control, or a particular balance of both.

2:423:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:42 · section reference included

Shorten the sequence before decoding

Across these decoder choices, the common encoder is Fast Conformer. Its central change in this walkthrough is additional temporal subsampling: conventional 40 ms encoder steps become 80 ms steps. That gives downstream computation a shorter sequence to process. The 80 ms figure describes encoder-frame spacing, not end-to-end recognition latency.

Shorter encoded sequences reduce training memory pressure and the number of temporal positions processed during inference. The presenters report quicker convergence with less training data, but give no quantities or experimental conditions. Their practical reason for sharing this encoder across decoder families is that the computational savings remain useful whether the output is produced by CTC, a transducer, or an attention decoder.

4:264:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Parakeet and Canary divide the work

The portfolio puts these architecture choices into two families. Riva Parakeet groups CTC and TDT offerings around fast, efficient streaming recognition. The presentation associates Parakeet with recognition, translation, and target-speaker applications. That broad application framing should not be read as translation support in every Parakeet checkpoint: the architecture documentation describes Parakeet as an ASR family and Riva Canary as supporting recognition and translation.

Canary emphasizes accuracy and multitask modeling, while still pursuing speed. Fast Conformer is shared by both families; it is not exclusive to Canary. Together, the families let a deployment choose among speed, accuracy, and task coverage instead of forcing every workload through one model. Variety is part of the system design: the goal is a model that fits the need, rather than one model expected to satisfy every need.

5:275:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:27 · section reference included

Connect speaker timestamps to transcript tokens

Recognizing words does not establish who said them. Sortformer extends the system with an end-to-end neural diarizer that assigns speaker order by arrival time: the speaker who appears first comes first. This ordering bridges diarization, which identifies speaker activity over time, and ASR, which must associate speaker tokens with recognized words.

A small Python example makes the ordering rule concrete. Given illustrative first-speech times, the output assigns speaker tokens in arrival order rather than in the arbitrary order of the input labels:

python

first_speech_seconds = {
    "voice_b": 2.4,
    "voice_a": 0.6,
}

arrival_order = sorted(
    first_speech_seconds,
    key=first_speech_seconds.get,
)
speaker_tokens = {
    voice: f"speaker_{index}"
    for index, voice in enumerate(arrival_order)
}

assert speaker_tokens == {
    "voice_a": "speaker_0",
    "voice_b": "speaker_1",
}

The neural diarizer supplies speaker activity; sorting alone does not detect voices or recognize their words. Its role here is to illustrate a consistent convention for connecting speaker identities to tokens.

The joint architecture fuses ASR encoder embeddings and Sortformer embeddings through a speaker kernel. This gives the recognizer speaker information alongside acoustic information, addressing who spoke what and when. Fine-tuning uses an objective described as similar to ordinary ASR training.

Optional query audio changes the task. With a query, the model can perform target-speaker ASR; without it, the model can perform multi-speaker ASR. The diagram shows the ASR encoder and Sortformer feeding the speaker kernel, a linear projection, and an RNNT decoder, with separate target-speaker and multi-speaker output examples.

Diagram connects a Fast Conformer ASR encoder and Sortformer through a speaker kernel and linear projection to an RNNT decoder, with TS-ASR and MS-ASR outputs and optional query audio.Open full source frame
Multi/target-speaker ASR architecture with Sortformer and optional query audio.

The same component pairing can also operate as a conventional cascade rather than a parallel joint model. That distinction separates the architecture’s integration possibilities from a requirement to deploy every component as one end-to-end system.

6:577:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:57 · section reference included

Make recognized speech usable

The recognizer sits inside a larger pipeline. Supporting models address distinct problems, from selecting speech regions to producing readable text:

ComponentFunction
Voice activity detectionDetect speech segments for noise robustness
External n-gram language modelRescore recognition hypotheses
Inverse text normalizationConvert spoken forms to written forms
Punctuation and capitalizationRestore readable sentence formatting
Speaker diarizationIdentify speakers across a conversation

For VAD, the presentation names DistilBERTaVAD and MarbleNet-based models. External n-gram language models add another place to improve accuracy and adapt recognition to a customer’s vocabulary.

Normalization has two directions: text normalization converts written text into spoken form, while inverse text normalization, or ITN, converts spoken-form transcripts into readable written forms. Riva uses weighted finite-state transducers for ITN. BERT-based punctuation and capitalization models then provide sentence formatting. Sortformer supplies speaker diarization; at the time of the presentation, cascade support is described alongside upcoming joint end-to-end models.

8:268:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:26 · section reference included

Recognize lyrics over music

At presentation time, Travis reports that NVIDIA supplies a majority of the Hugging Face Open ASR leaderboard’s top five models. He attributes the result to customization and model variety. It is a dated leaderboard claim, not evidence that customization alone caused the ranking, and it leads into a more concrete test: recognizing a song.

The demo plays rapid rap vocals over musical accompaniment. The browser interface shows an audio waveform, playback controls, and timestamped transcript rows under a Riva Parakeet V2 demo. This is a useful qualitative example because lyrical delivery and background music move the input away from clean conversational speech.

Slide titled The End of Awkward Transcriptions contains a browser demo with a partially green audio waveform, playback controls, and a timestamped transcription table.Open full source frame
Riva Parakeet V2 lyric transcription demo showing an audio waveform and transcript rows.

Travis judges the transcription accurate in the noisy setting. No word error rate, comparison baseline, or detailed test configuration is supplied for the song, so the result demonstrates a particular successful example rather than a measured robustness guarantee.

9:5010:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:50 · section reference included

Build coverage into the training data

The training discussion returns to fundamentals rather than introducing a special recipe. Robust sourcing, multilingual coverage, and dialect sensitivity begin with language documentation: understand the language well enough to decide which kinds of data the training set must contain. Open-source data contributes variety and domain shifts; proprietary data supplies a complementary emphasis on quality.

Pseudo-labeling expands the available transcripts. The team uses leading models available for commercial use to generate text from audio, allowing new training data to benefit from both community advances and NVIDIA’s own releases. The generated transcript becomes a training label, so improvements in the models producing those labels can feed into subsequent model development.

10:5311:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:53 · section reference included

Keep training fed and validation broad

Training uses the open-source NeMo toolkit. The operational focus is GPU utilization, data bucketing, and high-speed loading through Lhotse. These tools address the practical problem of supplying enough audio efficiently across different training settings. Most of the team’s data resides in object storage, which helps it move between cluster environments.

Validation mirrors the breadth of the training sources: it combines open-source and proprietary datasets and checks bias, domain behavior, and language coverage before release. A recognizer intended for varied customer environments needs evaluation beyond a single clean benchmark; otherwise, broad training coverage can remain an assumption rather than something tested.

12:0112:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:01 · section reference included

Serve streaming and offline workloads

After training, models move into Riva through NVIDIA NIM. TensorRT optimizations and Triton Inference Server provide the inference foundation. A gRPC microservice interface supports both low-latency streaming and high-throughput offline processing, while containers package the serving system for deployment.

The presenters describe scaling Riva to hundreds of parallel streams, without specifying the hardware, model, or latency target. Deployment targets include on-premises infrastructure, cloud environments, edge systems, and embedded platforms, serving applications such as contact centers, consumer products, and video conferencing. NIM supplies prebuilt containers, industry-standard APIs, custom-model support, and optimized inference engines.

13:2213:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:22 · section reference included

Adapt the layer responsible for the error

Customer requirements return at deployment time. Medical terminology and restaurant menu names require domain knowledge; telephone audio and noisy contact centers impose acoustic constraints. Those problems do not all belong to the same layer, so Riva exposes several adaptation points:

  • Acoustic models: Fine-tune Parakeet- or Canary-based models for the relevant audio and recognition task.
  • External language models: Adapt the n-gram model to domain language.
  • Output formatting: Customize punctuation and inverse text normalization to produce the written forms the application needs.

The point of this separation is to give vocabulary, acoustics, and presentation their own places to improve.

Word boosting adds a more focused option for product names, jargon, and vocabulary that matters in the current context. It biases decoding toward selected terms rather than guaranteeing that they will be recognized correctly. Stronger boosts can also introduce false positives, so the customization guidance treats boosting as a tradeoff rather than a universal correction.

14:2714:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:27 · section reference included

Find a model, then deploy or fine-tune it

The starting point is the NVIDIA speech model catalog, where Riva models are offered through NIM. The catalog now includes later offerings as well, so model capabilities and API options should be read for the selected release rather than assumed from this presentation.

The closing resources point to a Riva quickstart for deployment, the developer forum for implementation questions, and a NeMo fine-tuning guide for adaptation. That leaves two practical paths from the architecture walkthrough: deploy an available model when it meets the application’s requirements, or customize the relevant parts of the speech pipeline when vocabulary, audio conditions, or output formatting demand more.

15:3215:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:32 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…