AI Engineer Europe 2026
Beyond Transcription: Building Voice AI That Actually Understands Conversations
Loading slides…
Read the talk
Beyond Transcription: Building Voice AI That Understands Conversations
Words alone miss interruptions, backchannels and speaker changes. Hervé Bredin shows how diarization works, how to evaluate it, and why joining its output to a transcript remains difficult.
From a talk by Hervé Bredin
The missing speaker labels
A transcription system can return the words of a conversation without telling you which participant said them. Hervé Bredin came to that problem through years of academic research before co-founding pyannoteAI, where he became chief science officer. His open-source pyannote.audio toolkit focuses on speaker diarization: tracking which speaker is active in a recording. When OpenAI released Whisper, users found a natural pairing. Whisper supplied the transcription; pyannote supplied the missing speaker tags. Bredin connects an inflection in the toolkit’s adoption to that release.
The adoption story comes with a birthday request. With his birthday a week away and the project approaching 10,000 GitHub stars, Bredin asks the audience to give it a star as a present.
The underlying distinction is simple. Speech-to-text, or STT, maps recorded or streaming audio to a sequence of words. It answers what was said. But a conversation flattened into one text stream can be hard to interpret: the words alone do not tell you where one participant’s contribution ends and another’s begins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From what was said to who said it
Speaker-attributed transcription associates the words with speaker tags. That changes the question from what was said to who said what. For a meeting assistant, this may provide enough structure to summarize the discussion and, once the tags are associated with participants, assign an action item to the person who actually accepted it. A correct sentence attached to the wrong participant can produce an incorrect meeting record.
The same need appears in several applications:
- Video translation and dubbing: translated speech needs a consistent voice for each participant. Switching voices between unrelated speakers breaks that correspondence.
- Meeting notes: commitments and contributions need to remain attached to the right people.
- Podcast intelligence: finding the same guest across episodes—or across different podcasts—requires connecting speech to a recurring person.
These applications motivate speaker information, but they ask for different amounts of it. Maintaining consistent labels inside one recording is a narrower task than recognizing a person across recordings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Timing and delivery change the interpretation
Even correct speaker attribution leaves information out. In Bredin’s example, the timing reveals that the black speaker interrupts the green speaker. A sequence of speaker-labeled sentences without precise timing can conceal that interruption. It can also conceal a backchannel: a short acknowledgment made while someone else continues speaking. Such a response may indicate agreement without taking over the conversational turn. Missing it can remove a small but consequential contribution.
Pauses matter for the same reason. A gap between turns can affect how a response is interpreted, even though it contributes no words to the transcript. The next layer is delivery: how something was said. Bredin offers laughter and coughing as examples of sounds whose possible meaning depends on context. A laugh might respond to a joke or to something awkward; a cough might accompany discomfort. These are possible interpretations, not reliable labels for someone’s state of mind.
Prosody—patterns of stress, rhythm and intonation—and disfluencies add further information. Bredin’s example is “The dog ate the cake”: a child can use the same words while conveying different implications through emphasis. Preserving that information would give a downstream language model, or another analysis tool, more to work with than the word sequence alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who is being addressed, and where?
A broader view asks who is talking to whom. Addressing an entire audience is different from answering one person’s question during Q&A. The acoustic environment adds another layer: the same exchange in a quiet room and on a street comes with different contextual information. These are parts of the conversation-understanding problem Bredin wants to pursue, but he narrows the technical discussion to speaker diarization. Detecting who speaks when does not, by itself, solve addressee detection, prosody or intent.
To illustrate demand for that narrower task, Bredin shows a Hugging Face audio-model ranking sorted by downloads. In the displayed snapshot, three of the top seven models concern speaker identity or diarization; the remaining models concern speech-to-text. It is a historical illustration of how often developers need speaker information alongside words.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From speech activity to consistent speaker labels
Bredin builds up diarization through three increasingly specific tasks:
- Voice activity detection: identify the regions where anyone is speaking.
- Segmentation: locate speaker changes within those regions and find overlapping turns, including interruptions and backchannels.
- Speaker assignment: give the turns consistent speaker labels throughout the recording.
The fine detail in segmentation matters. A very short affirmative response may carry essential information about a participant’s reaction. Detecting a long region of speech while losing that small turn is not enough. Conversely, finding the turns is not yet full diarization: the system must also establish which turns belong to the same speaker.
The number of speakers is generally unknown in advance. A meeting’s attendee list can provide guidance, but it is not an exact inventory of voices: two people may share a channel, or someone who was not invited may join. Unlike classification against a fixed list of known classes, the system must infer how many distinct speakers are present.
Those classes are also anonymous. Diarization normally returns labels such as speaker one, speaker two and speaker three, not John, Hervé and Jack. The labels are interchangeable as long as their assignment stays consistent. In the two-speaker diagram, changing every green label to black and every black label to green leaves the diarization equivalent. What matters is the grouping of speech by speaker, not the spelling or color of the label.
Unknown speaker counts and interchangeable labels are only part of the difficulty. Speakers overlap, some turns are extremely short, and speaking time can be highly uneven. A participant who contributes only a brief acknowledgment gives the system much less speech to work with than the person leading the discussion. Noise and other acoustic conditions add the difficulties familiar from speech processing more generally.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspecting diarization errors in the notebook
The first Python notebook demonstration makes these errors visible. Its input is a roughly thirty-second telephone conversation between two people. After an uncertain exchange of greetings, they introduce their locations—New Jersey and Texas—and discover that both are originally from Chicago. Bredin presents a manually labeled reference for the sample, then applies the open-source Community-1 diarization model.
The local workflow is to download the model from Hugging Face, use PyTorch’s MPS backend on his Mac, and apply the pipeline to the audio file. After switching from an already processed notebook tab to the live run, Bredin compares the resulting prediction with the reference annotation. The aligned timelines make it possible to inspect where the output diverges from the expected speaker activity.
The comparison separates three kinds of error:
| Error | What went wrong |
|---|---|
| Speaker confusion | Speech was assigned to the wrong speaker. |
| False alarm | The system detected speech where the reference contains none. |
| Missed speech | Reference speech was not detected. During overlap, this includes detecting only one of two active speakers. |
That last case is easy to overlook. A system can correctly detect that someone is speaking and still miss part of the conversation because another person is speaking at the same time.
Bredin uses pyannote.metrics to combine these errors into diarization error rate, or DER. The quantities are durations, not word counts:
Bredin reports about 5% DER for Community-1 on this telephone sample.
Editor: The metric’s scoring convention counts speaker-time, not elapsed file time: one second containing two reference speakers contributes two speaker-seconds when overlap is scored. Comparisons also depend on whether overlap is included and on the boundary collar, a tolerance around reference boundaries.
He next submits the same sample to the cloud API for Precision-2. In the displayed result, it fixes a speaker confusion and reduces false-alarm and missed-speech errors. Bredin reports about 3% DER for Precision-2 on the same sample. These are results from the short demonstration recording, not dataset-wide benchmark scores.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accuracy depends on the recording
How well does diarization work beyond that short call? Bredin’s answer depends on the use case. He reports a best-system DER of about 8% for conversational telephone speech between two people. For a noisy restaurant conversation involving several friends, he reports about 41% DER even for the best system. These are separate scenario-dependent comparisons, not additional scores for the two models in the notebook. The number of participants and the acoustic environment change the difficulty substantially; noisy group conversation remains far from solved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a leaderboard score may not transfer to a meeting
Once transcription and diarization are available, producing speaker-attributed text might appear to require only assigning a speaker to each word. The first obstacle is that recognition itself may already be wrong. Bredin argues that most STT models are trained on single-speaker data and generalize poorly to multi-speaker recordings with overlap and speaker changes. His example uses NVIDIA Parakeet and the Open ASR leaderboard, where ASR means automatic speech recognition.
For Parakeet on the AMI meeting dataset, he contrasts the leaderboard’s word error rate, or WER, with his team’s result:
| Evaluation described by Bredin | Recording condition | Reported WER |
|---|---|---|
| Displayed Open ASR leaderboard | Individual headset microphones, giving single-speaker speech | 11.4% |
| His team’s AMI evaluation | A distant microphone in the middle of the meeting table, capturing multiple speakers | 26% |
The dataset name is the same, but the input conditions are not. Bredin explains that the headset recordings and the shared table microphone present materially different recognition problems. The comparison changes both microphone placement and conversational conditions; it does not isolate the effect of overlap alone.
The difficult conditions include distant speech, speaker changes, crosstalk and interruptions. Bredin also raises code switching, changing languages within a sentence. For a meeting application, the practical evaluation rule is to test the kind of recording the product will actually receive. A score from isolated headset speech does not establish performance on a shared room microphone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Speaker attribution is not a simple timestamp join
The second obstacle is reconciliation: making the transcription and diarization agree about which words belong to which speakers. Their timestamps can disagree. Their coverage can disagree too: diarization may detect speech that STT never transcribes, or STT may produce words where diarization reports no speech. During overlap, two active speakers may be represented by only one recognized word stream. A timestamp join cannot assign words that recognition never produced.
In the second demonstration, Bredin applies Parakeet to the same telephone sample. It returns a sequence of words with corresponding start and end times, providing the word intervals needed for attribution.
Editor: The published demo implementation identifies the variant as
nvidia/parakeet-tdt-0.6b-v3and requests timestamps withtimestamps=True.
He places those words beside Precision-2’s speaker turns and examines the assignments one by one:
| Displayed case | Why attribution is easy or difficult |
|---|---|
| A word interval is shifted slightly left | The yellow speaker still appears to be the clear match despite the timing mismatch. |
The third word, oh, falls between two diarized turns | The word has no unambiguous turn to attach to. Bredin replays it while considering the assignment. |
The word okay coincides with two active speakers | Diarization offers two candidates, but transcription supplies only one word. |
These are different failure modes: imperfect boundaries, a gap between turns, and competing speakers during overlap. Treating all of them as the same interval-matching problem hides the uncertainty that reconciliation must resolve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preserving the overlapping exchange
Bredin then submits a job to pyannoteAI’s cloud service that combines Precision-2 diarization, Parakeet transcription and reconciliation. The returned output contains words assigned to speakers, so the application does not have to resolve each conflicting interval itself.
The revealing example is the transition between the two introductions. One participant’s trailing um overlaps the other participant’s opening words. Bredin plays that passage and points out that the output interleaves words from the two speakers, retaining the filler as the other person begins speaking. He reports that the system gets this overlap right. The demonstrated success is specific: it preserves both contributions in this exchange, rather than losing one inside a single speaker’s turn. The demonstration does not reveal the complete reconciliation algorithm.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The tools behind the demonstration
Bredin closes the demonstration by pointing to the talk tutorial in pyannoteAI/tutorials. Its components have distinct roles: pyannote.audio supplies open-source diarization, pyannote.metrics evaluates the predictions, and iPyannote provides the interactive visualization widget. The initial i, he explains, stands for interactive. The SDK provides access to the premium models, complementing the local open-source workflow shown earlier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Exclusive diarization and independent STT models
An audience member asks where conflict resolution happens: is it learned during model training, or supplied by heuristics around the models? Bredin says the complete reconciliation technique is proprietary, but identifies one disclosed part available in Community-1: exclusive diarization.
During overlapping speech, Bredin explains, exclusive diarization selects the speaker considered most likely to be transcribed by the STT model. This removes a competing speaker assignment from that view and makes reconciliation simpler. It is one major part of the approach, not a complete explanation of how the earlier demonstration recovered and interleaved the overlapping words.
Editor: The Community-1 model card documents separate
output.speaker_diarizationandoutput.exclusive_speaker_diarizationresults. The exclusive output is a simplified view for reconciliation; it does not mean the original recording contained no overlap.
Pressed on whether this requires changes to model training, Bredin emphasizes the intended boundary: the orchestration should work without modifying the STT model itself. He describes support for different transcription systems—including privately fine-tuned models—as a design goal. That lets a team retain a recognizer adapted to its own use case while adding diarization and reconciliation around it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The exact tutorial for the two demonstrations, including diarization error inspection and speaker-attributed transcription. Full reproduction requires model access and cloud credentials.
Model card for the open-source local pipeline, including its exclusive-diarization output for reconciliation with transcription.
Reference for DER components, speaker-label matching and evaluation settings used to interpret the first demonstration.
Official model card for the exact Parakeet variant found in the talk's tutorial implementation.
Open-source toolkit for voice activity, speaker segmentation and speaker diarization.
Full transcript
Loading transcript…