▶ Watch ↗AI Engineer World's Fair 20251:01:42
Building Conversational AI Agents - Thor Schaeff, ElevenLabs
Read the full talk →Key ideas
Scroll to read ↓Follow a voice agent from speech recognition to language switching, tool calls and spoken replies, then examine what breaks with slow systems, mixed languages and domain vocabulary.
- Which languages—and which accents—should the agent speak?0:22 ↗
- Text to Bark: generated sound is not translation5:08 ↗
- Speech, text, intelligence, speech8:26 ↗
- A transcript can carry timing, speakers and events10:51 ↗
- Forward a voice message, get readable text12:39 ↗
- Choose the intelligence layer and the speaking voice17:05 ↗
- Configure a conference agent20:31 ↗
- Switch languages without restarting the conversation23:38 ↗
- Language recognition selects a configured voice28:49 ↗
- Give the agent appointment-setting tools30:58 ↗
- Balance response time, session cost and task complexity33:01 ↗
- Keep the caller informed while tools run39:06 ↗
- A mixed-language question exposes a recognition boundary43:56 ↗
- Voice generation needs misuse controls49:00 ↗
- Test the languages people actually mix52:00 ↗
- Generated voice is only one part of an avatar55:06 ↗
- Pronouncing SAP and recognizing Joule are different problems57:49 ↗