We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI

Cool. All righty. Can everyone hear me okay? All, all levels coming through good? All right. Sweet. Well, uh, how's everyone doing?

How's everyone doing? Better? Okay, there we go. Cool. Well, I hope, uh, the conference has been good for everybody. Um, appreciate you all being here to, to listen to a chat from, uh, from me. Um, for anyone who hasn't met me at the, at the booth yet, uh, my name's Matt Lawler. I'm a forward-deployed engineer at AssemblyAI, uh, mostly focused on bringing in inbound customers, um, helping win their business, um, making sure that they have a good time building with our API. Um, so how, how many other, like, forward-deployed engineers are out in the audience right now? Any of us?

Okay, a couple. Cool. Well, um, still coming through okay? All right. Um, yeah, so I wanted to make sure I, I gave you guys an overview of kinda some of the go-to-market engineering stuff that we've been doing at AssemblyAI that maybe you can apply to your own FDE team, uh, or maybe you can apply to your company, um, as something that you can do as a, an easy win for your customers. Um, so for a little bit of overview of, like, AssemblyAI, in case you haven't heard of us, um, we provide voice AI infrastructure. Um, so we train our own foundation models for speech-to-text primarily, um, but we're

starting to branch into other parts of the voice agent space. Um, so if anyone's ever had, like, a Fireflies notetaker join your meeting or something like that, you've also-- you've already seen a transcript from AssemblyAI. Um, so we train our own foundational models, not only for, like, async use cases, like meeting recordings and things like that, but also for real time, uh, real time, like, voice agents. Um, so if you've ever used, like, a, a phone customer support agent when you called in somewhere and it wasn't a human or, like, drive-through ordering, they're more than likely building with AssemblyAI's models. Um, and for anyone

who's building a voice agent, now you can use us for the entire stack, not just speech to text. Um, so you can use our voice agent API, and we'll string together the speech to text, the LLM, and the text to speech as well. Um, as our competitors know, as we know, as anyone in the voice AI space knows, demand is absolutely insane, um, for voice AI right now. Um, so Assembly, at our current moment, we see around a thousand API signups every single day, um, which is somewhat of the theme of why we were focusing on automation and wanting to build kind of a, a new support bot for our team.

Because at least until, like, two days ago, I was the only, uh, onboarding engineer at AssemblyAI, and so you can imagine, as much as I'd wanna talk to a thousand new signups every single day, it's just not feasible. Um, so we had to build something to, to solve that problem. And then for anyone here who doesn't know, like, what a forward-deployed engineer is, it's a very catch-all title. It varies a lot at every different company. But effectively, we wanna get deployed with our end customers. We wanna understand their use case. We wanna make sure that we are building exactly what they need. Um, so we, we

frequently commit code directly to their repositories. We usually meet with them several times a, a month, if not a week. Um, and we really try to understand their use case and get to know them very personally. Um, so, like, at least on the inbound side, for anyone who comes into Assembly who wants to build with us, we are the people, um, that are their main point of contact for anything they need on the technical, commercial side. Anything that they might need, they get from us, their FDE. Um, and we're responsible for winning their business, um, and building that trust with them.

So it's a great services model. It's great, uh, if we can actually get that kind of relationship with every customer. But again, with, like, a thousand API signups every day, I, I can't be one to many serving that many customers and getting on that many calls. Um, and so ultimately, this doesn't really scale super well. Um, we wanna make sure that we're freeing ourselves up to be leading h- uh, human-led deals as much as possible, but we can't do that with every deal. Not every customer win can be, like, hand-built by us. Um, and we can't just continue to hire more and more FDEs if we're gonna try and serve more and more customers.

Um, so ultimately, like, for anyone who's an FDE in here, if you're looking to become an FDE, if you already have an FDE team at your company, I would encourage you to take that knowledge that you have as an FDE and try to automate yourself out of a job. I think that should be your role every single day. If you wanna serve your customers better, if you wanna work with them more, um, and if you wanna deliver a better customer experience at scale, you can't be the bottleneck to having a good customer experience, and we learned that firsthand. So again, we were drowning in, like, inbound, um, volume. Um, you can see, like, we basically had

a, a thousand API signups up there earlier. Um, and we had the natural instinct that a lot of teams do, where we were like, "We just need to deflect more conversations. I just wanna talk to fewer people. I wanna make sure I'm freeing myself up to work on more important stuff." Um, so we basically bought this off-the-shelf support bot, and we're like, "Let's point it at the docs. It'll answer all the docs questions for us. It's gonna be great. We'll never have to answer a simple question from customers again." And it only resolved about ten percent of the conversations. Um, so it helped. Um, you can imagine if we had, like, a thousand conversations a day, a hundred of those were now answered by a bot. But then our

team still had to field nine hundred tickets a day. So it wasn't exactly very scalable. Um, so ten percent, ten percent, uh, end-to-end resolution rate wasn't good enough. And ultimately, we couldn't iterate fast enough to be able to fix this. We didn't have access to the system prompt. We didn't have access to the tools. We didn't have access to the RAG infrastructure. So any time we needed to change something, we'd go to this vendor, and they'd say, "Well, it's on the roadmap." And that didn't quite work for us. So we wanted to build our own. So we built, uh, another member of our team rather

than hiring one, and we named him Joey. Um, so anyone now who reaches out to us, whether it's on our chat widget on our site, whether you reach out to contact sales, whether you reach out to our support alias, you're gonna meet Joey instead of a human. Um, he's the first line of defense now for every single conversation that happens inbound at Assembly, and he's built to be the most infinitely scalable FDE on our team. Um, so he remembers past conversations with you. Um, we track all of his metrics via Pylon because he's still a member of our support team, and so he has access to Pylon to manage conversations and

remember who you are. Um, and he's built on top of the Claude Agent SDK. And so the cool things that he's able to do is he can manage his own infrastructure. He has a file system. He can write and debug code for you. He can call tools. He can give himself new abilities. Um, and so he's actually able to operate more like an FDE and less like just a standard chatbot that you'd be talking with, you know, on, on most major sites nowadays.

So, um, if you wanna know a little bit of, like, how to build your own Joey, um, the architecture kinda has, like, four major parts. Um, so as everyone might know from working with Claude, Markdown is your friend. You should be giving Markdown files to Claude so that he understands everything about your business. And so we wanted him to have access to all of our documentation and be trained up on it all the time. So we checked all of our docs out as local Markdown and gave it to Joey. Um, and so any time that we push an update to our documentation system, he's able to get all of those files, and he has them locally in his file system.

So actually, what's really cool is if our entire doc site was down right now, you could go to Joey and still get live answers on all the newest stuff that we've shipped without our team having to actually get involved. Um, for retrieval, so we can give you a good answer, we partner with Voyage, so we use their embeddings, um, so that he gets the most relevant documents up front, so he doesn't have to spend a lotta time searching, and he can get you a faster answer. And then since he has all these docs as local files, if he gets confused or he needs to string together multiple resources to create a response, he can agentically search the file system that he lives on so that

he can go and find those resources for you. Um, so he can kinda go into this, like, deep thinking mode, where he's able to search across all these resources. And then we shipped it on Railway. We didn't wanna deal with infrastructure. We didn't wanna spin up an EC2. We didn't wanna have to change instance types, so we deployed it all on Railway. Um, their service is pretty incredible. Basically, we can just go ahead and see if he has a bad conversation with someone in Slack. We can write a PR real quick, and we can deploy it, and within 30 seconds there's a new live version of Joey. So we're able to iterate really, really quickly. Um, we've actually had times where we've actually monitored a live

conversation, caught a bug, deployed a fix, and then it's been approved for the rest of the session, and that customer doesn't even have to know that we, we shipped that fix behind the scenes. Um, so Joey is able to, to basically get updated whenever we want, um, so we actually have full control over everything that, that he does.

Um, and so the results were pretty remarkable from us building this, and it's another reason why I want... and wanted to encourage everyone who's here to try and build your own Joey. So we went from 10% to 80% end-to-end resolution rate in just the first week of deploying this, this build with a pretty naive implementation. And we did that all for around $700 a month in both token and inference, uh, or infrastructure costs. So we resolved 80 e- 80% of our, our tickets end to end with no human in the loop. And some people like to game these metrics. They'll say, like, "Oh, we resolved 80%, but only on, like,

a subset of types of tickets." Not the case here. You cannot reach a human directly unless you go through Joey and you tell him to escalate. So he actually does handle 100% of these inbound tickets for us now. Um, and he only escalates 20% of them to a human for the right reasons. Maybe it's a rate change. Maybe it's a data opt-out. Maybe you need to sign an agreement with us, or you have needs that only an FDE or someone from legal needs to be involved in. These are things that actually still need a human, but eventually we can automate. Um, and so the best part is whatever Joey can't do yet gives us a very clear list

of what we need to do to continue to scale our team. Um, if he's not able to send them a BAA, well, okay, maybe we should give him a link that he can reference as a resource. Um, if he's not able to talk to customers about pricing, well, you can negotiate with Joey. Um, if you go on the site and you test him out afterwards, you can tell him exactly how many hours you have, um, and he'll actually quote you a rate, and then you can negotiate with him. Um, so we tried to automate a lot of these other things that he was still escalating, and there's still, there's still plenty more to come. But the real thing that, you know, we wanted to

test out by building Joey was we wanted to dog food our own product too. We didn't wanna just, like, build something that deflects more stuff from our customers. We didn't wanna build something that just, um, is kinda this general support bot that anyone can build. We work in voice, and our customers build voice agents, so I wanted to build a voice agent. And so I wanted to give a voice to Joey. So now rather than just chatting with him via text, you can actually chat with him using your voice as well. Um, so we have a voice agent API, as I mentioned earlier. Um, we integrated this into Joey this week, so now if you go onto our site, you can switch

to voice mode, and you can talk to Joey using just your voice. Um, 'cause I ultimately think for any FDE here, no matter how much you've been deployed with your customers, no matter how embedded you are with them, the best way to understand what your customers are building is to actually build their same product yourself, and that's exactly what we wanted to do with this. So we put our voice agent API into Joey. Um, it's all speech in, speech out, one WebSocket connection where we string together our speech-to-text model, an LLM, and text-to-speech, all optimized for voice. It's real-time and low latency.

It handles all the pauses, interruption handling, barge in, all of that kinda stuff for you. Um, we bring our own voices so you don't have to find another provider. And it's all the same tools that he already had if you were communicating with him via text. It's just now you can actually talk to him like you are a human. Um, and hopefully later this week we can hook him up to a phone number, so now we'll have an AssemblyAI phone number so you can get real-time support from Joey whenever you need it.

So let's go ahead and talk to Joey a little bit, and we can walk through some of the architecture live. Um, so for anyone who wants to know a little bit more about how this was actually built, um, we basically have all of the, the docs checked out locally in Markdown. Um, so every single time someone puts an update out to our docs, these files also get locally checked out into this repo, so they stay perfectly in sync with any new features that we're shipping, our changelog, our pricing. All of these major, uh, pages from our website get converted to Markdown and get given to Joey. Um, so he

sees all of these in Markdown format, and he's able to see what the URL is so he can cite his sources to the customer in case they wanna double-check his work or if they just want more information. Um, at the same time, um, all of it is provided as, like, CLAUDE.md instructions. I think it's, like, a 30,000-line CLAUDE.md of all these guardrails and, and advice on how he should be operating with customers. Um, and this is what we majorly update whenever we see that he had a bad experience with a customer or gave a wrong answer. We ship an update to this CLAUDE.md, and now for every conversation that he has going forward, he's able to be better.

Um, and it's all contained as a Dockerfile, um, and so it's really easy to go and just containerize and deploy him on Railway. Um, and so again, every time we, we ship an update to this repo, a new version's already live on Railway, and no one had to manually monitor the deployment. Um, so let's go ahead and, uh, I'll fire up Joey here. We can ask him a pretty typical question so you can see what our voice agent API is like

So I'm gonna simulate this locally 'cause I, I know probably many of you are gonna test Joey after this, and I don't wanna add to the queue for our actual support team. Um, so I'll, I'll go ahead and just do a local simulation of it. But typically, yeah, you would go into here, you'd meet Joey, you'd wanna have a typical question. You might talk to him via text, but let's go ahead and talk to him with voice.

Hi, this is Joey at AssemblyAI. How can I help you today?

Hey, Joey. Uh, I'm looking to build an ambient medical scribe with AssemblyAI, but before I can test, I need to sign a BAA. Is that something that you can do?

One sec. Let me check that for you.

Yes. You can execute the BAA yourself right away. Just complete our standard HIPAA business associate agreement. But note that you'll need a paid account with a card on file first, and signing the BAA automatically opts you out of model training. The relevant docs are linked in the chat.

Cool. Thanks, Joey. Well, I gotta, I gotta get back to my demo. Uh, I'd love to chat more, but uh, thanks for all the help.

You are very welcome. Good luck with your demo, and feel free to reach out if you need anything else later. Have a great day.

Isn't he so nice? I enjoy talking to Joey. Um, I think our customers will enjoy talking to Joey. Um, but this is a great example of, like two years ago, when I first joined Assembly on our support team, this is a conversation that would have to be handled by a human. This is something that if you reached out in, in chat and a, a customer, uh, or a support engineer wasn't able to, uh, answer you immediately, they'd have to wait. They'd have to get a response from us. It was a canned response. Um, it was just not a great customer experience. And now you have an FTE that you can talk to twenty-four/seven for these basic questions. Um, and you can do

it through voice, so you can actually test our product at the same time. Um, so there's, there's kind of a meta layer there where if you're talking to Joey about how to build a voice agent API, you're already using it, um, and you're already seeing exactly what your product experience could look like.

So if there's anything that you should get out of this talk, it's not just that you should build your own Joey, um, but I think that it's like if you're, if you're an FTE at your company, you need to be dogfooding your own product all the time. It's not enough to just be teaching your customers how to get started with it. Any new feature that you launch, any new product that you offer, you need to be teaching your customers, um, with all the knowledge that you've built from actually using and building and trying to ship the same product. Um, so I had to run through all the same walls. I had to figure out how to deal with latency. I had to figure out how to deal with turn-taking or, or barging or interruption handling.

Um, and I think I have a lot better empathy of how to, like, work with customers that you wanna use our voice agent API, um, because now I've actually used it. I've used the same API. It's the exact one that my customers build on, and I can give them better advice because I've actually shipped something with it.

So go build. Go build something. Go build whether it's a, an, your own Joey. Go build whether it's a, a voice agent, especially if it's a voice agent, build it on AssemblyAI. Um, but if you're, if you're actually working in an FTE capacity, try to build the exact product that your customers are building and do it on your own. Don't just wait for them to ask you for help. Don't just wait to remove all their blockers. You should be going out and building the exact same thing, and at the same time, you might actually be able to make your customer experience better simultaneously. So, um, if anyone has, like, questions on how to build Joey, if you wanna check him out at the booth

afterwards, our booth is just right over here. Um, if anyone has feedback on Joey, since this just shipped this week, if you have a weird conversation with him, I'd love to know about it so we can keep making fixes, and I can show you how we deploy that live. Uh, but overall, really excited to, uh, be here at the conference and talk to more people about voice agents. So thanks for, uh, coming and listening to this talk.