12-Factor Agents: Patterns of reliable LLM applications

[upbeat music] Who here is building agents?

Who here is-- leave your hand up if you built, like, ten plus agents. Anyone here built, like, a hundred agents? All right, we got a few. Awesome. Love it.

Um, so I think a lot of us have been on this journey of building agents. Um, and what happened with me was, you know, I decided I wanted to build an agent.

We figured out what we wanted it to do. Uh, we wanna move fast. We're developers, so we use libraries. We don't write everything from scratch. Um, and you get it to, like, seventy, eighty percent.

It's enough to get the CEO excited and get six more people added to your team. But then you kinda realize that seventy, eighty percent isn't quite good enough, and that if you wanna get past that seventy, eighty percent quality bar, you're seven layers deep in a call stack trying to reverse engineer how does this prompt get built

or how do these tools get passed in? Where does this all come from? Uh, and if you're like me, you eventually just throw it all away and start from scratch.

Um, or you maybe even find out that this is not a good problem for agents. I remember one of the first agents that I tried to build was, uh, a, a DevOps agent.

I was like, "Here's my Makefile. You can run make commands. Go build the project." Couldn't figure it out. Did all the things in the wrong order. I'm like, "Cool, let's fix the prompt."

And I-- Over the li-- next two hours, had more and more detail about what everything was and every single step and the exact-- And it's like I got to the point where I was like, "This is the exact order to run the build steps."

It was a cool exercise, but at the end of it, I was like, "You know, I could have written the bash script to do this in about ninety seconds."

Not every problem needs an agent. Um, and so I've been on this journey. I think a lot of you have been on similar journeys. Um, and what happened was is I went and talked, um, in trying to help people build better, more reliable agents.

I talked to a hundred plus founders, builders, engineers, um, and I started to notice patterns. One was that most production agents weren't that agentic at all. They were mostly just software.

But that there were these core things that a lot of people were doing. There were these patterns that were making their LLM-based apps really, really good. Um, and none of them were doing kind of a greenfield rewrite.

Rather, they were taking these small modular concepts that didn't have names and didn't have definitions, and they were applying them to their existing code. Um, and what's really cool about this is I don't think you need an AI background to do this.

This is software engineering 101. Well, probably not 101. But just like Heroku cr-- needed to define what it meant to build cl-- We didn't even call them cloud native back then, but this was how you built applications that could run in the cloud ten years ago.

Um, I decided to put together what I thought would be the twelve factors of AI agents, um, from everything that I've seen working in the field. Um, so we put up this GitHub repo.

You can go read it. Um, turns out a lot of other people agreed and felt the same thing. Um, so we were on the front page of Hacker News all day, two hundred K impressions on social.

Uh, I'm just gonna put this one up and no comment.

Um, and just for context, we got to, like, four thousand stars in, like, a month or two. Uh, there's fourteen active contributors. Um, it's very easy to read that thing and, and hear the talk and say like, "Oh, we're here.

This is the anti-framework talk." I am not here to bash frameworks. I would think of this as much as anything as a wish list, a, a list of feature requests, is how can we make frameworks serve the needs of really good builders who need a really high reliability and wanna move fast still?

Um, so what am I here to do? Uh, I want you to kind of forget everything you know about agents and kind of rethink from first principles how we can apply everything we've learned from software engineering to the practice of building really reliable agents.

Um, so we're gonna mix the order up a little bit. If you want all twelve factors in order, that's a thirty-minute talk, so we're gonna bundle some stuff together.

There will be a QR code at the end. You can go dig through it, uh, at your leisure. Um, factor one, the most magical things that LLMs can do has nothing to do with loops or switch statements or code or tools or anything.

It is turning a sentence like this into JSON that looks like this. Doesn't even matter what you do with that JSON. Uh, those are what the other factors are for.

But if you're doing that, that's one piece that you can bring into your app today. Um, factor four, this leads right to, uh... Did anyone read this paper, uh, "GOTO Considered Harmful," or maybe just heard about it?

I never actually read it. Uh, but it was all about we had this abstraction in the C programming language and a bunch of other programming languages at the time that said, "This thing GOTO, it makes code terrible.

It's the wrong abstraction. It-- No one should use it." I'm gonna go ahead and go out on a limb here and say tool use is harmful. And I put it in quotes because I'm not talking about giving an agent access to the world.

Obviously, that's super badass. But what I think is making things hard is the idea that tool use is this magical thing where this ethereal alien entity is interacting with its environment.

Because what is happening is our LLM is putting out JSON. We're gonna give that to some deterministic code that's gonna do something, and then maybe we'll feed it back.

But again, those are other factors. So if you have structures like this and you can get the LLM to output something that generates them, then you can pass it into a loop like this or a switch statement like this.

There's nothing special about tools. It's just JSON and code. Uh, that's factor four. Factor eight, and these are-- we're gonna do a couple kind of bundled together here. Owning your control flow.

Um, and I wanna take a step back and kinda talk about how we got here. Um, we've been writing DAGs in software for a long time. If you've written an if statement, you've written a directed graph.

Uh, code is a gr-graph. You may also be familiar with DAG orchestr-- Anyone ever use, like, Airflow or Prefect or any of these things? Um, so, like, this kind of concept of breaking things up into nodes gives you certain reliability guarantees.

But what agents were supposed to do, and I think a lot of people talk about this, and I think in some cases this is realized, is you don't have to write the DAG.

You just tell the LLM, "Here's the goal," and LLM will find its way there. And we model this as a really simple loop. You know, your LLM is determining the next step.

You're building up some context window until the LLM says, "Hey, we're done."

Um, so what this looks like kind of in practice is, you know, you have an event come in, you pass it into your prompt, uh, it says you wanna call an API, and you get your result, put that on the context window, pass the whole thing back into the prompt.

This is like the most naive, simple way of building agents. And the LLM's gonna call a couple steps, and then eventually it's gonna say, "Cool, we've done all the tasks from the initial event," which maybe was a user message asking it to do something, maybe it's an outage, um, but then we get our final answer.

And our materialized DAG is just these three steps in order. Um,

turns out this doesn't really work, uh, especially when you get to longer workflows. Mostly it's long context windows. There's o- other reasons you could poke at as well. Um, and people say, "Oh, like even as...

Like anyone put like two million tokens into Gemini before and like tried to see what happens?" Like you can do it. You'll get an answer. The API will return you something.

But I don't think anyone will argue with you that you will always get s- like tighter, better, higher reliability results by controlling and limiting the number of tokens you put in that context window.

Um, so it doesn't quite work, but we're gonna use that as our abstraction to build on. What's an agent really? You have your prompt, which gives instructions about how to select the next step.

You have your switch statement, which takes whatever the mo- model output JSON, uh, and does something with it. You have a way of building up your context window, and then you have a loop, uh, that determines when and where and how and why you exit.

Um, and if you own your control flow, you can do fun things like break and switch and summarize and LLM-as-judge and all this stuff. Um, and this leads right into kind of how we manage execution state and business state of our agents.

Um, a lot of tools will give you things like current step, next step, retry counts, all these like DAG orchestrators. They'll have these kind of concepts in them. Um, but you also have your business state.

What are the messages that have happened? What data are we displaying to the user? What things are we waiting on approval for? Um, and we wanna be able to launch, pause, resume these things like we do for any standard APIs.

Um, this is all just software. And so if you can put your agent behind a REST API or an MCP server, um, and manage that loop in such a way that normal request comes in and we load that context window to the LLM, um, we're gonna allow our agent to call long-running tool.

So we can interrupt the workflow, serialize that context window straight into a database, 'cause we own the context window. We'll get into that. Um, and then when we launch the workflow, um, eventually it's gonna call back with that state ID and the result.

We use the state ID to load the state back out of the database, and then we can append the result to the prompt and then send it right back into the LLM.

The agent doesn't even know that things happened in the background. Um, agents are just software, so let's build software. Um, and building really good ones ne- requires a lot of flexibility, and so you really wanna own that inner loop of, of how all that stuff is fitting together.

Um, that's unifying. That's pause and resume. Uh, factor two, this one is I think most people find first, is like you really wanna own your prompts. There's some good abstractions that if you don't wanna spend a lot of time handwriting a prompt, you can put stuff in and you'll get out, um, a really good set of primitives,

um, and a really good prompt. Like this will make you a banger prompt that like you would have to go to prompt school for like three months to build a prompt this good.

But eventually, if you wanna get past some quality bar, you're gonna end up writing every single token by hand. Um, because LLMs are pure func- functions, and the only thing that determines the reliability of your agent is how good of tokens can you get out.

And the only way-- The only thing that determines the tokens you get out other than like retraining your own model and something like that is being really careful about what tokens you put in.

Um, I don't know what's better. I don't know how you want to build your prompt, but I know the more things you can try and the more knobs you can test and the more things you can evaluate, the more likely you are to find something really, really good.

Um, owning your prompts. You also wanna own how you build your context window. Um, so you can do the standard OpenAI messages format, or in this moment where you're telling the LLM, pick the next step, your only job is to tell it what's happened so far.

You can put all that information however you want into a single user message and ask, "Hey, what's happening next?" Or put it in the system message. So you can model your event state, your thread model however you want, um, and stringify it however you want, and some of the traces that we use and some of the agents

we build internally, I'll get into that in a sec, um, might look like this. Um, but if you're not looking at every single token and if you're not optimizing the density and the clarity of the way that you're passing information to an LLM, you might be missing out on upside on quality.

So LLMs are pure functions. Token in, tokens out, and everything, everything in making agents good is context engineering. So you have your prompt, you have your memory, you have your RAG, you have your history struct.

It's all just how do we get the right tokens into the model so it gives us a really good answer and solves the user's problem. Solves my problem mostly.

Um, I don't know what's better, but I know you wanna try everything, um, so that's own your context building. Um, this one's a little controversial. Uh, that's why it's a, it's a standalone factor, and the way you make it good is by integrating it with other factors.

But you could, um, when the model screws up and it calls an API wrong or it calls an API that's down, um, you could take the tool call that it made and grab the error that was associated with it, put that on the context window, and have it try again.

Anyone ever had a bad time with this?

Seeing like this thing just like kinda spin out and like go crazy and lose context and just get stuck? Um, that's why you need to own your context window.

Don't just blindly put things on. If you have errors and then you get a valid tool call, clear all the error-- pending errors out. Summarize them. Don't put the whole stack trace on your context.

Figure out what you wanna tell the model so you get better results. Um, contacting humans with tools. This one's a little subtle, um, but I've seen... This is just like what I've seen in the wild.

Almost everybody is like avoiding this very important choice at the very beginning of output where you're deciding between tool call and message to the human. Um, if you can push that emphasis to a natural language token, you can, one, give the model different ways.

You can be, "I'm done," or, "I need clarification," or, "I need to talk to a manager," or whatever it is. And two, you push the intent on that first token generation and the sampling to something that is natural language that the model understands.

Um, so your traces might look like this if you're pulling in human input here. Um, this lets you build outer-- outer loop agents. I'm not gonna talk about this.

If you go on the site, there's a link to this, this post. I've written a lot about this. Um, I don't know what's better, but you should probably try everything.

Um, that's contacting humans with tools. It goes right along with trigger things from anywhere and meet users exactly where they are. People don't wanna have seven tabs open of different ChatGPT-style agents.

Just let people email with the agents you're building and let them Slack with the agents you're building, Discord, SMS, whatever it is. We see this taking off all over the place.

Um, and you should have small focused agents. So we talked about this structure and why it doesn't really work. So what does work? Um, the things that people are doing that work really well are micro agents.

So you still have a mostly deterministic DAG, and you have these very small agent loops with like three to ten steps. We do this at HumanLayer. We have a bot that ran- manages our deployments.

Most of our deploy pipeline is deterministic CI/CD code. But when we get to the point where the GitHub PR is merged

and the tests are passing on development, excuse me,

we send it to a model and say, "Get this thing deployed." It says, "Cool, I'm gonna deploy the front end." Uh, and then you can send that to a human.

A human says, "Actually, no, do the back end first." This is taking natural language and turning it into JSON. That is the next step in our workflow. Um, back end gets proposed, that gets approved, it gets deployed.

Then the agent knows, okay, I have to go back and deploy the front end. Once that's all done and it's successful, we go right back out into deterministic code.

So now we're gonna run the end-to-end test against prod if it's done. Otherwise, we hand it back to a little rollback agent that is very similar on the inside.

Um, I'm not gonna go into it, but here's it working in our Slack channel. Um, yeah, hundred tools, twenty steps, easy, um, manageable context, clear responsibilities. Um, a lot of people say, "What, what, what if LLMs do keep getting smarter?

What if I can put two million tokens in and it can do it?" Um, and I think we very much will see something like this where you start with a mostly deterministic workflow and you start sprinkling LLMs into your code, into your back end, into your logic.

Over time, the LLMs are able to do bigger, more complex tasks until this whole API endpoint or pipeline or whatever it is, is just run by an agent. That's great.

Uh, but you still wanna know how to engineer these things to get the best quality. This is someone from NotebookLM, and it's basically their take, and I think they did this well, is find something that is right at the boundary of what the model can do reliably, like that it can't get right all the time.

And if you can figure out how to get it right reliably anyways because you've engineered reliability into your system, then you will have created something magical, and you will have created something that's better than what everybody else is building.

So that's small focused agents. There's a meme here about stateless reducers. I guess someone actually tweeted at me, it's not a reducer, it's a transducer because there's multiple steps.

Um, but basically, agents should be stateless. You should own the state, manage it however you want. Um, so we're all still finding the right abstractions. Um, there's a couple blog posts I link in the paper, um, w- frameworks versus libraries.

There's a really old one from RubyConf about like, do we want duplication or is like, do we wanna fi- try to figure out these abstractions? Um, if you wanna make a Twelve-Factor agent, we are working on something called Create Twelve-Factor Agent because I believe that what agents need is not Bootstrap.

You don't need a wrapper around an internal thing. You need something more like shadcn, which is like scaffold it out, and then I'll own it, and I'll own the code and I'm okay with that.

So in summary, agents are software. You all can build software. Anyone ever written a switch statement before? [laughing]

While loop? Yeah. Okay, so we can do this stuff. LLMs are stateless functions, which means just make sure you put the right things in the context, and you'll get the best results.

Own your state and your control flow and just do it and just understand it 'cause it's gonna give you flexibility. And then find the bleeding edge. Find ways to do things better than everybody else by really curating what you put into the model and how you control what comes out.

Um, and my take, agents are better with people. Find ways to let agents collaborate with humans. Um, there are hard things in building agents, but you should probably do them anyways, at least for now, and you should do most of them.

Um, I think a lot of frameworks try to take away the hard AI parts of the problem so that you can just kinda drop it in and go, and, uh, I think it should be the opposite.

I think the tools that we get should take away the other hard parts so that we can spend all our time focusing on the hard AI parts, on getting the prompts right, on getting the flow right, on getting the tokens right.

So the reason why I'm here is like I do run a small business. Um, we have a startup where we try to help you do... A lot of what we do, we do in the open is open source, and I think it's really important, and we need to work on it together.

There's some other things that are hard but not that important and not that interesting, so that's what we're solving at HumanLayer. Um, working on something called A2H Protocol. Come find me if you wanna talk about this, but this is a way to get like consolidation around how agents can contact humans.

Um, but mostly I just love automating things. I've built tons and tons of agents internally for my personal stuff, for finding apartments, for all kinds of internal business stuff we do at HumanLayer.

Um, so thank you all for watching. Let's go build something. I'll see you in the hallway track. Uh, I'd love to chat if you wanna riff on agents or building or control flow or any of this stuff.

That's Twelve-Factor Agents. [audience clapping] [upbeat music]