Pydantic is all you need

[upbeat music] Hey, guys.

So I didn't know I was gonna be one of the keynote speakers, so this is probably gonna be the most reduced scope talk of today. [laughing] I'm talking about type hints. [laughing]

And in, in particular, I'm talking about how Pydantic might be all you need to build with language models. In particular, I wanna talk about structured prompting, which is the idea that we can use objects to define what we want back out rather than kind of praying to the LLM gods that the comma is in the right place

and the bracket was closed. So everyone here basically kinda knows or at least agrees that large language models are kind of eating software. But what this really means in production is ninety percent of the applications you build are just ones where you're asking the language model to output JSON or some structured output that you're parsing with a

regular expression and, and that experience is pretty terrible. And the reason this is the case is because we really want language models to be backwards compatible with the existing software that we have.

You know, codegen works, but a lot of the systems we have today are systems that we can't change.

And so, yeah, the idea is that although language models were introduced to us through ChatGPT, most of us are actually building systems and not chatbots. We wanna process input data, integrate with existing systems via APIs or schemas that we might not have control over.

And so the goal for today is effectively introduce OpenAI function calling, introduce Pydantic, then introduce Instructor and Marvin as a library to make u- using Pydantic to prompt language models much easier.

And what this gets us is, uh, you know, better validation, makes your code a little bit cleaner, and then afterwards I'll talk over some design patterns that I've uncovered and some of the applications that we have.

Um, this is basically almost everyone's experience here, right? Like, you know, Riley Goodside had a tweet about asking to get JSON out of Bard, and the only way you could do it was to threaten to take a human life, and that's not code I really wanna commit into my repos. [laughing]

And then when you do ask for JSON, you know, maybe it works today, but maybe tomorrow instead of getting JSON you're gonna get like, "Okay, here you go. Here's some JSON."

And then again, you kind of pray that the JSON's parsed correctly. And I don't know if you noticed, but here user is a key for one query and username is a key for another, and you would not really notice this unless you had, like, good logging in place.

But really, this should not happen to begin with, right? Like, you shouldn't have to, like, read the logs to figure out that the passwords didn't match when you're signing up for an account.

And so what this means is our prompts and our schemas and our outputs are all strings. We're kind of writing code in TextEdit rather than an IDE where you could, you know, get linting or type checking or syntax highlighting.

And so OpenAI function calls somewhat fix this, right? We get to define a JSON Schema of the output that we want, and OpenAI will do a better job in placing the JSON somewhere that you can reliably parse out.

So instead of going from string to string to string, you get string to dict to string, and then you still have to call json.loads, and again, you're kind of praying that everything is in there.

And a lot of this is kind of praying to the LLM gods. Um, on top of that, like, if this code was committed to any repo I was managing, like, I would be pissed, [chuckles] right?

Complex data structures are already difficult to define, and now you're working with the dictionary of json.loads, and that also feels very unsafe 'cause you get missing keys, missing values, and you get hallucinations, then maybe the keys are spelled wrong and you're, you're missing an underscore, and you get all these issues.

And then you end up writing code like this. And this works for, like, name and age and email, then you're checking if something is a bool by parsing a string.

It gets really messy and, and what H- Python has done to solve this is use Pydantic.

Pydantic is a library that do data model validation very similar to dataclasses. It is powered by type hints. It is, has really great model and field validation. It has seventy million downloads a month, which means it's a library that everyone can trust and use and know that it's gonna be maintained for a long period of time.

And more importantly, it outputs JSON Schema, which is how you communicate with OpenAI function calling. And so the general idea is that we can define an object like delivery, say that the timestamp is a datetime and the dimensions is a tuple of ints.

And even if you pass in a string as a timestamp and a list of strings as tuples, everything is parsed out correctly. This is all the code we don't wanna write.

This is why there's seventy million downloads. More interestingly, timestamp and dimensions are now things that your IDE is a-aware of. They know the type of that. You get autocomplete and spell checking.

Again, just more bug-free code. And so this really wa- brings me to the idea of structured prompting, 'cause now your prompt isn't a, you know, triple quoted string. Your prompt is actual code that you can look at, you can review.

And everyone has written a function that returns a data structure, right? Everyone knows how to manage code like this instead of doing the migration of JSON schemas in the one-shot examples.

You know, I've done database migrations. I know how some of these things work. And more importantly, we can program this way. And so that's why I built a library called Instructor a while ago, and the idea here is just, just to make OpenAI function calling super useful.

So the idea is you import instructor, you patch the completion API. Uh, debatable if this is the best idea. But ultimately, you define your Pydantic object, you set that as the response model of that create call, and now you're guaranteed that that response model is the type of the en-entity that you extract.

So again, you get nice autocomplete, you get type safety. Really great.

I would also wanna mention that this only works for OpenAI function calling. If you wanna use a more, uh, comprehensive framework to do some of this Pydantic work, I think Marvin is a really great, uh, library to try out.

Uh, they, they give you access to more, uh, language models and more capabilities above this, uh, response.

But the general idea here isn't that this is gonna make your JSON come out better, right? The idea is that when you define objects, you can define nested references, you can define methods of the behavior of that object, you can return instances of that object instead of dictionaries.

And you're gonna write cl- cleaner code and code that's gonna be easier to maintain as they're passed through different systems.

And so here you have, for example, a BaseModel, but you can add a method if you want to. You could define the same class but with an address key.

You can then define new classes like best friend and friends, which is a list of user details. Like, if I was to write this in JSON Schema to make a POST request, it would be very unmanageable, but this makes it a lot easier.

On top of that, when you have docstrings, the docstrings are now a part of that JSON Schema that is sent to OpenAI.

And this is because the model now represents both the prompt, the data, and the behavior all in one, right? You want good docstrings, you good want f- you good-- You want good field descriptors, and it's all part of the JSON Schema that you send.

And now your code quality, your prompt quality, your data quality are all in sync. There's just one thing you wanna manage and one thing you wanna review. And what that really means is that you need to have good variable names, good descriptions, and good documentation.

And this is something we should have anyways.

You can also do some really cool things with Pydantic without language models. For example, you can define a validator. Here I define a function that takes in a value.

I check that there is a string in that value, and if it's not, I return a lowercase version of that 'cause that just might be how I wanna parse, parse my data.

And when you construct this object, you get an error back out, right? We're not gonna fix it, but we get a validation error, something where we can catch reliably and understand.

But then if you introduce language models, you can just import the LLMValidator, and now you can have something that says like, "Don't say mean things." And then when you construct an object that has something that says that the meaning of life is to be evil and steal things, you're gonna get a validation error and an error message.

And this error message, "The statement is objectionable," is actually coming out of a language model API call. It's using Instructor under the hood to define that.

But, you know, it, it's not enough to actually just point out these errors. You also wanna fix that. And so the easy way of doing that in Instructor is to just add max_retries,

right? Now what we do is we'll append the err- the message that you had before, but then we can also capture all the validations in one shot, send it back to a language model, and try again, right?

But the idea here that this isn't like prompt chain, this isn't, this isn't constitutional AI. Here we just have validation, error handling, and then re-asking, and these are just separate systems in code that we can manage.

If you want something to be less than ten characters, there's a character count validator. If you wanna make sure that a name is in a database, you can just add a POST request if you want to.

But this is just classical code again. This is the backwards compatibility of language models.

But we can also do a lot more, right? Uh, structured prompts get you structured outputs. But ideally, the structure actually helps you structure your thoughts. So here's another example.

Uh, it's really important for us to give language models the ability to have an escape hatch and say that it doesn't know something or can't find something. And right now, most people will say something like, "Return I don't know in all caps.

Check if I don't know all caps in string." Right? Uh, sometimes it doesn't say that. It's very difficult to manage. But here you see that I've defined user details with an optional role that could be None.

But the entity I wanna extract is just maybe a user. It has a result that's maybe a user and then an error and an error message. And so I can write code that looks like this.

I get this object back out. It's a little bit more complicated, but now I can kind of program with language models in a way that feels more like programming and less like chaining, for example.

Right? Um, we can also define reusable components. Here I've defined a work time and a leisure time as both a time range, and the time range has a start time and an end time.

If I find that this is not being parsed correctly, what I could do is actually add chain of thought directly in the t- the time range component. And now I have modularity in how, some, in some of how, uh, in some of these features.

And you can imagine having a system where in production you, uh, disable that chain of thought field and then in put, in, in testing you add that to figure out what's the latency or performance trade-offs.

You could also extract arbitrary values, right? Here I define a property called key and value, and then I wanna extract a list of properties, right? You might wanna add a prompt that says, "Make sure the keys are consistent over those properties."

We can also add validators to make sure that's the case and then re-ask when that's not the case. If I want, you know, only five properties, I could add an index to the property key and just say, "Well, now count them out.

And when you count to five, stop." And you're gonna get much more reliable outputs.

Uh, some of the things that I find really interesting with this kind of method is prompting data structures. Here I have user details, age, name as before, but now I define an ID and a friends array, which is a list of IDs.

And if you prompt that well enough, you can basically extract like a network out of this data str- out of your data.

So, you know, we've seen that structured prompting kinda gives you really useful components that you can reuse and make modular. Um, and the idea again here is that we wanna model both the prompt, the data, and the behavior.

Here I haven't mentioned too many methods that you could act on this object, but the idea is almost like, you know, when we go from C to C++, the thing we get is object-oriented programming, and that makes a lot of things easier.

And we've learned our lessons with object-oriented programming. And so if we do the right track, uh, I think we're gonna get a lot more productive development out of these language models.

And the second thing is that these language models now can output data structures, right? That you can like pull up your old like LeetCode textbooks or whatever and, and actually figure out, like, traverse these graphs, for example, process this data in a useful way.

And so now they can represent, you know, knowledge, workflows, and even plans that you can just dispatch to a classical computer, uh, computer system, right? You can create the data that you wanna send to Airflow rather than doing this for loop hoping it terminates.

And so now I think I have about six minutes, so I'll go over some advanced applications. Um, these are actually fairly simple. I have some more documentation if you wanna see that later on, but, um, let's, let's go over some of these examples.

So the first one is RAG. I think-

When we first started out, a lot of these systems end up being systems where we embed the user query, make a vector database search, return the result, and then hope that those are good enough.

But in, in practice, you might have multiple backends to search from. Maybe you want to rewrite the user query. Maybe you want to decompose that user query, right? If you want to ask something like what, what was something that was recent, you need to have time filters.

And so you could define that as a data structure, right? The search type is email or video. Search has a title, a query, a before date, and a type.

And then you can just implement the execute method that says, you know, if type is video, do this. If email, do that. Really simple. And then what you want to extract back out is multiple searches.

Like, give me a list of search queries. And then you can write some, like, asyncio to map across these things.

And now, because all the prompting is embedded in the data structure, your prompt that you send to OpenAI is very simple. You're a helpful assistant, segment the search queries.

And then what you get back out is this ability to just have an object that you can program with in a way that you've managed sort of, like, all your life, right?

Something very s- straightforward. But you can also do something more interesting. You can then plan, right? Before we talked about, like, extracting a social network, but you can actually just produce the entire DAG.

Here, I had the same graph structure. All right? It's an ID, a question, and a list of dependencies where I have a lot of information in the description here, and that's basically the prompt.

And what I want back out is a query plan.

So now, if you send it to a query planner that says, like, you're a helpful query planner, like, build out this query. You can ask something like, what is the difference in populations of Canada and Jason's home country?

And then what you can see is, you know what, like,

if I'm good at LeetCode, I could query the first two in parallel because there are no dependencies, and then wait for dependency th-three to merge, and then wait for four to merge those two.

But this requires one language model call, and now it's just traditional RAG. And if you have an IR system, you get to skip this for loop of agent queries.

You know, an example that was really popular on Twitter recently was extracting knowledge graphs. You know, same thing here. Here, what I've done is I've made sure that the data structure I model is as close as possible to the Graphviz, uh, visualization API.

What that gets me is really, really simple code that does basically the creation and visualization of a graph. I've just defined things one-to-one to the API, and now what I can do is if I ask for something that's very simple, like, you know, give me the description of quantum mechanics, you can get a graph out, right?

That- that's basically in, like, 40 lines of code because what you've done is you've modeled the data structure Graphviz needs to make the visualization. And we're, we're kinda trying to couple that a lot more.

This is a more advanced example, so don't feel bad if you can't follow this one. But here what I've done is I've done a QuestionAnswer. It's a question and an answer, and the answer is a list of facts.

And what a fact is, is it's a fact as a statement and a substring quote from the original text. I want multiple quotes as a substring of the original text.

And then what my validators do is it says, you know what? For every quote you give me, validate that it exists in the text chunk. If it's not there, throw out the fact.

And then the validator for QuestionAnswer says, only show me facts that have at least one substring quote from the original document. So now I'm trying to encapsulate some of the business logic of not hallucinating, not by asking it to not hallucinate, but actually trying to figure out, like, what is the, like, para- like, the paraphrasing detection

algorithms to, to identify the, what the quotes were. And what this means is instead of being able to say that the answer was in page seven, you can say the answer was this sentence, that sentence, and something else, and I know they exist in the text chunks.

And so I think what we end up finding is that, uh, as language models get more interesting and more capable, we're only gonna be limited in, in the creativity that we can have to actually prompt these things, right?

Like, you can have instructions, uh, per object. You can have, like, recursive structures, right? It, it, it goes into domain modeling more than it goes to prompt engineering. And again, now we can use the code that we've always used.

If you want more examples, I have a bunch of examples here on different kinds of applications that I've had with some of my consulting clients. Um, yeah, I think these are some really useful ones.

And I'll go to the next slide, which is... [gasps]

This doesn't have the P-- uh, the QR code. That's fine. [laughs]

The updated slide has a QR code, but instead you can just visit jxnl.github.io/instructor. I also wanna call out that, uh, we're also experimenting with a lot of different UIs to do this structured evaluation, right?

Where, um, you might wanna figure out whether or not one response was mean, but you also want to figure out what the a- distribution of floats was for a different attribute and be able to write evals against that.

And I think there's a lot of really interesting open work to be done, right? Like, right now we're doing very simple things around extracting graphs out of documents. You can imagine a world where we have multimodal, in which case you could be extracting bounding boxes, right?

Like, one application I'm really excited about is being able to say, given an image, draw the bounding box for every image and the search query I would need to go on Amazon to buy this product.

And then you can really instantly build a UI that just says, you know, for every bounding box, render a modal, right? You can have, like, generative UI over images, over audio.

I think in general it's gonna be a, a very exciting space to play more with, uh, structured outputs. Thank you. [audience applauding] [upbeat music]