Advanced: Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

[upbeat music] Hello, guys.

Hello, hello, hello. Yes, sorry for being a bit late. There's a lot of traffic, [laughs] but, um, hello. Um, welcome to AI Engineer World's Fair. Thanks for coming to my session.

Um, and yes, today we're gonna talk about the deep dive into RL, kernels, agents, and quantization. Um, you might know me or you might not know me, but I'm Daniel.

Uh, my brother is somewhere. Uh, yeah, somewhere. But yes, um, thanks for coming again. Oh, we also have stickers and other, like, random stuff later. Um, that's after the talk.

Um, so maybe you might know me, maybe you might not. Um, so we, um, on Twitter, we tweet a lot. Um, we did like a gradient accumulation bug fix last year.

Um, we introduced something called Async offloaded gradient checkpointing. Um, we also work with the Hugging Face, Google, Meta, Mistral teams to, like, fix bugs in, like, their open source models like Gemma, Llama, Mistral, Phi, and more.

Um, yeah, so like we... If you wanna follow on the latest [laughs] stuff of AI, better follow us. Um, we tweet about random stuff. Um, you might even know when next models might be released.

You know, we sometimes tell people approximately, um, so that might be very interesting. Um,

we also do, like, open source contributions to the entire open source ecosystem. For example, we contribute sometimes to llama.cpp. Um, we work with the Qwen team and Mistral on their releases.

Um, we also do like, you know, Phi-4 bug fixes, Llama-4 bug fixes, which increase accuracy by a bit. Um, so definitely, you know, utilize some of the new, new uploads that we do, um, which fix bugs all the time.

We just surpassed ten million monthly downloads on Hugging Face. Um, and yeah, we also have a GitHub package, um, with, with forty thousand GitHub stars. Um, and we do like...

Essentially, we make fine-tuning faster and reduce memory usage.

And yeah, that's the GitHub package. Definitely check that out. Um, there's like free Colab notebooks. I'm not sure many people know, but you have like free GPUs that Google offers.

You can just use them. Um, please use them more. Um, and Kaggle, if not many people know, have thirty hours for free of GPUs per week, right? And there's no restrictions on it.

You know, please utilize the free re- resources as much as possible. Um, yes, they won't be unhappy. Just use them all. [laughs] So yes, we have notebooks, so if you scroll down a bit on our GitHub pa- uh, page, there's like free notebooks, um, for C- Colab, Kaggle.

We do reasoning, um, continued pre-training, supervised fine-tuning, and other stuff.

We also upload models to our Hugging Face page. Um, for example, DeepSeek-R1-0528 released a few days ago. We upload 1.58-bit quants, which are like very small.

They retain most of the accuracy, um, so, like, these can run on your local device. Um, even if you have, like, very low V- VRAM or, like, a not very good GPU, um, it will still work.

And we will constantly upload models, so, like, sometimes people complain to us, you know, "Please stop uploading fixed models," you know, "It's kind of annoying." Um, but too bad, you know.

Unfortunately, models have bugs, so we do have to fix them immediately. You will see sometimes, for example, the accuracy can increase by ten percent. Um, sometimes, you know, the large model providers won't tell you that they uploaded a fix.

They're not gonna tell you. But, you know, be sure to download the latest models. You will get all the fixes.

Now today, let's start off from history, right? Does everyone remember Llama? Um,

although finally it got leaked, right? It was just a research paper, you know, Meta saying, "Oh, we trained Llama. Where's the weights?" You know, it's only research access, and then suddenly it got leaked, and it kind of just kind of spawned the entire open source movement.

Um, you know, some of the people who are the, who are the authors, you know, are not part of Meta anymore, but Llama is extremely important for the entire ecosystem, and it was like the beginning of open source, kind of, um, for large language models.

The most famous plot from the paper is this, right? So, like, you know, if you keep making the model train more, it just, the loss just keeps going down.

Um, well, the question is, will the loss keep going down? That's the question, right? So Llama One was only trained at one point four trillion tokens, right? So now, one point four trillion tokens is actually very less.

Um, most models are trained ten times more. Um, and you can also see from the trend that the bigger the model, the lower the loss, right? So seven billion is the blue line, um, and then sixty-five billion was, like, the red line, and you can see that, in general, as the model gets bigger, it, it gets smarter.

Um, the training loss, the numbers are correct, so you should see generally these numbers from around maybe a bit higher than one. Um, if you see training losses when you do fine-tuning of, like, eight, thirteen, thirty, definitely something's wrong.

Um, so you should get at least losses around two-ish, three-ish.

And so now, uh, now Google, um, you know, Google's new Gemma 3 models are trained on fourteen trillion tokens, right? Which is much more. Um, Llama 4 is trained on thirty trillion tokens, right?

So, like, literally, like, at least ten times more. Um, so Gemma is ten times more. Llama's, like, you know, thir- uh, thirty times more.

Oh, yes, I forgot. You can also access the slides via the QR code if you want. It's also on the docs as well. Um, so if you go to the docs, there will be a link to all the slides.

Um, I will probably post the slides anyways on Twitter and elsewhere so you can access them as well.

Also, if there are questions for people, like, you know, raise your hand, ask. I will essentially do intermission between, like, some parts, and I will ask people if you have questions.

Please ask. Um, last time the talk, many people asked questions. I will answer every single one, even if it's stupid. I don't care. Please ask. Um, sometimes I get stuff wrong, so yes, just ask questions.

Um, are there any questions? [laughs] I mean, I'm assuming no. Okay. Okay, let's go to the... Okay, I will s- okay. So I, I don't know if people have seen this plot, very famous from Maxime.

Um, he shows the open source versus closed source performance of popular benchmarks. I think that, uh, this is MMLU five-shot. Um, you can see the green line is open source models.

Um, and then the red line, or the orange line is like, you know, closed source models. Um, and you can see that, in general, the slope of the open source models is more dramatic than the closed source models.

And I would say, in general, that, you know, the open source models and closed source models, in terms of MMLU, they've kind of, like, reached the same. Accuracy, right?

So, like, you can see that ... Okay, this is already outdated, but in general, you see like Llama 3.1 405 billion kind of reached, you know, GPT-4o level. Um, so open source models definitely have caught up to closed source models.

However, there is a however. Um, recent, you know, like since September 2024, I would call this something called the o- open source drought. Um, no, no one wants to talk about it, but I will, right?

So like September 2024, o1 got released, o1 preview. And to be honest, the open source community was shocked, right? So, like, suddenly the capabilities diverged, right? So there's something called the MMLU plateau, where most models, the open source models and the closed source models, they kind of converged.

So the open source models was equivalent to the closed source models. But suddenly in 2024 September, you know, OpenAI released o1-preview, and it kind of shocked the entire community because the capability or intelligence kind of skyrocketed, right?

So like with reasoning, long reasoning traces, it just was a total change of mindset. Um, and for four months, the open source community kind of died internally because there was nothing ...

We can't replicate it. We don't know what to do. You know, do we do this? Do we do that? I don't know. Like, and so, like ... But then suddenly in 2025 January, DeepSeek-R1 came along, and they released R1, and that's when the entire world kind of changed their view, right?

So, like, you can in fact train open source models to be as powerful as o1 or o3 or whatever, right? So, like, that's, that was what I call the open source drought.

However, there was a previous drought even before that. Remember when ChatGPT got released in 2022 December, right? So, like, before even ChatGPT, most models were base models. Um, they were not really instruct fine-tuned that well, and so most large pre-trained models were actually useless, or they were terrible.

But then suddenly ChatGPT came along, and they did better, you know, reinforcement learning from human feedback, better instruction fine-tuning, better instruction following, and it really changed the world, right?

So, like, I think large lang- large language models were already here before 2022, right? It w- they were already there, but it was just ChatGPT which showed that if you have good data, right?

Good instructions, good answers, good supervised fine-tuning, um, and good reinforcement learning, you can actually make the model very useful. Um, and yes, again, open source had a delay, a very long delay, right?

Until, like, Llama 1, I guess. Um, and so always, I would say open source always tries to catch up to the closed source models. Um, the next question is, what is after reasoning?

Um, is there going to be something else? Um, I think that's a very good question. My personal take is it's going to be very hard. Um, I think reasoning was, like, the last, most ...

The DeepSeek R1 paper said that most likely the model already has these reasoning capabilities, and we just need to accentuate them. Um, and so, like, I'm not sure if there's going to be some other new, you know, like, step function where we get to, like, the next capability.

Um, but in my view, I think, like, every single time the closed source models will always do, like, a step function. Um, but who knows? Maybe, maybe now it will plateau forever.

I don't know. So that's like ... You have, like, you know, you have, like, long discussions about, you know, if AGI is gonna come or not, like, but who knows?

Um, the talk is not gonna be about that, but, um, yes. Next. Um, so I call the first jump the SFT or RR, um, RLHF jump, right? So, like, that's essentially if you do good supervised fine-tuning, you get this large jump in performance.

And then the second jump is called the RL jump, right? So, like, this essentially can increase performance dramatically if you employ second methodologies like RL, right? So, like, but the question is, like, what's the next jump?

Um, I don't know. So I'm not sure if you guys saw this picture before. Um, it's very widely known in the community about, you know, by Yann LeCun. He essentially showed this cake, um, and essentially unsupervised learning or, like, just pre-training in general, um, you know, is just a cake, not that good.

Um, and then supervised fine-tuning is kind of the icing on, on top of the cake, so, like, it's a bit better. Um, and then the reinforcement learning is the cherry, right?

So, like, I'm not sure people like the cherry, but, like, some people like the cherry. Um, and so, like, the goal is how can we get the cherry? Um, but the problem is there is so less data about this, right?

Reinforcement learning is, like, very, very, very less data. And so the problem is most mo- large model labs will train these large pre-trained models, and then they will iteratively refine it to make the model better through supervised learning, through reinforcement learning.

Interestingly enough, this slide was actually shared last year, very popular, but actually this was from 2016 September. Um [chuckles], so I had to dig this up on YouTube. And so Yann LeCun actually talked about this back in 2016, so literally nearly 10 years ago.

Um, I was like, "How can it ... Wait a second, that's 10 years ago." Uh, very long. Um, so this slide was actually very popular on Twitter. Uh, I, I think it was in November of last year, people kept tweeting about it.

I don't know. I saw this, I was, like, shocked. Um, but yes, so this encapsulates, like, the current AI, um, boom.

And so, like, firstly, like, when we talked about these large models, remember they started from a base model. Um, and so we call these training stages, right? So when you have a base model, um, you then convert it to a chat model, right?

So, like, for example, ChatGPT is not a base model. It is a instruct fine-tune model or, like, some sort of fine-tuned model from a base model. So actually, OpenAI does have a base model somewhere sitting in their server somewhere.

Um, they're probably not gonna serve it ever, but it is somewhere on the computers, and they essentially fine-tuned it to make ChatGPT 4. Um, Claude 4 has most likely a base model, and then they fine-tune it to become Opus, right?

So, like, Gemini also has a base model, and they convert it to Gemini 2.5 Pro. So this, this phase, when you convert a base model to a chat model, is the fine-tuning phase.

Um, and then the question is, like, you know, what do we do in the arrow, right? Like, you know, is it reinforcement learning? Is it supervised fine-tuning? Is it, like, some other special sauce?

I don't know. But, like, we essentially will discuss about these, um, topics. Um, any questions first?

Okay. So for example, in open source models, you might have seen Gemma 3 PT, um, Gemma 3 IT, Llama 4, Llama 4 instruct, Qwen 3 base, Qwen 3, um, Mistral Small base, Mistral Small instruct, Llama 2, Llama 2 chat, right?

These, like, terminologies. To be honest, I think the open source community should standardize a meth- like, terminologies, like instruct or chat or, you know, no, no, not even a word, like, you know, IT and PT.

Maybe they should, like, standardize it a bit. Um, but in general, if you see IT, it means instruct and instruction tuned. PT means pre-trained. Um, instruct just means, you know, instruction fine-tuned.

Qwen-3 just removed it entirely. It's just called Qwen-3. Um, and then the base model's called with a base. Um, and so like essentially these na- naming methodologies, um, if you see on Hugging Face, um, hopefully you will now know, recognize these different types of, um, models.

And so, generally what we say for like, you know, reinforcement learning and fine-tuning is I would say it's called re... Fine-tuning's everywhere. Um, you start off with pre-training, you then convert it into a supervised fine-tuning model via supervised fine-tuning, SFT.

It's called SFT. You also might hear like IFT, which is instruction fine-tuning. They're the same thing. Um, and then we call something post-training, the post-training phase. Um, but actually recently it actually kind of changed.

Um, so I don't know if you guys have been keeping up with the latest stuff, terminology. Um, I actually don't really like terminology anymore, but like we have something called pre-training, which is you take like all of Wikipedia, all of the web, you know, everything, all of the data you can ever see, shove it into the model,

predict the next word. That's called the pre-training s- uh, pre-training stage. We then have something called the mid training stage, um, which essentially gives you higher quality data. Like for example, you can weight Wikipedia more because it's high quality.

Um, you can essentially do long context extension as well. You shove this during the mid, mid pre-tra- uh, mid training stage. So if your context of your model is very short, you want to extend it to very long context.

You shove this during the mid-training stage. And then the second stage is the supervised fine-tuning stage where you wanna convert the model to a chat model. And then we have the post-training phase, which is like press- preference fine-tuning, DPO, RLHF, and stuff like that.

And then we have this new thing called reinforcement fine-tuning or RLVR. Uh, if no one knows what RLVR stands for, it stands for reinforcement learning with verifiable rewards. And this is like a new paradigm, um, not the same as preference fine-tuning or DPO, where we consider reward functions to make models much better.

Um, and so this is how I would v- envision like, you know, the whole training phases of models.

Another way to put it is we have some random initialization of the model, like some random weights of the model, right? So like seven billion parameters, literally random numbers, right?

Like GPT-4, GPT-4, I don't know, 1.4 trillion parameters, like just random numbers. And then somehow we move in the space, like the black line, right? So pretend this is like some high dimension of 1.4 trillion dimensional space, and then we somehow move in this space, and then we get the final model, right?

That's the green dot. The question is how do we move in this space to get to the final model? That's the question.

Most people, what they do is firstly you start from a random initialization. You do the pre-training phase, which is very long. You get to this dark blue dot, right?

That's called the pre-train model. And then you do some supervised fine-tuning, instruction fine-tuning to get the blue dot. And notice the line for the light blue line is very short because it is very short.

Right? There is not that much data for supervised fine-tuning. And then somehow we get the blue dot, and then we keep doing more iteration to get to the purple dot, which is through preference fine-tuning.

And then finally we get the g- the green dot, which is reinforcement learning via, you know, verifiable rewards, like, you know, o3 or o1. And so like the goal is somehow we have to move from the black dot to the green dot, and essentially all of large language models, all of AI is just an optimization problem, right?

So like how do we make this easier to get to the green dot? You could, you know, you could kind of theoretically guess, you could just... Why don't you just go from the green dot, uh, black dot to the green dot, like skipping all of the dumb phases?

You know, just skip it entirely. Yes, you could do that, but it's not gonna be very efficient. You're going to be waiting there for like, you know, I don't know, millennia.

Your loss is not gonna go down. So the tricks that we found in reinfor- um, AI is like you have to do these phases to get to your final green dot.

Um, there is like a new methodology where you can actually bypass this supervised fine-tuning stage and the preference fine-tuning stage and directly go to the green dot. There is a way, and that's the dark red line.

Um, I think DeepSeek, uh, DeepSeek-R1-Zero kind of like sh- showed that. You can use a pre-trained model, a base model, and directly do some reinforcement learning with, uh, verifiable rewards and just skip it entirely.

Um, so that's like a new paradigm that people wanna focus on. In my view, I think you should still do the light blue, the purple, and then the green.

I don't think so you should like directly skip over to the green. If you wanna waste resources, you can skip to the green. Um, but I don't think so large model labs wanna s- waste resources.

Um, hopefully not. So I, I don't know if people have seen this diagram, agents in the old sense. So like everyone keeps connecting agents with reinforcement learning. Okay, but like why?

Um, so in general, an agent is you have some sort of environment. You have like the agent doing something in the environment. You get like an action. You do the action, and then you get like some, some sort of, some sort of reward.

Um, and the reward is R. S is the state, so the current, what the environment currently looks like. Um, and then essentially RL tries to optimize this loop. You're trying to maximize the reward, um, giving some sort of action.

Um, and that's why like, you know, that's why RL and agents are kind of connected. Assume the agent is the lang- language model, right? So assume the agent is in fact the language model, and so this, the environment is kind of fishy.

Like, you know, it's, it's hard to say what exactly is the environment. It's more like the language model's inference space. That's the environment kind of. Um, but like pretend this was a game, right?

So like the agent was the computer. The environment is like Mario, for example. Right? And so like you're playing a Mario game automatically, and your goal is to win the game.

And so like the whole goal of RL is to maximize reward.

Another one is like Pac-Man, right? So like you have the yellow Pac-Man. Um, you can either go up, down, left, or right, right? So like up, down, left, or right.

That is the action. That's the A. The yellow... The, I think, I don't know, orange or something, yeah, the orange little things are like rewards, right? So like if you eat, if you eat a, you know, a yellow, a red, an orange dot, you will get positive reward, so R+.

Right? So like you have R+'s. If you eat a very big one, you'll get like very large reward. But if you encounter one of the enemies, you will get minus reward, right?

So like the que- the an- the question is how do we maximize the reward based on this environment?

For language models, there is a trick. The trick is this loop kind of changes, because we don't actually have a continuous loop. Um, the state does not actually change over time, right?

So like, for example, in a game, in a game, if you do an action, the whole state changes, right? The environment totally changes, and so you have to, like, continuously keep a history of the past steps.

But in language models, there is no history, right? So like if you do a prompt, "What is two plus two?" If you ask another question, "What is four plus four?"

It's, like, totally not relevant to your previous prompt. Okay, well, fine, it is kind of relevant, but you... Like, it's not directly correlated, and so, like, you can actually delete one of the lines, um, that's the next prompt.

You can delete it entirely. So for example, what is two plus two, right? So, like, essentially, you have all of these options. It could be zero, it could be one, it could be two, it could be infinity, it could be B, it could be D.

I don't know. It could be anything that you like. A symbol. And so the "what is two plus two" is the state. So, like, what is the question? That's the question, right?

So, like, the question. The reward... For example, if you choose the one ... If you choose four, your reward is plus one. If you choose anything else, your reward might be negative infinity, zero, whatever number you like.

You can come up with any number you like for reward. It doesn't have to be plus one. It can be plus 10. It can be plus 100. And you can do anything that you like.

Um, you can also do distance-based scoring. For example, is, you know, is choosing the number five better than zero? So that's a question. What, what do you guys think?

Is choosing the number five better than zero, or is it worse?

I think it's better.

Yes, okay, better. So what would you do for the reward then? Pretend the model outputs five for what is two plus two.

Something less than one.

Zero.

Zero? Someone said zero? Okay, zero is fine, because it's wrong. Yes. Like, if you... The answer five is wrong, so you should probably give a reward of zero. But is there a better answer?

Less than one.

Okay, yes, less than one. So, like, some sort of, like, maybe zero point eight. I don't know, right? So correct. So, like, you could do, like, the answer divided by the correct answer.

Right? So, like, if it's five, you divide it by four. You could do some sort of re... Okay, may- no, wait, that's wrong. It's five minus four divided by four.

Um, so, like, something, some reward like that. Um, pretend if you're... Pretend if the model says A, what is the reward?

Minus one.

Minus one? Okay. Or it could be minus 10, because that's very bad, right? So you, you should not output a letter. It should be some sort of number. So that's how you design reward functions, right?

You, you... We just design a reward function. Shove this into, you know, take your reward function. It's like just if statements. Shove this into a language model, fine-tuning phase, and there we go.

You have o3. Okay, well, you won't have o3, but you know what I mean. Um, essentially, o3 is a collection of all these reward functions, right? So, like, for example, this, "What is two plus two?"

Is one question. Remember, it doesn't have to be, "What is two plus two?" It is a general maths question. What is 10 plus 20? What is 10 times 200 divided by 10?

You know, whatever maths equation you ever want. And this function can take your question and convert it into a number. And o3 is just a collection of all of these reward functions.

And so the goal of RL is to make the good ones more good. You want the good rewards increase in value. So for example, the four, you want the four to appear more, but you want the three to decrease.

You know, you don't want the three to keep appearing in your answer, but you want the D and the B to be very, very, very heavily penalized. That's the goal of RL, right?

So, like, we don't actually have the answer, right? Okay, this, this question's very easy. What is two plus two? Obviously, it's four. Yes. Okay, that's very easy. But pretend you have some sort of complicated question, like, for example, how do I win the stock market?

For example, like, dumb, dumb thing. Okay. How do I win the stock market? You don't know what actions you're gonna take, right? Like, but the point is you have the result, right?

You have the result, like profit or loss. But the question is, we don't know how do we get to the good profit. And so the question is, how do we maximize good actions as much as possible and decrease bad actions as much as possible?

And that is RL. And so OpenAI released something called RLHF in, uh, ChatGPT, and they sh- oh, actually, I think it was InstructGPT. But anyways, for ChatGPT, they showed that you just need some training data, you need some data.

You interact with the agent, which is the language model. You then get some actions, which is your answer to the language model. You feed this into a reward model, and then you get some reward, and then you keep ite- you know, iteratively doing this step, and you'll finally get ChatGPT.

So remember, the tr- the base model you start with, so ChatGPT base, you convert this into ChatGPT 4 via this method.

To expand on a bit, if you guys have heard of PPO. Right, what is PPO? Um, essentially, PPO is just... You just expand the box for the agent, right?

So the language model is like an agent. You expand it, and there's just three models inside of it. Um, there is a generating policy, the reference policy, and then there's a value model.

Um, and that's all. It, it's not that special. Um, we will talk about each of these things separately, um, but PPO is just a optimization algorithm to make RLHF work better.

GRPO, which is the algorithm behind DeepSeek-R1, smartly deletes one of the fact, uh, deletes one of the things, the value model. It just gets rid of it. Um, now, why would you delete it?

We will talk about this, but the trick is, if you delete a model, the value model, you just save parameters. You save compute, and it's much more efficient, right?

Remember, each of these models is kind of like a large language model, right? So, like, you know, pretend you're generating... Your model is, like, already one point four trillion parameters.

What are you gonna do? Make another one point four trillion parameter model for the value model? So we just get rid of it, delete it, um, and that is GRPO.

That's the only difference between... Okay, well, there's other differences, but the biggest difference is we get rid of the value model. Any questions?

Yes.

Um, so you, you talked about negative reward.

Yes.

It's confusing because in pre-training, isn't... It's like, um, it's-

Probabilities?

Well, it's like negative rewards, right? And then... So I guess it's the... Why is the phrase reward used when it can be positive or negative?

Versus comparing it to pre-training where it's always negative

Do you mean pre-training as in, like, the negative log likelihood or, like, some prob- ... So the, when doing pre-training, the goal is to maximize probability

Yeah

So like, you know, you output a, some number from zero to one, the probability of the next word, and you want to maximize that. For RL, you want to maximize reward.

So if it's a negative reward, you still want to maximize that. So if it's negative one, you just want to make this negative one go away, and you wanna be positive, in the positive range.

But also rewards can actually be just negative, right? So for example, if your, your reward function can be negative 10 and negative one. The good one is negative one, the bad one is negative 10.

Your goal is to move towards negative one as much as possible, because your goal is to maximize it. So I would say the reward is a misnomer. You can just add 10 to everything, and then the, you know, the sc- it scales the numbers away.

Um, does that kinda make sense? Or ...

It does.

Okay.

It was a question of nomenclature, I think.

Okay.

Yeah.

Yeah. Most people, I would say, like, to be honest, they don't like negative rewards. Actually, in the RL space, people just like to do positive reward. I don't know, I like negative rewards.

Um, I, I feel like it's more ... For me, it's more intuitive. Um, yeah. Any other questions? Yeah, yeah.

So you've got your language model, and then you've got generating policy and reference policy-

Yes

... right? What, what models are being used there? Is it the same as your language model, or is that another model?

Trick. Yes, very good question. There are some tricks that you can employ. Most people just make them at the same model. The reference model is, like, the beginning of the model.

The generating model is the model that you're updating. So, like, essentially the reference model is, like, the base model. Okay, that's probably not a good... Okay, fine. Just keep it as the base model.

The base model. The generating policy is actually the model where you update it. So, like, the model that you update the, the, the base model's update. So every single time you get a base model, you update one, update two, update three.

That is the generating policy. But we will talk about this. So it's, it's essentially the same model, but there is updates to the model. So the reference policy is the model that is not updated.

The generating policy is the model that is actually updated, and they're both the same model, so the, the language model. One of them's updated, the other one's not updated.

But we will talk about that. Um, y- yes.

So the actions, is it typically one token or is it more, uh, a series of tokens, like, uh, an actual full completion in a latent space? Or, like, how do you do it in, in context of-

That's a good question. In the Pac-Man case, the action will be a string of actions, right? So, like, you can go up, then down, then left or right. You know, some sort of, like, long history.

For, in the language model space, we just generally ... Th- this is called single turn and multi-turn. Generally speaking, currently, single turn is what most people do. It's just one action.

So the action will be, um, the action essentially is saying, "What is two plus two?" And the answer is four, and that is the action. The action is actually the inference space.

So, like, what is the actual chain of thought? That is the action. Um, and so, like, it's just one, though. But it is the total sum of the chain of thought.

So, like, if you have, like, what is two plus two? I think the answer is four. You know, let me do some working out, blah, blah, blah, blah, blah, blah, blah.

That is the entire thing. If ... Does that kinda make sense?

Yeah, it makes sense.

Okay. Yes.

Uh, the, there was a Claude conversation around how, like, to finish a poem's next line, it had to, like, think ahead to the last letter, to the last word to match the previous one.

Do you think, like, a reward focus versus next word completion allows you to do more think ahead, uh, inference?

Y- you could

... really greedy when you're trying to, uh, understand the reward function instead of maximizing the next word

I think for pre-training specifically, there are research base, papers which show that actually pre-training doesn't just predict the next word. It does try to predict many words ahead. And so, like, yes, maybe reward mo- In reinforcement learning, you can see, you can accentu- it essentially accentuates a pre-training behavior.

So maybe this behavior already exists in the model, we just see it more often. So maybe I would say that,

I would say that the model itself, it already has this capability. We just want to make it more obvious. And so maybe the model already even knows how to do that.

It already knows it predicts 10 words ahead or 20 words ahead. It already knows how to do that. But we just wanna make it more obvious. I'm not sure if that answers.

I mean, would it be safe to say, like, so if it's a-

Yeah

... poem generation, you want, like, every last word, every ... You want almost a circuit for the last word to match the previous line's last word.

Okay.

It's more likely to build that circuit with generative, uh, GRPO instead of, like, the classic ones which really rely on next 100 word predictions or next 500 word predictions.

I think for reinforcement learning specifically, yeah, I, I guess yes. Um, I think it essentially, your goal is to maximize the reward. And so, like, however, whatever way you try to get there, it's different from generally pre-training.

Pre-training's just maximizing next, the probability of the next word. But reinforcement learning is you're trying to maximize reward. The question is, how do you actually maximize the reward? Do you, like, do chain of thought?

Do you do, uh, what you describe, like thinking about the next, you know, in the future or something? I don't know. Like, I mean, the question is, like, what is the reward function actually doing?

I don't know. What is the language model actually doing? I don't know. Um, I don't know if that answers your question. Like, it's, to be ... Yeah.

Yeah.

Yes. Yeah.

Um, I was curious about when you were talking about, you know, the arithmetic, whether five is a better answer than, say, negative-

Yes

... 100 or something-

Yes

... for two plus two. And, like, given that there are these, like, closed circuits between all these different related, you know, mathematical functions you can do on numbers in space, like, whether it is, like, in the literature or in, like, the current state of the art-

Mm-hmm

... better to train it so that a closer prediction is more accurate, or whether just saying, like, the right answer is right and everything else is wrong, which in some logical sense is true-

Mm-hmm

... like, gen- tends to produce more performance-

Yes

... in, in that space.

Yes, you're correct. You should have data which is, like, getting more accurate data. Uh, is that what you're trying to say? Like, you should have data which is, like, for example, what is two plus two?

You should get more data which is, like, four. It shouldn't be, like, five or 10 or minus 100.

Well, like, saying that t- fi- like, the pr- the practice

Yes

... like, in general, is that what people do in previous studies?

Oh.

Do they tend to say like-

Yes

... like when there is, like, exactly one correct answer and everything else in a mathematical sense is equally wrong because it's not that answer?

That is a good question. I don't know. Um, I think large model labs won't tell you exactly what they do. In our experiments, when you use our notebooks, we actually show that if you do distance based, so the closer your number is to the actual number, you will get better results.

But generally speaking, it's easier to just say, "Five is wrong." Just give it zero reward. Everything is zero, and then the good one is, like, one. It's actually much easier to do.

Um, for example, if you wanna do execution of code, how do you actually do distance-based scoring? Right? So like, if you ask it to create a Flappy Bird game, you just have to find an output.

But you don't actually know how to verify, like, you know, oh, is this Flappy Bird ga- game better than the previous Flappy Bird game? It's only in the mathematical sense you can, like, do distance-based scoring.

Um, I'm assuming large model labs, they probably do the zero one better. Like, the majority of them just do, like, yes or no, yes or no, binary. Um, but in our experiments, for math specifically, you should do distance-based scoring.

Um, it makes the model learn faster.

Thank you.

Yeah.

Uh, for verifiable domains like math, um, is it actually tenable... Because two plus two makes sense, but it's not gonna scale for, like, really large numbers or large multiplications.

Mm-hmm.

So are we gonna end up... Is the end game using tool use to calculate that, or the model could potentially be trained to solve that?

That is a good question. In the olden days, before this paradigm came along, we would think that you can just use a tool, like a calculator, to... You should actually...

I, I would say you should still use a calculator to calculate two plus two. Right? You should not use a language model. But with RL, you know, VR, the trick is we actually found that actually, wait a second, if you just do two plus two, or you do another question like ten times ten, or you do some

sort of complicated mathematical expression, you know, like the derivative of X squared or something, I don't know, like some random mathematical equation, it randomly learns to actually solve that equation without actually doing over-fitting.

And so, like, I would say that with RL, you can actually make the model actually learn how to do multiplication, how to do addition. So it's actually in the model.

Um, does that kinda make sense?

Yeah. Would we use this in production or it's-

Oh, yes, yeah. People use that in product- I... Okay, maybe don't use it in production [laughing]. You know, you're not sure if the answer is correct. Maybe it will say three plus three is seven.

Okay, I don't know. It's pop- it's possible. Right? So like, essentially... But it's getting better. Um, maybe in the future, all of mathematical equations can just be done by a, a model.

Um, I think in the... Maybe a few months ago, maybe like Septem- you know, before o1 got released, actually not even o1, a few months ago, people would still say, "Use a calculator," you know, like some sort of tool calling.

Yes, you should probably still do it. Um, but imagine, you know, as time goes on, as models get better and better and better in terms of, like, training data, just, just for the maths equation, you know, two plus two, um, imagine in the limit as we get all of the world's data for just this maths question, right?

Two plus two, four plus four, ten times ten, or whatever, it should, in theory, solve them all. In theory. Um, yes, the... It's always in theory. Um, but yes, you don't need the tool calling.

It's not necessary. Um, yeah. Y- Y- Yes.

My question's about the reward model. In practice, are people using large language models as a reward model?

Good question.

For inverse reinforcement learning, or is it hand rolled or-

Yes, I ac-

... reliable solver?

Good question. Oh, I was gonna go in the next, next slides we'll be talking about that. Um, y- yes.

How does this change with, uh, multi-turn?

Multi-turn? W-

How does this, how does it... Does this change with multi-turn? I mean, you showed just a single turn, right? So in a multi-turn, you can sort of loop out like a tree in

Yes.

How does this-

You could do multi-turn. It's a bit more complicated. Um, you just imagine... There are tricks you can do. You imagine that your current step is good. Imagine it, and then you just continue doing inference.

You, you append, like, your next question. Like, for example, "How am I gonna... What is two plus two?" You say, "Okay, let me think about this question. What is two plus two?

Blah, blah, blah, blah, blah, blah, blah, blah, blah. The answer is four." And then the question is, what is your next question? Maybe the user interacts with it and says, "Okay, I, I don't think your question's correct.

Oh, I don't think your answer is correct." And then the model says, "Oh, okay, let me rethink about this. Blah, blah, blah, blah, blah, blah, blah. I still think the answer is four."

Um, so you could chain this all together and shove it into the, you know, the whole RL step. You could do that. Um, it's a bit more complicated. I think the diagram will be a little bit more different.

Um, yeah.

Yeah, I guess a follow-up to that, to that would be, um, do you, do, do you assume, like, a, a loop is a single turn, or a loop is a lot of turns, and then you only give a reward at the end?

Or you give, like, sub-rewards or-

Very good question. So there is... In the DeepSeek-R1 paper, you could do sub-rewards, or you could just do the reward at the very end. I think sub-rewards might actually do better in general, but the question is sub-rewards is very hard to calculate.

You would rather just wait, you know, all until the very, very end and just give a reward. That's probably the easiest. So it's more about efficiency. It's all... To be honest, all of AI is about efficiency.

What is more efficient? What is more... It's all optimization. Um, so the answer is, like, I would suggest people just to, like, shove a reward at the very end.

Um, uh, yes.

Once you give your reward signal, is it just the REINFORCE algorithm with the gradients to go back the weights?

We will talk about that, yes.

Okay.

Yes. Mostly, yes. Correct.

REINFORCE.

Yes. So we will talk about REINFORCE. We'll talk about PPO, GRPO, and stuff like that. Um, y- yes.

You mentioned with the slide with the squiggly lines that using RLVR, uh, you're all, like, to skip the intermediate steps is-

Yes

... a waste of resources. Can you just talk about, like, why it's a waste of resources and, like, how you would choose to use, like, more resources?

This one?

Yeah.

The problem is, if you skip from pre-training to the RLVA st- uh, RLVR stage, it's relatively hard because your model doesn't actually know how to do instructions, right? So like, you have this base model.

You ask the question to the base model, "What is two plus two?" It's not gonna say, "I think the answer is four." It might... You might be lucky, somewhere in your pre-training data, somewhere on the web, someone asked this question, "What is two plus two?"

And then, you know, the qu- the answer was like, "Oh, the answer is four." But you have to be lucky. Um, so like- So the problem of this is the whole trick of SFT is you wanna force the model to answer what is two plus two in an instruction way, right?

So, like, it will tell the model what is two plus two. You want it to say the answer is four. You don't want it to, like, blabber on and, like, regurgitate...

Like, get some Wikipedia article and shove it as the output. So the whole point of SFT, preference fine-tuning, and stuff like that is to make the model forced to make a more optimal, to, like, output conversation style.

Um, if you wanna skip, it's also fine.

It's just not efficient, um, because, like, you, you could do this. Um, I'm assuming large model labs are trying to do this. Um, so it's not like a you should or you shouldn't.

They are trying. Um, does that-

Yeah.

Okay. Any... Are there... Yes.

Yeah, I have a question about a couple of places. One question is that is this RLVR, is it a online policy optimizer or offline? Basically, does the reference model change after each block of spaces?

The reference model does not change. So the reference model is just a model that you didn't train. Um, it's like the... It's like the, it's like the base model or, like, the SFT.

Whatever checkpoint you started with, it doesn't change. You could change it. I think that'll be too expensive though. I think if you change it, that'll be more complicated. Remember, all of [laughs] AI is about optimization and efficiency, so I feel like you, you don't have to.

You could. I, I don't know if there are papers talking about it though. Um, maybe OpenAI does it. I don't know. Um...

So, so then the other question is that

do we need less, uh, examples of, you know, training data for, you know, any of this RLVR-

Mm-hmm

... compared to, like, weak pre-training and anything, you know, compared to SFT. Do we, do we need less training data or what do you think?

So the trick of RL is you just need a reward function. You need to make that. And you don't need data. You don't need the answer of the data.

Oh, actually, you do need the answer. You don't need the chain of thought. You just need lots of questions, like what is two plus two? What is four plus four?

Remember, you can actually automatically generate this, right? So, like-

The number of samples. Like, do you need less number of samples compared to SFT?

You should do as much as possible. You... Most large language models, I think for, like, you know, o3 or o1, I don't know what is the percentage of compute.

Maybe they spend, like, five percent or less. But the goal is what happens if you spend double the compute just on RL, right? So, like, previously, if you do fourteen trillion tokens on pre-training, make RL fourteen trillion tokens, and then the goal of large, large labs is to just do that.

So currently, it's very less, but over time it will increase. Yeah.

But, but compared to, like, SFT, you know, like they're saying here, the number of samples will be much lesser, right?

Currently, yes.

Because, because it's expensive. Like, we need, like, big models.

Correct. It's expensive. But over time, I think, like, maybe by, I don't know, next year or, like, this year, large model labs, their goal is to do this phase the most.

That's their goal. Because remember, you can automatically generate questions now. What is two plus two? What is two, two times two? What is ten divided by ten? I don't know.

Generate as many math questions as you like. But remember, you can also generate, you know, like, coding questions. You can generate any questions that you like, or you can use the supervised fine-tuning data itself for the RL step.

You can do that as well. Um, does that kind of make sense? Okay. Any... Okay.

How do you protect the SFT from being, like, screwed up by-

Oh, good que- yes. We will talk about that

... say, like, you have two plus two equals four.

Mm-hmm.

Five minus one is also a good answer. Two plus two equals five.

Correct.

But I don't want more equations. I want the answer.

Very good.

So is there, like, techniques to make sure that we're not violating our instructions to be concise?

Yes. We will also talk about that.

Okay.

Um, clipping and stuff like that, yes. Yes.

Is there any research that's been done on how, um, I guess RL specifically, but maybe training more generally affects specific circuits in the model? So, um, for example, you know, there's a circuit that, like, says two plus two is four.

Mm-hmm.

It just knows that.

Yes.

But can you incent... Is there anything that's, like, incentivizing the model to, like, learn addition, the concept generally?

You-- That is a very good question. I don't know. I think that's, like, the, during the pre-training phase. Essentially somewhere, somewhere in the internet, someone wrote what is two plus two somewhere, and then somehow maybe someone did a formulation, like, you know, some sort of derivation of, like, what is two plus...

Okay, prob- I don't think anyone has done... Uh, pretend there is some derivation of, like, some complicated math equation, and so the model somehow learnt to predict all of th- that entire trace.

And if it keeps seeing this, it would, like, accentuate the fact, "Oh, okay, I've seen this before. Let's make this even more, um, more prevalent in the model." So somewhere in the model, it has learnt two plus two is equal to four somewhere.

Yes. Uh, I, I guess what I'm saying is, s-s-so we, we can use, like, the RL stuff specifically. Um, you're saying that if you get, like, super bad reward or super small or super good reward, um, you can-

You're weighting it

... you can kind of poke the model into a certain direction.

Correct. So there is actually two schools of thought. The first one is the model already has this knowledge, right? It already knows what is two plus two, and you're just...

RL just tries to, like, maximize what... If it sees two plus two is equal to four, it tries to weight this factor more, weight this circuit in the model more.

So the model already learns, learns. But then the second school of thought is like, okay, maybe the model doesn't know actually, and RL actually learns a new thing. Um, I'm more in the first one.

The model probably already learns.

Yeah.

It already knows it, and we're just maximizing the... You're just trying to make it more accentuated. Um...

Exactly. So the extension of my question is basically, I want... I'm wondering if you can steer, or, or maybe there's like research of this, that you can steer the model away from the two plus two circuit to the always do the, like, do addition circuit, if that makes sense.

You mean, like, just do addition?

Yes.

You could. I guess what you could do is, like, get the language model, see which weights are changed during the RL phase, which weights are changed, and you just give it what is two plus two?

What is two plus two? What is two plus two? You just keep doing this question, and you can see which of the weights are changing, and essentially you can extract this from the model.

You could I don't know if there's research about this, but I'm just making s-- I'm just making stuff up on the spot. You could do that, if that makes sense.

Uh, maybe that's a research question someone should do, research paper.

A-any other... Yes.

If you go back to GRPO slide, like, uh, more and more in here, what about, uh, basically quality network, uh, that they save, you know, money in compute by just making equal solutions that they don't know, like you have to answer exactly four in your scenario for R1, right?

That's one of the optimization-

Yes, we will also talk about that. Yes, yes. Yes. Okay. N- I- I-

Two more questions.

O-okay, yeah, sorry.

Do we expect usually changes, the RL changes all the parameters in the model, or is it, like, more targeted, like this set of RL would be this set of, like, targeted parameters, um, as, as a subset?

You, you could. There's like two... Large model labs will most likely change all of the parameters. Every single parameter is changed. Um, but there are papers which show that actually not all of the parameters are actually changing that much.

Some of them are changed by, like, zero. Like, the majority of updates to the model is, like, zero, and only some very small updates to the model are seen.

And so that's kind of the circuit idea where, like, the model already knows how to do whatever question you give it. You just-- Most of the updates are, like, zero.

Um-

Is there any literature saying which one is better? Does it-- Is it better to aim for changing all the parameters versus a subset?

You can also do, like, LoRA, you know, parameter efficient fine-tuning. You can do other things. You don't have to fine-tune every single thing. Um, but I think majority of large language model labs, they just do everything.

Um, otherwise, this, again, becomes a optimization problem. You know, what do we-- which layer do we select, and stuff like that. So it gets more complicated. But yes, you could do parameter efficient fine-tuning.

Actually, we're gonna share a notebook for that. So you can actually do it on your own computer. Um, yeah. There was... Yes, one more question. One more. Yes.

Uh, yes. I wanted to add a side note about the model steering. It's called model abliteration.

Oh, okay.

You can steer a model towards a specific output that you want it, uh, to find that. But I also-- My question was, I wanted to, I guess, highlight what you said about the model can build capabilities outside of what it was pre-trained on.

For example, uh, there was a paper from Google Research that talked about how the KL divergence constraint, for example, limits the capabilities that can be learned by the model, and you can only have the previous priors of the model that pre-trained.

I did some experimentation with small language models and found that to be true.

Mm-hmm.

For example, I found that when performing o1 reasoning, it limits my sequence. So I wanna, I wanted to understand what are the implications for, for example, working with smaller language models?

How important is this base model? And I guess, how can we work towards that kind of not make RL, uh, leading to this solution?

Good question. For the KL divergence term, do you mean, like, removing it? Would that make it better?

Oh, I was saying, uh, the KL divergence term... Well, I, I guess they removed it in the GRPO paper, um, because they found it didn't have an effect on the end result.

But-

Because the whole point of the KL divergence term is to, like, not make it too stray away from the supervised model.

Reference-

Okay. Maybe I'm not kept up to date with ref-- with research papers. But anyways, I... So your q-- your point was like, if we remove the KL term, it will be better, it learns new capabilities?

Well, it was more... No, it was more, uh, there was some-- There was, like, a proof paper-

Mm-hmm

... that described you can't build additional capabilities than what you had previously built in the model, almost your point about amplification capabilities.

Mm-hmm.

So I wanted, I wanted to understand what are the implications of that, and how can we prevent RL from being, uh, akin to distribution, for example? Or, or how can we kind of...

It's almost like the essence of your talk, but I guess understanding how can we build on top of that.

Do you mean, like, do you want to have more capabilities into the model?

Yeah. So how, how can... Are there, are there strategies you thought of for-

Uh, strategies? [laughs] Hard to say. I'm assuming the large model labs will probably know strategies. Uh, we will show, like... I, I will show examples of how to, like, make RL better, like how to, like, reach higher reward faster.

I'm not sure about new capabilities. It's actually very hard. It's actually very, very, very, very hard to es- elicit new capabilities in the model. Um, the question is, like, is this new or not new?

I think that's the question. Like, is this actually part of the model or not part of the model? Um, and most research papers are, like, hand-wavy. They say, "Oh, most updates are sparse."

You know, like, so most likely it's not, you know, new capabilities. But what happens if, you know, one year later, all of the model updates are, like, not sparse?

Is this considered new capability? I don't know. Like, you know, th-those are the questions. It's more like... I don't know if that answers your question. I probably didn't answer your question, but...

No worries. Uh-

Maybe we-- Maybe the other parts of the talk maybe might answer some part. Um, yeah, okay. I will keep going on. More questions later. Um,

okay. So, like, the reward model, right, was actually a language model that-- or, like, some sort of model, some neural network, some AI model that predicts the reward. In RLVR, we delete this entirely, and we just call it the reward function.

So, like, the ground truth reward, you know, if it's correct, you plus one. If it's bad, it's just zero, right? So you essentially delete another part. GRPO essentially deletes another part, right?

So, like, you remove... Remember, GRPO, you delete the value model, get-- totally remove it, and then you delete the reward model, and it's just a reward function.

And yes, as a reward model, you could use LLM as a judge. So you could ask a language model itself to say, "Is the answer good or bad?" You could do that.

You could do regular expression check. You know, like, is the formatting of the answer good or bad? Is the maths equation good or bad? You know, is the final output good or bad?

You can do distance scoring, stuff like that. You can also execute the Python code, and then you can see if it actually executed, right? So, like, is there, like, import errors or, like, format errors or, like, some sort of Python error?

And you can use this as a reward And so this blue box, the reward, can be anything that you like. It just needs to output a number, you know, minus one, plus one, I don't know.

It just has to be a number. In fact, you can make a dumb reward, just everything, it just does random. You know, plus one fifty percent of the time, minus one fifty percent of the time, I don't know.

And confusingly, a paper recently showed that actually random rewards works. Um [chuckles] so like, yes, go ahead. You can try it. Um. [laughs]

But also, why... Did someone say why? Um-

Yeah. Why?

Probably read the paper. [laughs] [laughs] But, but, why? To be honest, actually, I think the paper might be a bit...

There was an-

I don't actually believe it. There was a... Yes, there was an update showing that actually this was wrong. Um, that actually it's because the model, they don't-- The benchmarks are incorrect.

So when you say that you actually increase accuracy, like from twenty percent to fifty percent, but actually the model itself was already fifty percent, they just didn't check the accuracy of the correct model before.

Um, so there was a recent rebuttal to those types of papers. Um, but you know, interesting results. Um,

yeah. [laughs] I don't know. [laughs] Okay, so remember in RL the goal is you don't know the best action to take in the space, right? When you're doing Pac-Man, I don't know if going left, right, up, or down is the best.

I don't know. But at the very, very, very end, you will either d- you know, win or like, you know, get some reward, or you will die. Yes. Um, but the goal of RL is to maximize the best action you can ever take, right?

So like what is a better action than all of the other bad actions? So RL just tries to maximize the best-- well, not the best action, the better action.

Normal pre-training, you already know what is the best answer. It's like you already know what is the next word, right? You-- If you wanna predict, you know, "Hello, my name is Daniel," you already know the next word is gonna be Daniel, right?

So you already know it. But RL, you don't know in advance what is the actual correct reward. Um, so you c- the only thing you can do in RL is to maximize the, you know, one of the better options.

And so yes, okay, now more maths. Um, the goal is to maximize this equation. Um, that's the goal of RL.

So what is this equation? The J is like the total gradient. Um, well, actually it's, it's more like the, it's more like we wanna maximize this. It's not actually the total gradient.

Okay, maybe I misread that. [laughs] Anyways, pretend I didn't write that. Um, the-- We wanna calculate the gradient with respect to the policy language model, and the action is given a state, and the R is a reward.

If you wanna write this down in like English, it's like we wanna take the derivative of the log probability of the action given the state times the reward. Now, I don't know if you guys understand what that means, but I did like a example.

Um, Pac-Man case, okay? So you are Pac-Man. The red is your enemy. You don't wanna go there, right? So like you definitely don't wanna go to the red thing, right?

But you wanna eat the two gray dots, right? So that's-- Remember, you can only go up, down, left, or right, right? You only have four actions. Remember, the action space is just up, down, left, or right.

So if you do rewards, I just randomly made some rewards up. If you go to the red thing, you will get minus ten reward. Or actually it should be minus infinity, you die.

But anyways, minus ten. If you eat the gray dots, you get plus one or plus one. And if you go up, it's just zero reward. There's nothing there.

Now, when you get this language model or like some sort of model, it has to tell you what is the next action, right? It tells you what to do to the, the next action.

For now, we will just assign every single action up, down, left, or right as one quarter probability. Right? So like you go up twenty-five percent of the time, left twenty-five percent of the time, and so on.

So these are your numbers, right? This is the entire state.

So the goal of RL is you want to do that red, going towards, um, the right. You want to do this-- You wanna go towards the right less. You wanna do a much less, right?

So like you wanna push the probability of the zero point two five of the right much less. And you wanna go bot- like, you know, down and left much more.

Right? So you wanna push the probabilities much more. And the top are not really that important. And so RL essentially, your, your goal is to avoid doing the bad thing, and you wanna do the good thing much more.

That's kind of RL. If you convert this into a table, right, you have the probability of the action given the state. Right? Remember up, down, left, or right, we just assigned twenty-five percent chance.

Right? Just, just pretend twenty-five percent chance. The reward which we can calculate, right? We calculated it. We calculated the reward. We just made some numbers up, right, as zero, one, one, and minus ten.

The probability times the reward, we get some numbers, right? So like zero, zero point two five, zero point two five, minus two point five. And then if you take the log of the probability times the reward, you get some number.

Right? So like zero, minus zero point six, minus zero point six, and six point zero two. So from this table, does anyone know which row do we want to maximize?

What is the goal-- like, what do we wanna maximize?

Which row?

Bottom. Bottom.

You want to maximize the bottom row? What is the reward of the bottom row?

You wanna minimize it.

Correct. You wanna minimize the bottom row. Remember, the reward is minus ten. We do not want to maximize the last row, because the last row is the worst. And so that means the six point zero two, we want to actually decrease this number dramatically, right?

That's way too large. We want to decrease it. The other rows we want to maximize. And so the goal is, okay, we just take the sum of all of that, right?

We take the sum of the four numbers, and it's four point eight.

And so remember, okay, let's try, right? So like by hand, by hand we shall... Remember the right, remember all the probabilities are one quarter, zero point two five. By hand we shall do the bad action even more, right?

We'll actually do the worst thing. What happens to the... What happens, right? So like the reward, the probability times the reward is now minus four. It used to be minus two point five.

And so the reward, the log probability times the reward, the sum actually decreased. Right? It decreased to 2.58. Before it was 4.81.

Is 2.58 smaller or bigger than 4.81?

Mm-hmm.

Obviously smaller, so actually this is worse. You should do, not do this, right? This is actually bad. Remember the goal is to maximize, maximize this equation. Maximize it, right?

Maximize. And so 4.81 is actually better. The original state is actually better than the 2.58. So this, the thing that we just did is worse. So do not do this.

However, let's do the right thing less, right? Let's not go to the right and actually maximize the rest.

You shall see that if you do the log probability times the reward, sum them all, you will get 8.9, which is a larger number. And so the goal is to maximize this as much as possible.

You could say, "Wait, wait, we know the answer," right? You should go towards the right. Just make this 100% probability. Let's just, you know, and you'll have an infinite reward.

Okay, okay, not infinite, but you'll get maximum reward, right? Why don't we just do that?

But you should not do that because you're actually forced-- If you do this, your model will be like learn, "Oh, okay, let's just keep going right. Let's keep going right," and it just gets stuck, and it just becomes very bad for optimization.

So definitely don't do that. Now there is someone who's, who talked about REINFORCE. Um, we don't just multiply the reward, right? Remember this equation we did. Where is it?

The probability of the action given the state times the reward. We don't actually multiply the reward. We should not do that. Um, you actually multiply by something called the advantage.

Um, and what is the advantage? The advantage is a reward minus the average reward, the base reward. So you shouldn't actually just see you want to maximize reward. You want to actually maximize the reward, but also looking at the average reward across the entire model.

So it's called the baseline. And so this B, this baseline model, is what is the value function, the value model. Remember GRPO deletes the value model? This was the value model, and this value model essentially estimates what is the average reward if we just see the current state.

It does not take in-- It does not look at like, you know, what is the next, uh, next step. It does not look at, you know, what is the next action.

It just takes a snapshot of what you're currently-- So essentially it looks at this, it looks at this, and just guesses what is reward, right? It doesn't-- You're, you're not supposed to give it the rewards.

You're not supposed to give it minus ten, plus one, plus one or zero. It just looks at the current state and produces a number, and this number is called the average reward.

And so the goal is now we don't actually want to maximize this, you know, just the reward. We want to maximize the advantage as well. Um, so like we multiply all this together and the goal is we want to maximize this new equation.

Does anyone have any questions? There's lots of maths, but questions? Yeah. Yes.

So in terms of probability, how do you-- Is the large language model emitting an estimated probability, or this is a known probability of all the possible states? But how do you get that in practice in, in the normal world-

So large-

It's not a game

So a model, a large language model predicts the next word. So for example, you take the entire Wikipedia, and then you like chunk it into small little tokens, and then the output is just what is the next word.

So for example, my name is Daniel, but it could also be my name is Michael, my name is Bob, my name is whatever, whatever, right? You have all of these probabilities for every single word in the entire language possible, like one hundred and twenty-eight thousand words.

You assign a probability for every single one.

And so it's-

So the probability is based on the token.

Yes, correct.

But they'll be... Okay.

The trick of this for language models is you can utilize the probabilities directly.

Right.

That's the trick, and so like that essentially makes everything easier.

An-any other question?

Yeah.

Yes.

So it's kind of the intuition behind this, you're kind of almost like normalizing the gradient to kind of give it a little perspective [inaudible].

Yes. Yes. Yes, correct. Correct. Y-yes.

What about multimodal models? Uh-

Oh.

They have, uh, uh, one model that is not only based on text, but it can be also based on vision.

Multimodal models. Do you mean like doing RL multimodal models? Oh, that is more harder. I would say you, you could, you could look at the Sudoku puzzle, just convert the text model into a vision, just cheat.

I guess you could do that. You could like say, oh, you know, I guess you could give it the Pac-Man, you know, give, give it the Pac-Man thing and tell the model, "What should I do next?"

You, you could. I mean, vision, it's, it's kind of the same thing, but it's more--

I-- Does o3 do vision plus reinforcement learn? I, I, well, I think it does. Um, y-yes, you could. I think for open source, I don't think I've seen... Yeah, I don't think s-- Yeah, I don't think I've seen open source models do that very well.

Um, it is still very hard. Um, yeah. Any other questions?

N-no? Okay.

You can answer this.

Someone did ask a question? Oh. Oh.

So what is the-

Yes. So sorry.

So the average, what is the average, average of all the actions? Um-

Oh, what is the B? What is the base model?

Yeah, I understand average, but like what is the average on reference model for what action, you know?

So it just like, your goal is to, uh, your goal is you see this current state of the model, like whatever the environment currently looks like, and you just wanna produce a number that approximates what is the total average reward.

Okay, I'll give you an example. Pretend you're playing chess or like Go or remember AlphaGo. You look at the board, the current state of the board, and just say, "What is the probability of the white player winning?

What is it?" You're not supposed to do any prediction. You just have to predict what is the probability of the white player winning by just looking at the board.

That's kind of the average reward.

So it's always low.

It's always low? Yes. Yes, correct. But remember, at the very, very, very end phases, like, you know, you might get higher reward, but that's the goal. You wanna, you essentially wanna predict what is the probability.

Always, you know, i-- for example, in chess- I'm sure there are, like, some steps you can take to make the reward higher. The question is, when the model sees this, you need to-- essentially, you need to say, "Is this board better than the previous boards?"

And so this model, you have to train as well. You have to train this model. It needs to output a probability of it winning. That's for the chess example.

Uh, does that kinda make sense or... No?

Yes?

But, so there's a policy model and reference model. Do we use reference model for this?

The value-- No, no, no. The value model's totally different. It's an-- There's three models.

Okay.

There is a value model which predicts the average reward of the state. The reference model is just the model that you started with, and then the policy or the actual model that you're changing is the out-- the final result of your model.

Like, the actual chat model. So there's actually three models.

So you would use policy model to get B as well?

This one, the B.

Okay. Got it.

You will see the current state. You will see the current state, and then you will see, okay, what is the actual... I think you do use the policy. No, you, no, you just look at the current state.

You look at the current state, and then you output a probability of whether this chessboard is good or bad.

I see.

Of some, like, zero point eight percent you're gonna win. Um, okay. Yeah, something like that. Y-yes?

So when we're estimating the advantage in PPO, uh, they tell you to take, like, the ratio of the new policy and the old policy to determine whether it's within-

Oh, yes, yes. Yeah, we'll talk about that. This is just a general, uh, simpler formula.

Okay.

Yes. We'll talk about that.

Okay, cool. I guess my question will be relevant then if you want to ask it then or no?

Oh, you can ask. Yeah, go.

Okay. Uh, I guess because the policy is predicting the next token-

Mm-hmm

... we're-- that, that's the probability we're trying to reduce or increase based on the reward.

Yes.

Are you taking the ratio per token on the advantage while you're estimating the advantage, or are you taking the ratio across a turn?

That is a good question, and that is an active area of research because you could either normalize by all the tokens or the entire just one turn. Remains to be seen which one's better. [laughs]

Um, it's actually still people talking about that.

Because if you keep adding it, uh, uh, sorry, keep taking the,

the multiplication of all the tokens across your context window, you'll probably get a super low number, right?

Correct. Yes. So generally speaking, normally people just assume this, assume this rollout is correct, assume this chain of thought is correct, and they just do the very end. But then you do have to multiply probab-- Wait, I don't...

Yeah, you do have to multiply probabilities. So there is a multiplication somewhere. You will get very small, yes. Um, but you know, you'll get very small, but the numbers are relative, right?

So everything is very small, but then the smaller one-- the bigger ones are still very small, but it's still better. So they're all relative.

Yeah.

Any... Was there one more? Yes.

Yeah. It's relating to the REINFORCE, uh, slide.

Yes.

Um, so are you referring to the classical, uh, reinforcement learning algorithm, REINFORCE, or is it, like, some new thing?

Oh, no, no, it's very old. Yes.

Okay.

Very, very old. Y-yes.

Um, I wonder if you can, like, give some, like, advice on, like, how to think about this as a sort of framing or abstract level about, like, error propagation between, like, if you have a trained model which does the scoring or does the value function or whatever-

Mm-hmm

... that itself is trained from data, it has some, like, error margin. Um, and that,

you know, you have some softmax function, for example, that, like, only one in a hundred times will produce the wrong thing, but, like, it has that probability.

Yes.

Like, how do we think about the, like, development over time of these models and, like, to what extent that error propagation is something that you can observe and measure-

Mm

... or, like, you know, systematize and engineer around? Um, like, just, like, I don't really understand, like, what the state, the mindset is in this sort of-- in this process right now around that.

In my view, I think all of these formulas are just

made up. And so, like, the goal is to maximize reward.

Yeah.

But the question is, you need to, like... You can't just maximize reward because otherwise you might make the model really silly. Like, you might say, "Okay, what is two plus two?"

It just says four. Pretend your data set was just, what is two plus two, right? So, like, you literally just cheat. What is two plus two? What is two plus two?

What is two, two plus two? Just make the model just say four, four, four, four, four, four, four. Just four forever.

Yeah.

Do you want this as a model? Definitely not. So, like, we want-- It needs to learn, okay, if I give it the next question, "What is eight plus eight?"

It should not just say four.

Right.

Or what is two minus two? It shouldn't say four. And so the goal of all these algorithms is to somehow force the models not to, like, overfit to your question.

And so, like, these formulations are trying to do these things to, like, not overfit.

Yeah. Well, I'm thinking about, like, the chess example for-

Okay

... you were saying. The thing which scores the board and produces this, like-

Number

... a number-

Yeah

... which is like, is this good or bad?

Yes.

Like, sometimes these well-trained models have these theoretical novelties that, you know, where they say, like, make this move, and it's like not-- the new state is not obviously good or whatever, but they somehow have, like, figured out to do this.

Yes.

Um, and suppose that your training mechanism for the value function model, you know, hasn't picked up on something like that.

Mm-hmm.

In fact, there's, like, some error in the tendency of the value scoring model.

Okay.

Like, its probability of producing, like, a, you know, some sort of, like, perfect scoring of the board's position-

Yes

... for, like, black versus white-

Yes

... you know, is not always exactly right.

Yes, that's so-- Always not right. Yeah.

And, like, that will bubble up into your training process-

Yes

... in some sense slowly.

Correct. Yes.

And I'm like, how do you think about that? Like, how do you-- Like, what is the, um-

The, the value model, you have to train it together. So it's like a combination of the entire algorithm. So the value model predicts what is the probability, but you actually have to train this as well.

And so that is actually the problem. Some people, you could train this, you could train this separately. You know, you can, like, get all the chess possibilities and then output what is the final number.

I think that's what some people do. You could train this in tandem with the model, actually. I think that's actually more harder. I don't know if that- You-- So there is always error in the value model, always.

But you have to train this model as well, so you will reduce the error. But there's always error. So I think there's like some numbers you can like force the value model to be like less, less prominent.

Like don't forcibly utilize the rewa-- uh, the value function. But in GRPO, we just get rid of the rewa- reward model, uh, the value model anyways. Um, so totally gone.

Um, no, no, so you don't need to worry about that anymore. Um-

Great.

Okay. I will keep going on. Let me just check time, actually.

Okay. Okay, so remember, the goal is the advantage-- We want to maximize advantage, not reward anymore. Advantage is the reward minus the average reward or the base, the base reward, right?

If the advantage is less than zero, it means that it is worse than average. If, if the advantage is more than zero, it means that it's better than average.

And so the goal is we want to do the action more if it's better than average on general.

Now to PPO, right? So like, I don't know if you guys have seen the PPO formula. It is ugly, but this is the PPO formula, right? So like it looks-- it's more confusing because there's like a clip, and then there's epsilon and blah, blah, blah, whatever.

But we could just sc- strip everything away. It's just, it's just the probability of the action given the state times advantage, right? We literally just discussed about this. Okay, minus a log.

Uh, okay, the log's gone. But anyways, it's just that. And then the rest is, the rest is trying to reduce overfitting.

And so remember-- So essentially this-- There is a thing called the division of the old model, and essentially it's the model that created the action. And the goal is we now want to maximize this likelihood ratio.

We don't just want to maximize the probability of the model. We don't want to maximize the probability of the action given the highest reward. We actually want to maximize the likelihood instead.

But what is this likelihood? So I did some numbers. I just made some numbers up. So this is the pack-- So pretend the, the top, the, um, numerator is zero point zero one, and the denominator is zero point zero one.

Zero point zero one divided by zero point zero one is one.

If the denum-- if the top is zero point zero one and the bottom is zero point nine nine, remember, these are all probabilities, you divide the top from the bottom, you'll get zero point zero one again.

If the top is zero point nine nine and the bottom is zero point zero one, you'll get ninety-nine, and so on, right? The last one's one. And so the goal is zero point zero one divided by zero point nine nine is zero point zero one.

This means that the action that you do is actually very likely, right? Because the bottom, the bottom thing is zero point nine nine, but we actually don't like this, right?

Remember, the top is zero point zero one. We do not like this. So the ratio is zero point zero one. And then the bottom is this, this action, the bottom-- the denominator is zero point zero one.

It is actually not likely, but we actually like this because the top number is zero point nine nine. And so when you multi-- when you do the division, you get ninety-nine.

So this is actually good. And so actually, we're not actually trying to maximize the probability. We're actually trying to maximize the likelihood now.

And so the question is, why don't we just maximize the probability, right? The first equation. Why do we need to do the division thing? Because if we maximize just the top, you will have reward hacking.

What is two plus two? It might say, "To solve this question, we need to do blah, blah, blah, blah, blah, blah, blah, blah, blah." And then suddenly it says, "Hello, hello, hello, hello, hello.

Hello, hello, hello," and then it says, "Four." Is this good? I don't think so, it's very good. We don't want it to say hello, hello, hello something, or like some weird trace in the reasoning model.

It does something weird. We don't want this to happen. And so actually, this, this, you know, hello, hello, hello, hello is actually very not likely. And so the goal of the division is to reduce these issues.

The epsilon part is called the trust region. Essentially, we don't want to make-- we don't want to do large steps for, um, PPO, right? We don't want to do large steps.

And the, the trick is we want to restrict them, right? So like you don't want to overfit the model, so now we restrict the model. And so epsilon could be like zero point two, zero point one, zero point three.

And the one minus epsilon is zero point eight. One plus epsilon is one point two. And the trick is we just want to not move the direction of the gradient that much, right?

We don't trust the model that much. We don't trust the algorithm that much, so we want to constrain it.

And then also the PPO, there's also a KL term. Um, there's another term. Um, essentially, what this does is we want the model to be as close to the supervised fine-tune model as much as possible.

We want it to be-- We don't want it to go so far away from the base model, um, or the pre-tra-- or the supervised fine-tuning model. So essentially, if it deviates too much, we want to tax the, uh, uh-- we wanna tax it.

And so this beta is like zero point zero five, and the KL divergence is the dist-- Okay, it's not a distance. It's like the distance between the current model and the pre-trained model.

And essentially, we want to also shove this into the equation. Um, so you can see with PPO, there's many moving parts. It's-- Who cares about the equation? It's not that complicated.

The point is all of these extra add-ons are just to reduce overfitting and not to make the model like randomly go to some weird state, um, that like, you know, overfits to like s- your questions.

And so the trick of PPO is they just added all these terms in to make training more stable.

And so the final equation is like this, this again. Um, hopefully you will... To be honest, no one even cares about the formula. It's not that important. Um, but I just tried to like break it down into pieces.

And the go-- Remember, the goal is to maximize this equation, right? We wanna maximize it. And normally, I just like to think about this one, right? You just need to learn this one, right?

You want to maximize the probabi-- So it's just this equation. Um, remember we did the table? Just this is enough. You don't need to learn the rest of the formulas.

It's not very interesting. Um, yeah. Any questions? Yes.

Like, the reason why PPO was preferred over, like, REINFORCE or whatever is because of the stability of-

Yes, correct

... previous one. Could you go more in depth, like, what causes this instability? Why this, uh, minimizes the instability?

So the biggest problem is pretend you are like... Pretend you just started RL. Like, you, you have the pre- you have, like, the base model. You have, like, a supervised fine-tuned model, and then you do RL.

The gradient updates at the very beginning are gonna be gigantic, right? You're like, "What is two plus two?" It says four. But if it says five, you want to, like, penalize it dramatically.

And so the problem is you don't actually wanna do large steps, and so the goal is you wanna constrain it. And so the constraint factor is, like, you know, if the, if the, if the num- if the gradient update is extremely large, you just want to, like, constrain all the numbers, if that makes sense.

Yeah.

The goal is just to constrain the update, not to make it too large.

And then, so that's the clipping. What about the ratio?

Oh, the, the ratio, it's the KL divergence. Oh, sorry, not the KL, the likelihood. To be honest, I think I need to do more research. I would ask Gemini exactly [laughs] what it is. [laughs]

That's my answer. I'm probably not the best person to answer every single question. Yes, any other questions? Yes.

So for the denominator, you know, so that, this, this probability coming from, like, the reference model, is that the process that corresponds to it, or how we can... That you see on the down there.

Yes.

Um, it's the old one.

Yes.

So is that, uh, for the correct answer of, uh-

It's the model that actually created the action. And so the top one is all of the numbers that are actually, like... How do I explain this? The bottom one is the model that created the action.

So for example, you- the model says you wanna go up, down, left or right.

No, is that a, is that a correct action or the, the action-

Oh, it just created the action. Like, it could be anything.

Okay.

It could be the... So it's the ma- it's the maximum. It's whatever action the model says currently.

I see.

It might be wrong, it might be good, it might be bad. It's just any action.

Okay.

Any other questions? Okay.

One question.

Uh, yes.

Uh, has this been tried in latent space instead of the probability, uh, uh, PPO or for different probability as, uh, people try to simulate thinking in the latent space?

Or REINFORCE, PPO and REINFORCE.

I don't think so. Can I answer that question? I don't know. That's why-- [laughs] I don't know. Maybe research papers show it. I, I'm not sure. Any other? Okay. I will...

Okay, so GRPO, the trick from PPO is we re- remember, remove the value model. We get rid of it entirely. We do not want to estimate the average reward.

It's totally removed. Um, and the reward model is now removed as well for a reward function.

So we get... Yeah, we remove it, right? Remember, the value model's removed. B is a reward, value model. We get rid of it entirely. But what do we replace?

So the trick of GRPO is we do rollout or inference sampling. We get the answer, what is two plus two. You literally make four inferences. You just literally call the model four times.

It could say the answer is zero, the answer is one, the answer is two, or the answer is four. You can do, do... You do, like, you just literally call the model four times.

And you take the reward, right, zero... What is two plus two? The correct answer is four. So you want the last number to be one, but the rest is all zero.

And the trick is you literally just take the statistics of your current rollout. You take the statistics of all of this. You literally take the reward minus the mean divided by the standard deviation.

You get the Z-score. And this is your ro- this is your base model. This is your value model, right? This is-- There's no more value model anymore. It's just a number.

And so I did this on the table as well, right? What is two plus two? If you think it's zero, uh, remember the prediction could be zero, one, two, or four, and your reward could be zero, zero, zero, or one.

And if you take the mean or the average of all the rewards, you get zero point three seven five. If you take the standard deviation, you take zero point four three three zero one, and then you do the reward minus the mean divided by the standard devia-deviation, you get some numbers.

Remember, the number four is correct. That is why, you know, reward minus the mean divided by the standard deviation is one point four four. It's the largest number. And so that is why we need to, like, max- we need to essentially maximize that good answer, and we want to reduce the bad answers.

But why is it called group relative in GRPO? Because it's not just one question, it's many questions. It could be what is two plus two? What is four plus four?

Okay, but my graphs are all the same. My plots are all the same. But anyways, imagine there's like four different tables. What is two plus two? What is four plus four?

How do I create this Python function? You know, whatever. And there'll be four tables. And so the goal, group relative just means you-- for each question, we take the statistics within each group.

For example, what is two plus two? You create four-- You literally call the model four times, and you get some, you know, answers. What is four plus four? You call it four times, right?

Create Python code, you call it four times.

Yes, there are other factors of GR-- So essentially we already explained what GRPO is, right? Everything you need to know about GRPO we already explained. In to- in the total mathematical formula, it looks kind of like this.

Um, there's some rearrangement. For example, the minus beta, the KL divergence, is just taken out of the reward function. That's the only other difference.

Um, hopefully, it makes more sense about the parts of the GRPO formula and stuff like that. It's actually not that complicated to understand. The majority is just trying to reduce overfitting, right?

That's the whole goal, right? Minus beta times the KL divergence is to reduce overfitting. One minus epsilon, one plus epsilon is to reduce overfitting. The division reduce overfitting.

Everything's reducing overfitting, right? So that's all of machine learning and AI. It's just to make the training more stable and to reduce overfitting.

I would highly suggest there is-- these are the two things that I really highly suggest. Um, Nathan Lambert's Policy Gradients, um, book. It's online, though. Very, very, very, very helpful.

Um, and, um, Yannic's video on GRPO and stuff, very, very, very helpful as well. Um, yeah. And then now I will go into a Colab demonstration of GRPO. Um, uh, before that, like, does anyone have any questions?

Let me just check time. Questions, yes.

So looking at the formula, the GRPO formula, it looks like everything there is about exploitation and not so much exploration. Yeah, you can say that the exploration is between the new model and the old model, but how about, uh, if the models can look in a totally different part of the search space?

Because maybe the answer is there or, or, or better answer is there.

Because it, it looks to me that all of, all, all, all, all of these is, make more salient whatever information is in the base model in the end.

Yes.

That can mean memorization, but you make more salient the right memorization. In this case, the question of the two plus two.

To answer your question another way, I think it's actually because GRPO itself is the problem. Remember, the goal of all these algorithms is to force the model not to detract too much from the original model, right?

With this, like, minus beta KL divergence, you know, one minus epsilon. All of this is trying to make the model not go towards too far away from the original model, and I think that's the problem because you're essentially forcing the model not to go too far.

And so maybe there might be some new algorithm, I don't know, something, some other formulation which, you know, you want to go very far away. You could do that.

Um, I, I don't know if there are any...

I don't know if there's any research papers about that. I, I don't know.

Okay.

But you could-- Yes, you could do that.

Thank you.

Yes. Any other questions? Yeah, yeah. Yes.

You probably heard that Yann LeCun is saying that, like, don't believe in LLMs, right?

Okay. Yep.

And that's why he's pushing towards the JEPA-

Yes. Yes. Yes

... for the energy-based models.

Yes.

So what do you think about that?

What do I think about it? I can't really comment, but I mean, he definitely is-- You should listen to what he says. Um... [laughs]

But you don't see anything like movements in open source, uh, in that direction?

I don't think so. I think open source, we kind of got captivated by RL, GRPO. I don't think so open source people are doing whatever he's talking about, unfortunately.

Of JEPA?

I think he needs to talk about them more. Yeah, JEPA. Yeah.

Yeah. Yeah.

And energy-based models. I-- unfortunately, I don't think so open source... Yeah.

Okay.

Maybe we should talk about them more, but yeah, I, I don't think so. Um, yeah. Yeah. Yes.

Yeah. So for group relative, uh, so GRPO,

I was-- So it's, it removed the value function of like-

Yes

... the GRPO.

Yes. You got rid of it.

And then the... So for the rewards, the sampling of these rewards, I always thought it was very interesting, like, because you, you, with the LLM, you can actually create different types of samples.

Yes, correct.

My question is like, are there strategies related to understanding whether you want to have a larger variance or you want to have a larger group? And-

Yes

... making sure that this mean convert, uh, is more related to like the, the value function, the true value function of like that LLM, let's say, let's just say at that state.

Because I'm not sure, like... Because it seems like when you, you constrain the group to be very, very small, you're going to almost, uh, certainly go through some kind of bias in the unobserved variables unless you have like maybe a large enough sample that it might--

So one of these larger traces would be way, way more valuable than, let's say, traces you would never get if you had a very small batch size.

So this is more about an optimization question. So in theory, you should make the-- You should make-- For example, I just selected four, right? What is two plus two?

Create four examples. You should do, you know, as many as you like. You know, three thousand, whatever number you like. You should do as much as possible. But remember,

AI is about optimization. This is gonna take forever. You know, it's all about efficiency. So probably don't do as many as you like. Um, but you should. In the limit, you should do that.

But you know, everyone can't just wait there waiting for the computer to spin. Um, so yes, you, you should do as much as you like. Um,

that's it. But yes, also for like recommendations, when you do inference sampling, you should set, set temperature as like, you know, one point two, one point five, set min P to be zero point one.

You know, something like that. If you set temperature to be zero, you will have the same answer every single time. So definitely don't do that. But you should have high temperature numbers to make the model, you know, produce new output as much as possible.

Maximize variability?

Yes.

Uh, or distribution.

You should try your best to maximize variability. Your outputs should not be all the same. Um, if it's all the same, I don't think the model is going to learn.

So you should make it as different as possible. So that's why you should set temperature to be one point two, one point five, whatever, some large number. Don't do too large though.

Um, any other questions? Yeah. Yes.

Yeah. I'm, I'm curious of like the situation where like all the rewards are, are null or zero, basically.

Mm-hmm.

And the model doesn't have any way of learning anything.

Yes, correct.

So you're starting, like you mentioned, like yeah, you could kind of pivot them all away from their behavior maybe. Yeah. You were probably gonna talk about KL later. Um, but like, yeah, if the model doesn't have the capabilities of actually-

Yes

... answering the question-

Yes. Yes

... like all the first steps are only-- all the, all the rewards are basically giving no signal at all of the underlying thing you're trying, trying to train. Like how do you deal with that?

I, I have faced that a lot of times with GRPO. Like moving to a larger base model helps sometimes

Yes, that's a good question. So essentially, you're saying the model, if the model starts off with, like, no reward, like every single update is like zero, zero, zero, zero, zero, zero, zero, zero, zero, it's not gonna do anything.

Yes, that happens all the time. But by chance, just by chance, you know, you have like some small, little, little, little probability, just by chance you will have some reward.

So that's the trick. You will see this

after 10,000 inferences, what is two plus two? Suddenly, the model says four. Suddenly, okay, just suddenly, just by random probability. Let's make this more. That's all. That's all of GRPO.

Yeah, like this is addition, like it's a super simple task. If it was like a proof of mathematical concept, it may never come up with the right solution.

Yes, may never, but remember, you're not doing this one question. You're also shoving this together with other questions. What is two plus two? What is four plus four? What is m- the, you know, derive the derivative of blah.

Do this Python function. This step is very large. You essentially shove this all together, and the trick is, in general, it works, in general. Maybe by bad luck it might not work.

But I feel like the bad luck won't last forever because remember, you're changing the samples, right? So like the question, what is two plus two, you're changing that. The next phase will be some other question.

And so the trick is just by chance you will have a good reward, just by chance, and we just force that to be more.

Does that kinda make sense? So it's all-- To be honest, it's all luck. Yes. It's all luck. We're just guessing or we, you know, we're, we're praying that there's gonna be some positive reward somewhere in the model.

There will be negative reward, right? So like if your model is really, really bad, you, you can do negative reward. And so you just don't wanna do the negative one.

You just wanna do the negative one less.

And by miraculous probability, you know, just rely on probabilities, you will get a good reward somewhere, just by chance.

Does, does that kinda make sense? I mean, all of the large model labs are just literally relying on the fact that that's what they're doing. They're just guessing. We're just praying for the GPUs to work, and then suddenly the reward comes out.

I'm being serious. That's exactly what they do. They're just waiting for the algorithm to work, and then suddenly, oh, okay. That's why people do random seeds as well. So for example, the initialization of the model might not be good, so you just kill the training run.

You do like five hundred training runs. Oh no, four hundred and ninety-nine of them are like zero reward. Oh, just kill them all.

No, like-

Don't release them

... to extend to that topic, like maybe it's, it's worth it. Like if... I have seen online trainings are like GRPO, most of them stays flat forever.

Yes, very common.

And, and spending some more time on doing a small SFT step, so the model may get like five percent of those rewards correct-

Yes, that's the co- yes, I was gonna show you guys that

... before it take off.

Exactly. So like you could force the model to answer some question. Like for example, you ask the question, what is two plus two? It's very easy, it's four. You just force it to learn, oh, okay, it should be four first, and then you do other steps.

That is actually why... Remem- Okay, I have to go back to all the slides. I don't remember... Okay, where is it? How do I... Okay, I'll exit. Uh, it's the same as this problem.

Where is it? This one. Right, someone asked about why don't you just start from the blu- you know, the pre-trained model to go to the green one. It's the same thing.

Essentially, the trick is we want to do some supervised fine-tuning to make it know some instructions, so it knows something, and then you wanna go to the reinforcement learning phase.

But if you wanna start from nothing, like just the pre-training phase, that's the hard part, right? Your reward might be zero, zero, zero, zero, zero, zero, zero, like zero forever, and then suddenly one, you know, suddenly.

No, but I start from ping most of the time. Like I always-

Oh, if you start from ping, ge-

...

Okay.

Different models.

That's just unlucky then. [laughs] I, I think like if you, if you see z- zero rewards, most likely either, one, your reward function's not that good. Two, yes, doing priming or like, you know, make the model learn a little bit about your data is actually does wo- does help.

Um, so there are tricks to make it work. But generally, I would just say it's bad luck, just bad luck. Um, and unfortunately, you can't do anything. It's not your fault. [laughs]

It's, yeah, just unfortunate. Uh, yeah, yes.

How should we think about like when is... Or like what to expect from GRPO? Like are we... Is, is it gonna be just that this is the way open source catches up with closed source models?

Or is it something... Is another tool for like your average or like your, your competent ML engineer to be able to like specialize a model for a task? Like I, I just don't...

Yeah, is, is there consensus on like where, what this is gonna bring us to? Is smart people like you gonna give us a really good open source model, or is it that, you know, we should think about this as a new tool for specialization?

The algorithm is not special. The hard part is actually the reward functions itself and the data that you're gonna shove into the model. That's the hard part. So like I, I think there is a misconception like, you know, the algorithm is important.

No, it's not. It's useless. Who cares about the algorithm? You can literally just use, you know, the general, the function which I gave. You can just use any algorithm that you like.

But the problem is actually the reward function itself. I gave you some examples, you know, like what is two plus two? The answer is four. Yes, you can do distance based, but that's just one example.

Can you... Like, can someone make a function-- Can someone make a reward function for like trading? Stocks. That's-- You do that, do that, right? And then there you have like a model for trading.

Go ahead.

So you, so you think it's going to be more that because people are able to create reward functions like a little bit more easy, like similar to like you could create a prompt.

It's, it's an easier thing for most people to, to like iterate on, um, than-

It's actually quite hard

... coming up, coming up... It's, but it's easier than coming up with the algorithm itself or whatever.

Actually, I think col- collecting... So in the olden days, large model labs will ask like, you know, large data providers like Scale or whatever to create data. Like, what is two plus two?

You literally have someone sit there and write, "Okay, the answer is four." But then you also have to do the chain of thought. Like, "Oh, I think the answer is four because of blah, blah, blah, blah, blah, blah, blah."

Or like, you know, "This is my working out." You literally have to ask someone sitting there to make the data.

Mm-hmm.

The trick is ge- no more. You don't need the data labeling step anymore. It's totally gone You have the answer and you have the question. The middle step is totally removed, but you still need to make the reward function.

You need to ver- you need to say, "Is the answer four good or bad?" For maths, it's very easy. For code, for code, it's somewhat easy. You know, you can check, oh, did you import the correct function?

Did you import the library? Did, did your code execute, you know, some other reward functions?

Mm-hmm.

But it's still hard to verify if your actual function's correct. For example, let's say your question wa- let's say the task was create the Flappy Bird game. How do you actually know that the output's good?

How do you actually know? We don't. You, you could, again, ask a human to verify, or you know, [smacks lips] let's test the Flappy Bird game.

Mm-hmm.

And then give it a good reward or bad reward. Or the trick is, did the game actually run? If it ran, plus one. Did you see the word Flappy Bird in the, you know, Flappy Bird inside of the functions?

If yes, plus one. Did you see the, you know, the image of the Flappy Bird sprite be used? If yes, plus one. Yeah, something like that. So like you don't-- It's still...

I would say the hardest part is writing the reward functions. And for open source specifically,

if, you know, the whole open source community starts writing reward functions, we can probably beat o3, o1, like, you know. Okay, plus compute, you still need compute. That's the problem.

You still need compute. But if you write good reward functions, you'll probably catch up in no time.

So the end state here is that you want an open version of the closed model. It's not... [stutters] I guess my original question is like, is this a thing that m- people are gonna use like prompts to have specialized models separately, or is this more that we want one or many good open models?

It de- Okay, that is a good question. It depends on which school of thought you're in. If you're in the thought that large language models already have the capability and you're just trying to accentuate it, then there'll be just one model, yes.

But this model can only learn some facts, because otherwise you're like over fitting and like it's not... But if you're in the second ha- camp, that the model actually learns something new...

Oh wait, did I say it right? I think it's the opposite way around. Um, the first one is you have many models. I so- sorry, I said it wrong.

The first one is you have many models, because the model doesn't actually learn that much. But if the second one is the model actually learns everything, then you have this one gigantic model.

I think OpenAI probably ascribes to that point. You know, like most large model labs think that actually RL can get you to AGI, right? It will know everything about everything.

Any single question you ask, it already knows. And so like, that's, I think that's where they're trying to go for. For open source, it's more harder. I think the open source community consensus for now, for now, is the model, it already knows your questions, and you're just trying to accentuate it.

And by doing reward functions, you're trying to weight the circuits more. You're trying to weight the model to know how to do these equations and stuff like that. So I think like

the goal of open source is, you know, if the entire community comes up with good reward functions, write them all, [smacks lips] then the problem is we need compute. That's the second problem, right?

If you shove both of them together, you will get O- O10 or something. I don't know. Um, right, imagine if every single person writes a reward function once per day.

Okay, that's probably too hard. Once per day, we have like, you know, seven billion reward functions, more than OpenAI can ever come up with, and you will defeat OpenAI.

But, you know, you need the compute part. That's the only problem. Um.

Thanks.

Okay. Any other questions? Yes.

Oh, sorry. Okay. I'm curious because when you said that like, um, yeah, we know GRPO is gonna get traces that may get correct, so those get rewarded.

Yes.

Um, how do you feel about, like, saving those traces-

Mm.

-and just doing SFT on those traces?

Very smart. That's what-- Yes, yes, yes. I don't know if large model labs do that. You could do that, yes.

'Cause I feel like, yeah, we're just mining for those-

Good examples

... traces that are correct.

The only problem I would say is pretend the question was what is two plus two, right? And then the model says... [smacks lips] Okay, the, pretend the model just says this, okay?

It says, "Oh, let me work out what is two plus two. I think it is, oh, you know, the number two means two apples, and I want to add two more apples.

I think the answer might be three. Hmm. But let me rethink about it. Wait a second, it's like four." Is that a g- Should you fine-tune on that? I mean, you, you could, but maybe it's like cheating.

Maybe it just says four by chance. Maybe like... Okay, I'll give you another example. Let's say that by chance... Okay, the question was what is two plus two? It says gibberish like, um, "I like to go to Paris for fun or whatever.

I don't know. I like to go to the, you know, this event," blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah. And then suddenly it says four.

Yeah.

Just by chance. Remember, we're still rewarding this. We're literally rewarding this as good. But this is not good.

Yeah, but that's how we reward that slide back as-

No, so we, we reward it at the very end. Remember, we see the number four, it is good. The question was what is two plus two. The model can generate anything it likes.

It could-

And we can reward that path too, right?

Well, you, you could, but that gets harder. So the trick is people don't actually reward the steps in between. They just do the final step because otherwise it gets too complicated, right?

So like, what is the intermediate steps reward? It's way too complicated. So what you do is you just reward the final step. So if, if you see the number four, it's good.

But we don't know how we got there. So you should-- So yes, you could maybe at the very, very, very end step of RL, you can then use some data to do, do, um, fine-tuning.

Yes, you could. But I think like in general, it's because we don't know what the process is in between.

When you say we don't train on that thought or random-

Oh, no, we don't train on the thought. Yes, we don't. But rem-

Is that a problem if we train then?

We don't train on the intermediate step in between. You don't have to.

No, you don't have to. Yeah.

You, you could. You could. But remember, we don't know if the trace is a good or bad. We don't know. So you can't just take this trace and then do supervised fine-tuning.

Because pretend the answer was four is good, but we don't know the intermediate steps unless if you read the data, right? You could ask s- you know, some human labelist to like, "Oh, you know, please verify if this trace is good."

You could, you could. But then that kind of defeats the whole purpose of RL. So, like, you don't, you don't wanna do this. Um, does that kinda make sense or not really?

Yeah.

Okay. Uh, yes.

Can we see the demo?

The what, sorry?

The demo.

Oh, yeah, yeah, yeah. We... Yes, we will do that. Yes. Yes. Uh, yes, uh, que-

Um, if we have multiple categories of reward function, like one for math, one for Python-

Yes

... what's the normalization among the rewards that are, like, best practices?

W- what do you mean by normalization amongst rewards?

So, like, if we have, you know, for four, you know, for the math one, two plus two, four gets a one, five gets 0.5, A gets zero. Then you have one for Python that might just be a does this compile, it's a binary, um, reward function.

Yes.

If we're running this in one big clump-

Yes. One batch. Yeah

... like, what... How do we normalize the fact that-

Good question

... we may have totally different types of rewards?

Correct. It could be like minus 10.

One row at plus 100 to negative infinity, the other person wrote a binary-

Very good question. That's your choice. Unfortunately, that's the problem of RL. It's all about human choice. Like, you, you will have to decide. You know, is the Python one more important than your, um, than your maths, what, what is two plus two?

Then you can weight it more. For example, your make two plus two the reward as minus one and one, and the Python function as, like, 1,000 and zero, right?

You, you have to decide on the weighting functions. That is, that is your choice. Um, unfortunately, it is a...

It's kind of like an art. You could, like, dumbly, you could just do everything as the same scale. I think that's what most large mo- I think that's what the large model labs probably do, is, like, everything, all the reward functions have the same scale.

You know, plus one, minus one, plus one, minus one. Not plus 10 and then minus 1,000.

So it's up to you.

Okay.

Yes.

And what about stuff like, you know, is this a good summary? Is this a poor summary? How, how are we, how are we creating the reward function?

That is the question. So, like, now you want to, like, analyze... Okay, that's where the LLM as a judge comes in. So, like, there is a school of thought that you can use a language model itself to, to make a number.

You can ask ChatGPT, "Is this a good summary or is this a bad summary? Please give me a score from minus 10 to 10." Ask ChatGPT. You, you could do that.

That's called the LLM as a judge thing. Um, there is a paper which shows that you can do this for some time, but then it breaks down. So you can't just keep calling the language model, you know, like...

It's kind of like cheating, if I would say. Like, you're trying to call ChatGPT to train ChatGPT. Like, it will work for some time, but then it will break down.

There was a paper, uh, I, I need to find the paper, but the paper showed that if you keep doing this, your actual reward actually goes backwards. Um, so, like, you will get more and more and more reward, and then suddenly it just, I don't know, by bad luck, again, it's always about bad luck, the reward just

goes back. Um, so... Yes, I'm being serious. Like, all of AI's about bad luck and good luck. Um, and, you know, optimization, trying to do efficiency, um, that's what everyone...

Yeah, un- unfortunately. Um, so, yes, you can use LLM as a judge. Um, is that kind of-

Okay. If I use LLM as a judge, doesn't it just end up being a teacher and student model or?

Yes. Y- but that's why, like, that's why sometimes... Essentially, the problem is, I, I need to find the paper. If you keep doing this, it will actually do bad.

I mean, intuitively, it kind of works, but then at some point... There is actually another way. You could ask a language model to generate reward functions. That is actually another school of thought.

You can actually ask a language model to generate 10, 7 billion reward functions. But the question is, are, is the reward functions good or bad? I don't know. Um, so, like, you now you need to, like, rely on the fact that the models are good or bad.

You could then ask another language model to verify the reward functions. So, yes, you could do this. Maybe that's what OpenAI's doing. I don't know. Maybe OpenAI's goal this whole time is, like, "Oh, let's generate all these reward functions, verify each of them, and then shove it into the function.

Let's see what happens." Maybe that's what they're doing. I, I don't know. Um, but yes, it is a student teacher. Um, uh, yes.

What's your opinion on how to make reward models at scale efficient for the open source world? Is it a bunch of distributed experts? Is it inverse learned? Is it, is it something else?

Do you mean, like, how to make reward functions more efficient in general?

No, I mean scalable in the sense of going over many different domains-

Mm

... comprising, uh, past just coding and math.

Yes. The majority of reward functions currently, that's why it's called verifiable rewards, is, like, maths and coding. To be honest, I think coding's also hard. I th- I don't know why people lump...

So coding, you can't actually verify technically it's correct. You can just say it ran, or the output is most likely correct, right, for some functions. But for example, the Flappy Bird game, tell it to create a Flappy Bird game.

How do you actually verify if it even is the Flappy Bird game? I don't know. But you could, right? That's the whole point of the LLM as a judge.

You could take the output of the Flappy Bird game, ask the language model, "Does this look like the Flappy Bird game?" And if it's yes, okay, plus one. If no, minus one.

You could do that. Um, but you can only go so far. If... Does that kind of-

I'm just wondering what, what your opinion is to scale beyond that.

Oh. Hard to say. I think most... I, I think large model labs, they're currently just trying to use their own model to, to r- to, like, literally reward, reward it.

Like, a- as I, like, described. You know, ask it, "Oh, does this look like the Flappy Bird game?" If yes, plus one. If no, minus one. And I think maybe that's what...

I think large model labs, their view is if you keep doing this, you'll get to AGI. That's their view. I mean, if you think about it, it... You could, maybe.

But then I always go fall back to, oh, but you might be bad luck, it's not gonna work. Um, so, like, I, I think in general, I think bad luck will just, it won't work.

Um, you will only get so far, and then suddenly it just doesn't work.

Does that... Okay. Any other? Y- yes.

Um, how do you think about the scope of the task that you're, um, instilling to the model? For example, you can, you can train for the math, you can train for coding, you can train for planning.

Do you think if you keep it very, um, task diverse, does it work well or, or should we specify the specific task?

That is your choice again. So, like, if you want to specify, for example, you just want to make a legal bot, you're given some sort of court case, and if it's like, you know-

... the plaintiff wins or the defendant wins, I don't know. You could just do law. Yes, you could. You could do that. But in my view, you should combine it with other sources.

You should combine it with some maths. You should combine it with some programming because the point is you don't want the model just to know, like you just, you don't want the model just to like overfit just to just law.

Maybe maths might be helpful just by, you know, by chance again, maybe. You know, maybe coding might be helpful. Uh, probably not, but like, you know, in general. Um, so like you should combine other source, um, other domains together.

Um, I feel like, you know, all the large model labs, their goal was to do every single domain possible, right? Like mine every single reward function in the whole world, make all the reward functions, shove it into the model, and just learns.

Um, so like, yes, you should do more domains if... Yeah.

There's another... Yeah.

Yeah. I have a follow-up question to the practical point of view. So, so what, what is the latest research in should we fine-tune... Say, say let's say for incredibly one, a smaller model.

So should we fine-tune the smaller model first or, or go with feature distilling to smaller model? Is that better for this particular thing, you know, uh, using GRPO and Q-learning?

So the notebook I will share will showcase you should probably do some supervised fine-tuning first. It's called the priming stage. Um, otherwise, you'll... Remem-remember the plot over here, the, the, the, this one, right?

You don't wanna be in the situation where like you're starting from like some bad, you know, pre-trained state and you're trying to go to the RL stage. Very not efficient.

But remember, AI is all about efficiency. You don't wanna do this step. So we do have to do some priming, you know, the SFT stage and the other stages, if that's your question or...

So should I, should we use bigger, bigger model first?

Oh, if you want to, yes. The bigger the model, the better. Yes.

I mean, it's, it's extra cost, you know. So this would... If you, if you can just do, if you just train the smaller model, do you think it's still efficient?

What's your-

That's the trick. So essentially, the research papers show that small models actually do work, confusingly enough, because essentially these small models, it just does longer thinking. It does longer reasoning traces if the model's smaller.

And if it's a larger model, maybe the m-reasoning trace is like smaller in general. So like I, I feel like the small models actually do work. They do break down though.

If you wanna do like very complicated reasoning traces, then maybe the small models might not work because, you know, there is only seven billion parameters. There's no-not that much space you can move.

Um, and so the large models, you just have more space to move around. And so that's why large models are better. If... I don't know if that maybe kind of...

I don't know if that answered your question, but-

No, yeah, yeah. That makes sense. So another question I have is one task that... It's like just why we, if we want to fine-tune for our use case-

Mm-hmm

... like we need one task. Can we use the-- We can also use one, the distilled DeepSeek model, you know, which is already fine-tuned in other tasks. So we can, instead of working with base model, we can take that and then fine-tune.

Yes, correct. Exactly. Yes, that's what you should do. Yes. You can take a distilled model, like a already reasoning model, and then further fine-tune it. Yes, you could. Um, I would say it's a bit more complicated because you could do that, but remember, the reasoning model itself is already a reasoning model, and you're trying to fine-tune it

to become other re- Like you're trying to do some other domain. It might be easier, it might be harder. It's all about luck again. I don't know. So you have to try.

It's all trial and error. Um, yeah. Yeah, yes.

Two questions for you. One, like, um, in sort it's pretty empirical.

Mm-hmm.

So you can just like try and see what works and what doesn't and see what sticks.

Yes, correct.

Okay. And then the other side, uh, how are you keeping up with all the papers and all the content that's being put out? Like I'm sure it's a lot.

How are you learning, like some... The principles that you're like using to follow like latest work?

To be honest, I don't-- You don't need to follow. That's my view. Don't try to follow the latest research because sometimes there may, like in the next day, it's like rebuttal of the previous paper, and then the next paper says, "Oh, it's a rebuttal of the rebuttal."

I don't know. So I would not try to keep too much up to date with the latest research. I think the field has kind of matured, and it is mostly stable now.

You might have like some algorithm increasing accuracy by one percent or two percent or some efficiency. Remember, all of the papers are about efficiency. It's always about efficiency, making the training more stable, reducing overfitting.

It's always these similar, similar papers. Um, so I would say don't... You can keep up to date with papers. Twitter is very good as a resource. Sometimes I tweet about papers, although I, I don't suggest...

The Nathan Lambert paper-- The Nathan Lambert, um, book is very good. He keeps updating it. Where is it? Where did I put it? Um, this one. The RLHF book.

That is very good, so definitely read that. Um, he updates it all the time. And so maybe follow Nathan Lambert. He's actually a very good tweeter on like the latest research.

Um, so he's very useful. In general, there's a lot of noise in the RL space as well. You don't know if the research is good or bad. Like, you know, rebuttals on top of rebuttals.

So I would suggest people just to like try... It's just trial and error, right? Try to see if a reward function is good or bad. You know, is the loss not...

You know, is the reward just zero, zero, zero, zero, zero? Like unfortunately, something's wrong or you just... It's bad luck. Try again. Um, so it's just empirical. Yes. Um...

Are you gonna put these slides up anywhere?

Sorry?

Are you gonna put these slides up?

Oh, yeah, yeah, yeah. Yeah, yeah. Um, yes, these slides should be up. I was supposed to make a Bitly link. Um, I'll probably do that later, but I will share the slides.

Yes.

Like can you just post them on Slack, the engineer Slack?

Oh, yeah. Okay, I'll do that then.

Appreciate it.

Okay. Okay. Any... Uh, yes.

Yeah, I have, I have two questions. So, um, the idea that the capabilities already exist and-

Yes

... during, uh, reinforcement learning, you're not adding new capabilities. When you're talking about the value model and the reward model, is that something that you are creating in the-

In the old PPO sense, you are-- the value model is a new model. The reward model is a new model. Yes. Remember, in GRPO, we delete the value model.

The value model's totally gone. We, we create the value from just statistics from the distribution. We essentially just create, you know, four... What is two plus two? Just create four examples, four trials, and then find the mean, find the standard deviation, and the data is your va-- data is your value model.

It's not even a model anymore. And then the reward model is no more as well. It, it's just reward functions. And that is why we call it reinforcement learning with verifiable rewards.

It's not normal RL anymore. It's like you replace a reward model as well if... Does that kind of make sense?

That does make sense.

Okay.

And my follow-up question is-

Yeah

... so is it sound intuition that then there are some circuits within the existing model that have w-this, uh, idea of a reward, and so you're kind of weighting those, uh, circuits to approximate the reward that we are getting during training?

Yes. So the-- [lips smack] Again, there's two schools of thought. The first one is like

the, the question, what is two plus two? Somewhere in the model, somewhere, I don't know where, right, this high dimension of one point four trillion parameter space somewhere, it knows to calculate it as four.

Right? There is some sort of circuit inside the model, and then the goal of RL is just, just to maximize this circuit somehow by via, well, by these formulas and stuff.

You're just trying to maximize it. But that's one school of thought. The other school of thought is, oh, you know, RL's actually learning something. You know, like, it's actually learning how to do two plus two is equal to four, and it's not actually in the model.

Thank you.

Okay. Any other-- Yeah. Yes.

When you say capabilities-

Mm-hmm

... of, of a model that already has them inside-

Yes

... you mean knowing actually the, the answer to a question or knowing how to reason to get to a question?

That is a good question. Maybe both. I, I think it depends. It probably knows how to do the re...

It might-- For example, a contrived example. You get all of the entire world's data, right? Like, you know, thirty trillion tokens, get all of the data, and you just make a question that is not part of the data, right?

You, you could do that, right? What is ten billion, you know, some random number times some random nu-- You can make a math equation which is not in the data.

But somehow the model has learnt to do multiplication, has learnt to do addition somewhere. And so maybe this circuit for addition, for multiplication, you know, for whatever, some, you know, many, many, many circuits of these, like, functions, we just wanna accentuate them all, and that is what kind of RL is trying to do.

If... And yes, there's a reasoning circuit. So like somehow the model also learns how to do reasoning, and so we also want to make that more important. And so, you know, two plus two is very important, addition is very important, multiplication is very important, and so on.

We're just trying to like make all of these circuits more prevalent. But that's only one half of the AI community, right? That's only one half thinks like that. The other half is like, "Oh, but the model's actually learning.

You know, we're actually training the model to learn, and the base model actually doesn't know how to do reasoning." If... Does that kind of-

Okay.

Okay. Yes. Okay. Hopefully. Okay. Any other questions? Yes.

So last question on that. So for, uh, RL framework, what frameworks do you suggest? Like, I, I see the notebook, we use, uh, a new base one, but there is also popular one, VERL, V-E-R-L.

Yes.

So which one-

There is VERL, there is TRL. Um-

Yeah

... Unsloth, like we, we also make a... It's not called a framework. It's more like we showcase that you can do, uh, GRPO and reinforcement learning with very low resources.

So we are the only package which allows you to do GRPO on like a free colab. And so, like, that's the only difference between us and everyone else. So for VERL, VERL was very good for large training runs.

But for now, Unsloth, like if you want to do like small experimentation, you want to try stuff out, you don't know what reinforcement learning is, you don't know how to make reward functions, you don't even know what reward function to do, you should utilize our notebooks.

And that's what I was gonna demo. Um, yeah. Yes.

So one question is, uh, RL, like let's say you don't want to use a thinking model or do chain of thought. You just have a regular task, right?

Mm-hmm.

Like which say calls some tool, et cetera. So, uh, what are your, like, uh, if you've done any experiments on that, what is the, if, uh, effectiveness of that?

Like using RL to just improve, like, let's say, tool use for your use case.

Yes. Yes, you can do that. Exactly. It should increase accuracy by quite a bit. If your accuracy, if your accuracy before with tool use was not very good, RL should definitely help.

And if you're like R-- The trick of RL is it reduces overfitting. I think that's the trick because you do, you do multiple inferences. You don't know which one's correct, but you're trying to like maximize some good ones, right?

Some good ones. The problem with general fine-tuning is you're kind of like overfitting the model. And so the trick of u-- the trick of reinforcement fine-tuning is you can essentially reduce overfitting.

So the model actually learns how to do tool calling, not just, "Oh, you know, I see someone's trying to do a restaurant order. I just want to call DoorDash to do order," or something, right?

But it actually learns, "Oh, okay, because the person wants to order food, I should or-order DoorDash." It's like reverse thinking. Um-

Yeah.

So it should definitely help.

Yeah. Because in some of my experiments, like unless, like I try not explicitly asking the model, like a regular instruct model, right? Go and open it to, like, just do some task, right?

And it does not, or unless it, uh, it does not automatically start the thinking process unless you explicitly prompt it that, "Okay, first, uh, think and then answer."

Yes.

I was, I was trying to check if like j-just without explicitly asking it to, uh, start the thinking process, just to improve the tool use accuracy itself, does it start thinking?

And that, I think if that did not happen because it, uh, yeah.

You don't need to do... So GRPO, generally people utilize GRPO and like reinforcement learning algorithms to create the thinking process.

Mm-hmm.

That's because it's like a artifact of GRPO. Just by chance, they see the reasoning process. You don't need it. So maybe by chance, by luck-

Somehow it learns how to do tool, tool calling, and it's not some thinking process. It could be some weird symbols maybe. You know, I don't know. It could be like using some other ...

You know like sometimes models have like different languages suddenly? It could be like that. You know, randomly it learns how to do tool calling. It made some new programming language internally, I don't know, but it could have done that.

In this case there was no thinking process.

Yes.

It just directly gave the output because, like, let's say you sample like 10 trajectories, and none of them have a thinking process. Then it's never, it never explores those paths at all.

For now it will never. But remember, it's all about luck. Over time you will have a thinking process. Just by miraculous chance there was a thinking process somewhere, and then, "Oh, you should do this more," and it would just do this more.

But to make that more probable, like we should prompt it, uh-

You can prompt it. So essentially in the system prompt, you can say, "Please put your working out between this box." You could force the model to create the working out.

You could do that. Y- you could. Um, but is this the most efficient? I don't know. Y- you could. You could say, "Oh, please create a new language that I don't understand which does tool calling," and then it does some weird symbols and then, oh, it do- does tool calling.

I don't know. But yes, you, you should prompt it. It should make it more effective.

Mm-hmm.

Okay. Any other ... Yes. What's the secret sauce of Unsloth to make the training fast and, uh, efficient? Oh, we, we utilize Triton kernels. We do like kernel optimizations.

We reduce memory usage by 70%. Um, there are lots of optimizations that we do to make training faster, and more memory efficient. Can you talk about it later? Yes.

Later. Yes. Yes. It's not the main focus, but yes, we will talk about that. For vLLM specifically, we use ... Okay, that's actually the notebook. Oh, before that, do we have any other questions?

I, I ... Yes.

Real quick, like, with all the focus on test time scaling these days, and also papers, uh, claiming that even the test time reinforcement learning can get efficiencies in performance-

Mm-hmm

... do you see any kind of reinforcement learning that kind of improves the test time learning capabilities in terms of-

The ... Actually, the DeepSeek paper talks about this. There is pass@K and majority@K. I think they said that if you do test time, test time scaling, I think it improves majority.

I, I think that was correct. You need to read the ... I'll have to reve- revert back to the DeepSeek paper. But they did say ... Remember, test time scaling is different from reinforcement learning.

They are different methodologies. Test time scaling is calling the model 10,000 times, and then, you know, then you just check, you know, by average what ... Okay, you, you ...

For example, you ask ChatGPT, "What is 2+2?" It might say 4. You know, it says 4, 4, 4, 4, 4, and suddenly it says 5. I don't know, by chance it says 5.

It says 0. And you just take the most likely answer. That's called test time, test time scaling.

Right.

And then reinforcement learning's kind of different. It's more like, oh, we want to like make ... We want to actually train the model to actually do the whole trace, and you don't, you don't need to do ...

You don't need to output 10,000 examples and get the best answer. You just do one.

There's also test time optimization. We have few, for example, kind of optimizing your, uh ... Things like DSPy that, that does, like, you know, prompt optimizations things like that, right?

So there's optimizations you can build on a test time-

Yes. Correct

... as well.

Yes.

Is there anything that is feeding into the reinforcement learning methodology itself that improves the test-time optimization?

Yeah, so the trick is you do the RL step, and then you do test time compute. You will actually make the accuracy much higher.

There's no pre-time there.

You ... That's a good question. You could. Kind of GRPO's kind of like that, right? So GRPO you do test time scaling in, in the actual reward function. You literally call the model, "What is 2+2?"

Do test time scaling, and then you aggregate the results. So it's like GRPO itself is doing test time scaling internally. You could

... That sounds like a new research paper. You could do that, I guess. Yes.

Okay. I will have to go to the notebook now. Um, so in order to access the notebook, you can go to our GitHub page, um,

which ... Until my internet actually loads. Um, so if you go to Unsloth, right, the GitHub page Unsloth, there is a button called Qwen3 GRPO, right? And you can click Start for free.

That's how you get the notebook. Or you can go to our docs, um, which have the notebook. Um, so remember, go to the GitHub page and then click Start for free, um, yes, for Qwen3 GRPO.

And then you will get this notebook. Um, generally it's dark mode, but I know in presentations, you know, presentations, I don't think so people can see that, so I will change this to li- light mode.

Um, so we utilize vLLM behind the hood. So vLLM ... Does, does anyone not know vLLM? I think that's a good question. Who does not know vLLM?

Okay. 100% you must use vLLM, right? So, like, for all open source, how do you serve a large language model? Please use vLLM or SGLang, um, or, you know, uh, I think Hugging Face has one as well.

So, like, these are the best open source libraries to serve open source models. Um, you know, like you have a GPU, how do we actually serve Llama 3? You know, how do we serve Llama 4?

You use vLLM to serve it. The trick of Unsloth is ... So Unsloth is a package for fine-tuning, for GRPO, for reinforcement learning, for whatever you like, continue pre-training, whatever.

And the trick is we just optimize it. We make it much faster, you know, use two ti- use, um, 70% less memory, um, make it fit on a free Colab.

Um, remember, please use free Colab resources. You know, Kaggle has 30 hours for ... I already said this again. Ka- Kaggle has 30 hours of, for free per week of GPUs.

Please utilize them. Um, they won't be unhappy. You know, please utilize them. Um, and yeah, so you install Unsloth and vLLM. Um, and we have this thing called the FastLanguageModel class, which essentially you can call a model.

Um, for example, we, we will now utilize the Qwen3 base model, right? So, like, remember I told you not to do this, right? But we are ... Anyways, we're gonna do it.

Um, this plot, where is the plot? Um-

We are going to do the dark blue dot to,

to the, um, the dark green. Um, yeah, that's what we're gonna do. [laughs] We're actually not gonna do what I suggested not to do. Um, but anyways, um,

we're gonna do that. Um, you also have to se- set a max sequence length. So for example, if you wanna make it longer, you can set it for longer, right?

If you want longer reasoning traces, you can increase the maximum sequence length. We set it as two thousand and forty-eight. Um, if you set it for larger, the free GPU will run out.

Um, so that's the problem. You can also load in four bits. So if you wanna do four-bit quantization, you can make the model go to four bits. You can reduce memory usage by quite a bit, so you can do that as well.

Um, and remem- we are utilizing LoRA, which is a parameter efficient fine-tuning method. Um, you don't need to fine-tune every single weight inside the entire model. Um, this will be very, very, very costly.

Instead, we add small weights to the model to fine-tune it. Um, and so that's the trick that we do.

And because we utilize vLLM directly, we do a trick. We actually re- reduce memory usage by fifty percent. The trick is we share vLLM's weights directly. Um, other training frameworks, what they do is they have to copy vLLM's weights, um, because you have to...

You have the model for fine-tuning, and you have the vLLM weights. The trick that we do is we actually share the vLLM weights directly, so you can reduce memory usage by a further fifty percent.

We use something called the unsloth gradient checkpointing, which reduces memory. Okay, essentially everything in AI is about reducing memory usage, more efficiency. You know, everything that we do is just efficiency.

Um, so everything that we set is for efficiency purposes. Um, your LoRA rank for, if you do LoRA, please set it to be the alpha to be two times the LoRA rank.

Um, it, it speeds up training dramatically, so please do that.

Some, lots of stuff, right, compiling. We do, like, automatic compiling and stuff like that. You don't need to read the, all of this. Um, and here is the bulk.

This is the most important part. Someone was asking about, you know, about a prompt. You make a system prompt, right? You are given a problem. Think about the problem and provide your working out.

Place it between reasoning start and reasoning end, right? So, like, the reasoning start is start working out and end working out. So it should look something like this. Um, if you, you know, it does, uh, start working out and end working out.

Right. You are given a problem, think about the problem, provide your working out. Place it between start working out and end working out. Then provide your solution between solution start and solution end.

So it should look something like this. Um,

and this is the system prompt that we're going to use for reinforcement learning. Remember, you can customize this to however you like, [snaps fingers] right? You don't have to say you are given a problem.

You are given a legal case, right? Think about the case and provide, provide your, I don't know, legal thinking. I don't even know. I'm just making stuff up. I don't know.

Whatever. Provide... Place it between, I don't know. It can be literally anything. S- What, um, thinking. I don't know. I don't... You can even make spelling mistakes. It doesn't really matter, right?

End thinking. The whole goal of RL is you can design your reward, you can design the system prompt to whatever you like, right? I think that's the main problem, is like people think, "Oh, you must follow DeepSeek think," right?

Think, right? You need to... People see in DeepSeek think, right, and </think>, right?

You do not need to follow, uh, you do not need to follow this, right? You do not need to follow this at all. You can make it up entirely, right?

This is customizable to whatever you like. The hard part is because we are using a base model, remember this is a base model, you have to make a chat template as well.

Um, this is the more annoying part. You can just copy and paste this chat template. You do not need to do anything else. Um, just, you know, literally copy and paste it.

Um, the base model does not have a chat template, right? A base model, when you call it, you can't actually call it for conversation. You can't... It's not ChatGPT, right?

It's just a base model. It doesn't do anything. So you need to specify a template for it to understand how to do conversations. And so this is kind of like a template that we did.

It's a very, it's very generic. You can just copy and paste this. It should be the same for anything.

And then we show... So after you, uh, do the chat template, we show an example of how to actually utilize the chat template and the tokenizer, right? So for example, if you ask what is one plus one, you do the reasoning process.

It will say, "You are given a problem. Think about the problem and provide your working out," blah, blah, blah, blah, blah. And then the question, this is the question.

"Your question is what is two plus two? Remember, the answer is four." And so start working out. This is what you give to the model. You give all of this to the entire model,

right? All of this, um... Okay, I can't really highlight. All of this you will give to, into the model, and the goal of RL is you want to create the working out process automatically, right?

The RL algorithm will automatically create the, uh, you know, the working, the working out or the thinking process. And then finally, it will say four. Well, hopefully it will say four.

And the goal is if you see the wor- if you see four, you want to make the reward higher just for that.

Now, someone was talking about, um, you know, fine-tuning with the, um, you know, instruct fine-tuning first. Again, remember, we go back to this diagram. We wanted to start from the blue dot to go to the green dot, but we found it doesn't actually work. [laughs]

So don't actually do this. The trick is we go back to this diagram. We actually want to take the pre-trained model, do some fine-tuning, do some supervised fine-tuning, and then go to the green dot.

And so this part we show that you should actually do some supervised fine-tuning. You need to do some fine-tuning to prime the model, right? The goal is you want to learn, you make the, you want to make the model not just output zero reward forever.

And so this dataset allows you to prime the model to do supervised fine-tuning.

So for example, the problem is, you know, what is the sum of all the real numbers, blah, blah, blah, blah, blah. And then you use DeepSeek-R1, so this is a trick, this is a hack.

You use DeepSeek-R1 to create some examples, and you shove this during the fine-tuning step. And essentially the model already learns how to do some reasoning, and that's the trick.

This data set is very small. It's only seven thousand rows, right? You don't need to have that much data for just this first step. You can have, like... I think I only used six hundred rows.

Very, very, very less data. So this is just data preparation step, not that important. Um, I need to skip to the reward function. This is the most important part of the model.

So this is the supervised fine-tuning step. So all of this is the supervise- supervised fine-tuning step. So this, this part, this part of the model, um, that's this part.

So not that important. Um, okay, we skip all of that. This is the most important part. The reward function creation is the most important part, and I feel like, you know, the majority of people, like, neglect this part.

It is the most hardest part to do. Um,

okay, let's see. Where is the reward function? Oh, here it is. Okay.

For example... Okay. No, no. Th- this one is a regular expression to match if your format is correct. For example, remember we have-- we asked the question, we asked the model to say, "Please put your working out between start working out and end working out."

This regular expression essentially, um, essentially rewards the model to see start w- working out and end working out. If it doesn't have this, you will actually penalize it, and so this is one reward function that I created.

For example, I give it an example. If you say, "Let me think, end working out,"

it extracts two. Yes, that's good, but remember, we force the model to say, we force the model, "You must generate the answer between this and this." And it successfully extracted two, so that's good.

But, you know, also sometimes a model might generate some random spaces. It's possible. You know, the model might not actually-- the model might not follow your exact format. We still try to match it, right?

Even if it generates extra spaces, we still successfully match the number two, so that's good.

This is a reward function, right? So essentially what we see is if it matches the format exactly, we add the score by three.

If not, we just put zero. And remember, this match format essentially matches regular expression. We have to create it by hand for matching the format.

This number does not have to be plus three. It can be plus three hundred, whatever you like. It can be plus one. I don't know. It can be anything that you like.

But I just found plus three to work fine. Um, so you can do whatever you like, um, yeah, anything. And remember, the score is zero if you don't see it.

You can also do minus one. For example, if it's else, right, if it's not good, you can also just score minus, right? Minus three. You can minus three points from it, so up to you.

You can design your reward function as whatever you like.

But remember, if the model, if the model output is not exactly following your format, we should still at least reward it a little bit, right? Otherwise, the reward would just be zero, zero, zero, zero, zero.

So the trick is, if we see a keyword, we plus one, and if we don't... Oh, sorry. Plus zero point five if you see the keyword. And if you don't see the keyword, right, if you see the keyword this, if you see this in the output, you should at least plus zero point five.

But if you don't see it, then you minus one, and so this essentially allows you to partially reward the model.

More reward functions, now it gets more complicated. This alle- this large reward function essentially allows you to calculate the distance-based scoring. For example, remember we said, um, over here, um...

Where is it? Um, this one, right? What is two plus two? Four is, is correct, but three is also a better answer than D, right? If you ri- if you output D, it's definitely wrong, right?

If you output five, it's okay, but it's wrong. And so this function essentially allows you to take the good answer... So, sorry. This is the guess divided by your true answer, and it's like a ratio, and this ratio essentially allows you to reward...

If your number is close to the actual answer, you give it higher reward. And if your, if your answer is very, very, very far off, then you penalize it by minusing reward, and so this essentially allows you to do that.

If it's exactly correct, you also add five points.

And so this, this is probably the mo- this is probably the most important reward function. Um, but this is only for maths. Um, for other, like, code and stuff like that, you have to create more reward functions.

Now we test if our reward functions actually work. Um, and yes, it extracts the numbers. Um, this is just format reward, sorry. This is just extracting the solution, and you can see that it extracted zero point three four.

It extracted this number, it extracted this, and extracted this. Um, if your reward function is not working very well, you probably did something wrong in the re- regular expression, so please, like, edit that.

And then this is helper functions. Oh, this is another, another reward function. If you see, if you see the number one two three comma four five six, we want to remove the comma because you can't actually convert this into Python.

So you wanna remove the comma. Um, and then if it's equal to the true answer, you plus three point five reward, and if it's not, then you minus one point five reward.

More data set preparation functions. Not that important. Um, and here is the meat of the code for training, right? We call vLLM. Top P is one point zero. Um, you don't-- It's not that im- you can probably-- It's probably not a good...

One point zero just means you're sampling the entire space, um, so that's good. You can set this to, like, zero point eight or something else, up to you, but I generally set it to be one point zero to be, like, full sampling of the entire space.

Min P is zero point one. I suggest people to use this because otherwise the model might, like, go into, like... It might do inference of, like, random outputs, so p-- you know, use zero point one.

And temperature. I did suggest people to increase temperature to zero point-- one point two, right? Or you can do one. Um, try to increase your temperature as much as possible.

The more temperature you increase, the model becomes very, very, um, creative, right? It, like, creates random outputs. If you increase the temperature too much, like, you know, two Your model will be like gibberish.

So probably don't do too large numbers. So I normally suggest one point zero, one point one, or one point two, or somewhere around there. You should utilize min P together, right?

You should utilize min P together with high temperature numbers. Um, there is a paper about using temperature one point five and zero point one min P. You should utilize that.

There are some o- some other things that we utilize. Um,

num generations is very important. I set this to be four. This number is, is this thing. Um, where is it?

This number is this. How many, how many, like, uh, how many roll outs do you wanna do? How many inference steps do you wanna do for GRPO? Right. We chose four, and so four will just means what is two plus two, it will create four, four options, and so that is this number.

If you increase this number too much, you will use much more memory. Um, you should increase this as mu- as much as possible, if you can. Um,

and there's like gradient-- uh, there's like batch size. We set this to be one. Um, the trick is the batch size times the gradient accumulation is equivalent most of the time.

Um, GRPO is not. But essentially, what this does is if we do one, it just means we're doing one... What is one, what is two plus two? If you set batch size to be three, then we shove all of these three examples together into one.

Generally, you should set batch size to be much larger. Um, the problem is if you set batch size to be too big, you're going to use more memory. So the trick is instead you do gradient accumulation, you set this to be sixteen.

That's the trick. Um, gradient accumulation, it essentially allows you to do addition of gradients over time, and you can skip using too much memory.

And then there's like evaluation. If you wanna do evaluation, there's some functions for that. Um, and then we shall see the training. Um,

you will get a large table of numbers during the training process. This took two hours and fifty-four minutes on a free Colab. Um, look at the reward column, right?

The reward column. Minus seven point five, minus five point five, minus five point five. All very bad. Oh, and then suddenly, plus thirteen. Just by chance, suddenly, it's plus thirteen.

Remember, GRPO, the trick is if you see this plus thirteen, let's maximize this even more.

And then it's... Oh, but then it didn't really work, so it goes back to minus seven point five, minus five point five, minus seven point five, and so on.

And then plus eleven. You see another good reward. We want to maximize this as well. And so so on, so on, so on, right? That, that's the trick of GRPO.

By luck, by chance, literally by luck, you will have good answers. By good answers, you wanna maximize this. And if you keep looking down, right, if you s- keep scrolling down, in the end-- Okay, I need to make a plot.

But in general, your reward will increase over time, right? Look, these are all positive numbers now, right? These are all positive numbers. Your minus numbers are getting less and less and less and less.

Um, I think if I can-- I don't know if I can plot this, but okay, I pr- I'll plot this later. But essentially, if you plot this over time, the reward will actually increase over time.

There is also other numbers, like completion length. So essentially, remember, when you use a reasoning model, the reasoning trace can be extremely long. So this column just tracks how long the reasoning process is.

Um, it's... Over time, in general, the reasoning length should get longer and longer, um, but sometimes not always the case. Um, yeah, not always the case. There is also another column called KL divergence.

This essentially tells you how far the model is, the, you know, the final model from the original model. And the larger the number, it means it's getting very, very, very far away from the original model.

Um, in general, this number should get bigger over time, in general. Um, sometimes it doesn't move, but you should make this number go as much, much higher as possible.

Um, and then we also f- we made separate reward functions. Each of those reward functions also has their own reward. Um, the most important one is the last column, right?

Or the second last column. These are the two numbers. So in, in RL, there is a problem. Most RL fr-- uh, most RL training runs just follow the format, and it doesn't actually learn.

So the format th- columns are not important. Do not look at the format columns. These are useless. You need to look at the last two columns. And if you look at the last two columns, right, rewards check, check numbers, this essentially checks if the output is good or bad.

You see that it's minus two point five, minus two point five, not very good, and then suddenly, three point five, right? Three point five is good. We want to maximize this.

If you keep looking, over time, in general, if you take the-- if you take like a rolling average, in general, the model gets better and better and better. Obviously, we only trained this for two hours and fifty minutes.

You know, if you train it for twenty days, it might actually do very well. Um, but remember, this is a free Colab GPU. Um, so in general, remember, so like the goal is, the goal of GRPO is suddenly we see a good answer with a good reward.

We want to maximize that. And that's the whole point of GRPO. And it, it's, it's nothing fancy. It's just like by luck we see it, and we just wanna maximize it.

We can also see some output from the model. Right, at the very beginning, right, at the very beginning of the model... Let's see. Um, where is it? An example.

Compute the number of positive integers that, that divide at least to a blah, blah, blah, some question. And then it does some reasoning trace. Remember, we already fine-tuned it a little bit.

So it does something, right? It does something. But the answer, it just goes on and blah, blah, blah. It just blah, blah, blah. It just keeps going on, blabbering on.

Um, but then if you look at the actual answer, um, where is it?

If you keep-- Okay, there's a lot. Um, we print out every single-- We print out a lot. Okay, it just keeps... Okay, whatever. It keeps going on and on and on.

Um, this is the output of the GRPO algorithm. And you will see over time, if you inspect this, you will see that the model actually gets better and better and better.

Um, for-- This is an example, right? Let's say we ask the model, what is the square root of one hundred and one, right? It's n- we don't just say, what is the square root of one hundred?

That's just ten. We say, what is the square root of one hundred and one?

If you do not train the model, this is what you get. It will say answers, education, math and arithmetic, what is the square root of one hundred and one, Wiki user.

Ooh, Wiki user. [laughs] This is what it actually will say, right? That's actually what it will say. Does, do... Where do you think this data comes from? Does anyone know?

Can you take a guess where do you think this data comes from? Probably Wikipedia, right? So if you ask the question, what is the square root of one hundred and one, it doesn't do anything, right?

Remember, this is the base model. The base model is useless, right? You're not gonna get... It's not gonna answer the question.

But after we do GRPO, right, what is the square root of one hundred and one? We ask the question again. It says, "Okay, so I need to find the square root of one hundred and one.

Hmm, let me think. I remember that the square roots of numbers between the perfect numbers are ra- rational, blah, blah, blah, blah, blah, blah, blah." Right? It just says blah, blah, whatever, some whatever, and it says solution, ten point...

Right here. Solution, 10.049875. I think that's correct. I don't know if that's correct, but probably it's very close. And so the goal of... So the whole point is GRPO allows you...

The GRPO algorithm produced all of this, right? This reasoning trace. In the olden days, you actually have to have a human write all of this, and then you have to fine-tune the model.

GRPO, you skip. You don't need to make this anymore. It's automatic. Right? That's the trick of GRPO and reinforcement learning. All of this reasoning phase is automatic, totally produced from nothing, and i- in the end, it gets a solution.

Question.

Yes.

You had that seven thousand rows-

Yes. That's the trick. If you do the base model... The trick of doing this is if you just do the base model, go into this phase, you will still get this, but it'll be too long.

I see.

Otherwise, you'll wait there for, like, twenty days. And for demonstration purposes in the Colab, you have to do the supervised fine-tuning step. That's the trick.

Okay.

Yeah. Y- yes.

What is the advantage of doing this seven thousand examples fine-tuning versus using a instruct model out of the box?

You can use an instruct. We actually have notebooks for that. So if you go to GRPO in general, um, we have notebooks for using instruct model. Okay, the internet's very slow.

Um, GRPO, we actually have other notebooks. For example, if you use Llama three point two three billion, that is using instruct model. You don't need to use a base model, but we showed that you can use a base model.

Um, yeah, you can use... I suggest people to use instruct. Um, you sh- probably shouldn't use base. It's all about efficiency as well. Um, yes.

Can you quickly touch on the KL divergence, so, like, using GRPO with beta equals zero?

Sorry, what? What did...

You mentioned the KL divergence-

Yes, yes

... along with drifting away from the original.

Yes, correct.

Can you talk a little bit about, like, why using that or putting beta equal zero so you get, like, a non-KL divergence GRPO?

Yes. So the goal of KL divergence is you want the model not to stray too much away from the original model, right? KL divergence is... I shouldn't say distance, but KL divergence is like a distance between the, the current model, the current model that you're training, and the previous, very, very, very beginning of the model, right?

And so essentially, if the model is too far away, your KL divergence will be very large. Right? If you look at the plot... Uh, where is the table? Uh, I'll scroll up a bit.

Um, right. Which, which one is the KL divergence? Oh, here. This is the column, right? This column is a KL divergence, um, column. Over time, it should get larger and larger and larger over time, right?

The numbers should get larger and larger and larger because the model is straying away from the fine-tu- uh, the original model. If you set the beta to be zero, then you remove this term.

Yeah.

Maybe this might make the capabilities of the model more, maybe, because you're essentially f- not forcing the model to be as close as possible to the base model.

Active area of research. So yeah, some people might set it to zero, some people might not set it to zero. You know, I, I think it's zero point zero f- I think the default is zero point zero five or zero point zero three.

It's that.

Yeah.

Okay. Any other questions for... Yes.

You just took the, um, what you did, the s- the seven thousand rows or whatever. Um, where between the gibberish to, uh, the proper reasoning would it happen? What would the output look like for the same question if you did it right then before doing any GRPO?

So for fine-tuning... Actually, fine-tuning is actually very helpful already. Um, where is the loss? Where is the loss? The base model... So the base model already is very bad.

Um, if you do, uh... Here. There's a loss. There's a loss. This is using the fine-tuning step, the pre, the priming stage, right? You use, like, a data set to firstly prime the da- to prime the model.

The loss does decrease. Remember, if you see a loss of zero point six four, that's good. Um, if you see a loss higher than thirty, definitely something's wrong. Um, higher than three is very bad.

Um, you can see the loss definitely decreases over time.

Right.

So yes, doing the fine-tuning stage does teach the model a little bit to do reasoning, and it learns how to do some stuff. Also, a good, a very interesting, um, fact is we used DeepSeek-R1, some of the reasoning process, a- to do the fine-tuning step.

And interestingly, if you just call the model without doing GRPO, it kind of does reasoning already by doing seven thousand examples, right? It already says... Remember the question was, um, "What is two plus..."

Oh, wait. This is just a general question, right? It kind of learns how to do reasoning somewhat, but it's not perfect. And so the goal of GRPO is to forcibly make it perfect.

Okay, not perfect, but as much as possible. Um, so actually, the fine-tuning step already kind of learns a little bit.

Is the seven thousand the minimum examples that require the-

No, we... No. Actually, you don't need to use seven thousand. I think I only used...

I think I used one hundred and eighteen.

Oh.

It's... So it's, uh, uh, two training epochs. I think it's... Yeah, it's one hundred and eighteen. I only use one hundred and eighteen rows. You don't need to... You can use ten rows.

You can use... Yeah, use as less. You must use more than three rows, though, um, because when you do LoRA, the gradients become, are zero, so you must use more than three.

But l- but anything more than three is fine

Yeah, so even if you use 118, it does fine. Um, yeah.

Yes.

Are there any things to consider if you need to scale this training for-- to start with the bigger base models, like, uh, apart from, uh, taking a bigger GPU, are there any tricks or strategies?

Do you mean, like, a small model versus a big model? What's the difference?

Um, can you, uh, tell any tri-tricks if you want to do this training on a bigger base model?

Oh yeah, go ahead. You can, you can take the notebook. You will need a better GPU though. Take the notebook, edit,

edit this here. Not four, you can do 14 billion. Wait, I think there's a 14 billion, I think. Or is it 12 billion? I can't remember. You can do whatever you like.

You could even do, you know, Llama 3.3, 70 billion. I don't know, up to you. Um, do whatever you like, and... But the goal is, for a Colab demonstration, because it's a small GPU, I use, like, a small model.

Um, we actually have notebooks for free Colab which fits 14 billion. Um, so Phi-4 14 billion actually fits in a free Colab. Um, so you can do big models in a free Colab.

Um, Kaggle. Again, I said Kaggle, use Kaggle, free GPUs. Um, there are like no-- So essentially, this whole page has, like, notebooks. Uh, where's Kaggle? Uh, Kaggle. Kaggle has, like, notebooks for GPU as well.

Um, so you can do whatever you like for large models. Um, yeah.

Do you have examples on sampling and training, separating them on different machines?

Oh, do you mean, like, for vLLM rollouts, like... Oh, so the trick, what we do is you-- we co-locate. So you use the same machine for inference and fine-tuning, and the trick is you can reduce memory usage because you're sharing the vLLM weights.

So some other trainers, like VERL and TRL, you do have to put the inference on another server, and then your training is, like, a separate server, and they have to do communication.

We don't-- There is no communication for us. There is none. So we do s-- It's very close to asynchronous training. Nearly no delay in training.

But yes, we d-we don't support it yet, but we do plan to support, like, you know, larger training runs. Um, yeah.

Y-yes.

I have a quick question. [clears throat] What's the TPU, uh, roadmap look like for Unsloth?

Yes, people have asked that. No roadmap. [laughs] I don't know if we're gonna support it. It's a bit more complicated. You could use, I think, XLA. Like, they, they do have PyTorch converted down to TPU, so maybe it might work.

I don't know if it works. I've never tried it. Um,

maybe later. Um, yeah. Maybe later.

That's a valid answer.

Okay. Yeah. Yes.

Just a follow-up. Uh, what's the reason why we chose the base model instead of the instruct model?

Oh, you don't have to. You, you... The, the whole point, the, like, this thing, um,

uh, here, right? We chose the base model because you can show that you can do a base model going to the green dot. But then, unfortunately, in the Colab, we do have to do some supervised fine-tuning.

Otherwise, you'll wait there forever. The reward, again, will be zero, zero, zero, zero, zero. We just want to... Remember, all of AI is about efficiency and speed, so we just wanna showcase, okay, you do need to do the light blue step.

You still need to do the supervised fine-tuning step. Um, yeah, yeah.

So we need to construct model with APPs to do that?

Oh, no, you don't need to. You can take an instruct-- So the notebooks over here, so for example, if you go to the, uh,

the Llama 3.2 three billion notebook, we don't do any fine-tuning step at all. You skip directly because it's an instruct model already. It already learns how to do chat.

It already learns how to answer some questions. You can skip directly to GRPO, um, if it loads. Um, but yes, the notebooks... Oh, okay. You'll have to wait for it to load.

Um, whatever. [laughs] The internet's very bad. Um, it is loading. Um, yes. Any other questions?

Oh, over here.

Yes.

Over in the REINFORCE algorithm, you had the log probability of a state-- or no, of an action given a state.

Mm-hmm.

Is that happening inside the sampling model, or is that in the notebook happening?

Oh, the, the, the algorithm itself for GRPO? Oh, it's like behind the scenes. Like-

Yeah. Is that happening over in the G-GRPO trainer?

Yes, it's inside the training itself. Like, somewhere in the code, somewhere it does that.

It's figuring out, like, what is the probability of a given token versus all the tokens?

Oh, y- the calculation is inside the trainer. So, like, somewhere, you know, on the GPU, you're doing this calculation. But you do get the probabilities. Remember the language model, you get the probabilities already.

Yeah.

You just get the reward function, and you just wanna maximize it. So...

So do you, do you take the logits that come out of the large language model and turn them into pseudo probabilities, and then just assume that's the distribution?

I think so. Yes, that's correct.

Okay.

I think that's exponential of... Yes, I think that's correct. There is... If you go to, like, the code, there is, like, some derivation for it. But yes, you're correct.

Any... Oh, okay. The notebook loaded. But yes, there is another notebook which does the instruct here, right? There is the instruct model. And there is no fine-tuning step at all.

It just does the reward function, um, and stuff like that. Um, Will's notebook, for example, is also very good. So, like, if anyone wants to check other notebooks out, Will's notebook, um, also utilizes, I think, also the instruct model and then does GRPO.

Um, okay. The GR-- Okay, kind of time is running out, but okay, technically, the GRPO for-- portion is done. [laughs]

Oh, there's actually more portions. Um, I will have to breeze through them. There's only ten minutes left. Um, whoops. Um, any other... I, I will take questions at the very end.

Anyways, I'm gonna stay here anyways afterwards. Um,

quantization. We'll now sw-shift over to quantization. Um, so I don't know if you guys know about the DeepSeek-R1 1.58-bit quants that we did, um, but you can essentially download these models.

DeepSeek-R1 is seven hundred and thirty, I think seven hundred and thirty GB. You can quantize them down to one hundred and forty GB, um, without that much loss in accuracy.

Okay. There's obviously loss in accuracy, but the trick is you can quantize them down to be very small, and miraculously, they work.

Um, Llama 4 Scout, for example, right? You can't really see the accuracy plot. Um, the smallest number is eighty percent accuracy on MMLU five-shot, eighty percent. And the highest accuracy is eighty-one-point-something, right?

So it's actually only one percent difference. And the one on the left is a one-bit quant. It's very small. It's tiny in comparison to the full precision like float eight.

And so essentially you can make the model eight times smaller and you only decrease accuracy by one percent. Um, so that's very interesting. And so essentially we showcase that you can actually quantize the MoE layers, the mixture of experts layers, very heavily, but you must leave the attention layers, the shared experts, and other layers in higher precision,

and that's called-- that's what we call the dynamic quantization methodology.

Um, if you see there was like a benchmark of Llama 4 Scout, for example. Um, if you use a two-bit, a two-bit quant, it actually gets high accuracy than other, other full precision, um, providers, which is very interesting, right?

So like, for example, the two-bit quant gets, um, seventy-three percent accuracy, and then other inference providers get sixty-five percent accuracy, sixty-seven, right? There is a very large difference. And so the...

Okay, there is like some bugs in the models and there's maybe they quantize it incorrectly, but the goal is to show that if you quantize the model down to be very small bits, it still works.

We showcase this with an example. For example, if you take a vision model like, um, Qwen seven bill-- uh, Qwen two billion. Um, if you naively quantize all the layers to be four-bit, right?

You ask the model, "What does this image show?" It will say, "The image depicts a vibrant and colorful scene of a coastal sc- area," which is totally wrong, right?

The answer should be, "The image shows a train traveling on tracks," or something like that, right? If you quantize everything to four-bit, it's one point three six GB, but it's definitely bad.

So the trick is you must quantize some layers to be-- You must leave some layers to higher precision, and you only need to increase it by five hundred MB also to one point eight GB, and it works.

The image shows a train traveling on tracks. It suddenly works.

But the question is which layers do you not quantize? That's the question, right? Which layers? You could do an exhaustive search, right? You can check, oh, let's not quantize layer zero.

Let's check layer one, layer two. You're gonna check every single one, but it'll take forever. So definitely don't do that, right? You have seventy choose one or something. Seventy choose one plus seventy choose two.

Horrible, right? And remember, all of AI is about efficiency, so don't do that. The trick is you can check the activation quantization error and the weight quantization error, and you will see these large outliers.

For example, for Qwen, if you quantize the first few layers, it's extremely bad. So you must leave the first few layers not quantized. And also, this gigantic jump for the weight quantization error, this means you probably shouldn't quantize that layer as well.

There are some other plots that we show. For example, for Llama 3.2, it's interesting. All of these graphs are very different from each model. Um, you will notice Llama 3.2 has these weird, you know, continuous spikes.

Um, it's because they use attention, and then they put the attention back to the vision module. I think every single three-- I think it was every single three layers.

So every single three layers, it has these big jumps. Um, this means you should not quantize those layers.

Pixtral, for example, is also a different graph again. Um, Pixtral seems like you can't quantize many layers, unfortunately. Um, and so, like, the whole vision module must not be quantized.

There is a very popu-- uh, there is a very important paper talking about like, you know, why-- which layers you should quantize and which layers you should not quantize.

It's called the Super Weights paper. Um, you, you guys definitely should read that. Essentially, it says that in all language models, the first few layers of the down projection, there is a very, very, very important number in one of the numbers, one of the numbers in the models.

Very, very, very important. And you should never quantize it, ever.

But the trick is... The, the interesting finding is it's not actually a very large number. So there is a trend in the-- There is a trend in, um, language model space where people think that you should not quantize outliers.

The problem is, you know, these models have these big outliers. Like, suddenly in the model, there's like this big number, like three thousand, and if you quantize it, it essentially ruins the model.

But actually, this paper shows that it's not actually the outliers that are the problem. These numbers could be very small, and if you look at the plots, if you remove-- if you select these numbers, and if you make them zero,

the accuracy dr-- uh, the accuracy decreases dramatically. Um, and so, like, you should see, like, for example, if you remove one of the numbers, um, you know, the activation value totally decreases.

Very bad. They have very large activation values, and then if you remove them, it's very, very, very, very bad. Um, there is another trick that you can do. If you have a model that has seven billion parameters, make every single number go to zero.

The first parameter, make it go to zero. Check accuracy. The second number, make it go to zero. Check accuracy. The third number, go to zero. Check accuracy. You can do this seven billion times, and then you can see which number is the most important.

You could do that as well. Um, but remember, AI is about efficiency. Very not a good idea. Um,

more later research, for example, the new Blackwell chips, um, instead of doing a quantization to like one bit, two bit, three bit, four bit, NVIDIA chips also have this new architecture, this new format called FP4 or MXFP4.

Um, and essentially this is float four. Um, and float four is ver- it's, it's most likely going to be very used a lot in the future. Um, and then there's these like new formats for quantization as well, which, which essentially allows you to train models in very low precision.

I also made this plot, um, going from float thirty-two... So the question people always ask is, "Why is GPUs getting faster and faster and faster?" My take is actually this year is probably the last year you're gonna get a GPUs that is actually faster.

There is no more faster GPUs. Why? Because the majority of GPUs getting faster is because of numerical precision. From float thirty-two to float sixteen, you get five times faster.

Why? Why is it five times faster? Because, because, um, the calculation of faster is when you use transistors, it's the exponent plus the mantissa squared, right? In float 32, you have to use 23 numbers for the mantissa, and 23 squared is very large.

Float 16, you reduce the mantissa to 10, and that is why you get five times speed up from float 32 to float 16. It's because the number itself is getting smaller.

The representation inside the models for each of the weights is getting smaller. And then we moved from float 16 to be bfloat16. It is again maybe around two times faster than float 16.

We then have float 8. Um, float 8 is even more faster, um, you know, uses even less space. Um, but then there is a problem. Float 4, we get to float 4, and it's around two times faster than float 8, around.

Um, the problem is, what's next? Do we go to float 2, float 3, float 1? You know, I, I... There's-- You can't push anymore in terms of numerical precision.

There is not much more to go in terms of that space. And so, like, you can only get maybe 180 times faster than float 32, maybe 200 times faster.

But essentially, my take is float 4 might be the final float that's getting faster, you know, the final, um, precision, numerical precision. And in the future, GPUs are not gonna get faster.

Um, so maybe if, you know, people wanna buy Blackwell GPUs, that's probably, you should probably buy them. It's most likely not gonna get faster anymore. Um, that's kinda my take.

And also, for... Okay, I was gonna talk about kernels and stuff, but I don't think so I have enough time. You must use torch.compile. You know, every single function that you see, wrap it in torch.compile, try it out.

You know, like I, I always tell the PyTorch team, "Please make it by default." You know, definitely use torch.compile. Um, why? Because it makes your training faster sometimes, only sometimes, um, [chuckles] not all the time.

It reduces memory usage most of the time. If you see bugs, they'll probably fix it. Um,

but remember, torch.compile is not as easy as you think. You don't just do torch.compile the model. There is actually many options you can tune, right? I just listed a few options.

Um, this is, this is literally just a few. There is, like, ten more pages of options. I'm being serious, ten more pages you can tune. Imagine if you can, like, use torch com- torch.compile and tune every single one.

And that's why I highly suggest people to use torch.compile more effectively. Um, it's probably the biggest thing that can change your entire, you know, training run, um, make it more m- more memory efficient, make it faster.

So definitely look through this. Um, okay, so in general, yes, thank you. Definitely star us on GitHub. Um, join our Discord if you wanna have any questions on RL and stuff.

Um, we have a website as well. Um, and finally, we have stickers. Um, yes, there are some limited time stickers as well that we have somewhere, I think over there.

Um, and remember, if you have any questions, I'm still gonna stay around and ask. Um, yeah. Thanks a lot. [applause] [upbeat music]