AI Engineer Code 2025
RL Environments at Scale
Loading slides…
Read the talk
RL Environments at Scale
A reusable environment connects evaluation, synthetic data, and model training. Will Brown’s WikiSearch example shows how that abstraction can make research part of everyday AI engineering.
From a talk by Will Brown
Before you start: Basic familiarity with language models, evaluation, and Python will help; no experience implementing reinforcement learning is assumed.
How do you scale the ability to do research?
Running thousands of parallel rollouts and sandboxes on hundreds of GPUs is one kind of scaling problem. Another is increasing data, compute, parameters, or inference time to improve model performance. But neither fully explains where better algorithms, useful tricks, and new research ideas come from. That raises a different question: how do you scale the community capable of producing those improvements?
Ideas develop through a cycle of applications and experiments. Someone builds an application, encounters a problem, and develops an idea; the application then becomes a testbed for trying it. Sharing the result gives the next person a starting point. Reusable tools and accumulated knowledge reduce the work each researcher must repeat, making more ambitious experiments accessible.
This is the talent bottleneck that Will Brown wants to address. Competing to hire existing researchers is one response. Expanding the pool of people who can do research is another—and it depends on making the practice easier to enter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An open stack for research as a practice
Prime Intellect combines a research lab, a compute provider, a platform business, and an open-source ecosystem. These roles support a common goal: give engineers a research toolkit they can use while improving applications, systems, and products, without first joining a large lab, acquiring a massive cluster, or completing a PhD. Research becomes part of the engineering workflow rather than a separate institutional activity.
The useful analogy with open-source software is broader than releasing model checkpoints. Linux, Node, and Apache illustrate how ecosystems accumulate abstractions, tools, and working practices. Brown applies that analogy to research as a practice: better tooling makes iteration faster, reusable foundations lower barriers, and those improvements allow people to build increasingly complex systems.
Prime Intellect calls the supporting infrastructure the open superintelligence stack. Its layers include compute, orchestration, training and evaluation libraries, and services for code execution, evaluation, inference, and fine-tuning. Together, these provide the machinery for running research, not just accessing a finished model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train the model inside the product
Giving product builders the ability to train models creates another way to improve the user experience. Customization might affect one component, one model, or a larger part of the product. It need not be the first step, but having the option expands what engineers can change when improvements around an existing API are insufficient.
Cursor’s Composer and OpenAI’s Codex are Brown’s examples of products whose experience depends directly on a model trained for that product. In his framing, the product increasingly becomes the model: using the model and using the product become closely connected experiences. The mechanism is to take a harness representing the product and train the model inside it. That harness becomes part of an RL environment.
An environment is a harness together with tasks and rewards. The abstraction applies beyond reinforcement learning:
| Use | Role of the environment |
|---|---|
| Evaluation | Run tasks through the harness and measure the results. |
| Synthetic data | Generate examples that can feed supervised fine-tuning, or SFT, and distillation. |
| Reinforcement learning | Train the model using rewards from its attempts. |
| Deployed agents | Handle incoming user tasks and monitor outcomes. |
The tasks may come from an offline dataset or a live stream of product requests. Keeping the harness, tasks, and rewards together gives these activities a common experimental foundation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small project that can grow into research
Brown calls environments the “web apps of AI research.” They can begin as simple, self-contained projects and grow to represent a full product. As complexity increases, the builder encounters new questions about systems scaling, hyperparameters, and algorithms. Those questions arrive through a concrete task, without requiring the builder to implement an entire training infrastructure first.
The distinction between an agent harness and an environment is the requirement to define tasks and rewards. A harness alone lets you try an agent and decide whether its behavior seems promising. An environment also requires you to specify what it should do and how its attempts will be assessed. That supports experiments comparing models and hyperparameters, and provides a path toward RL, distillation, or fine-tuning when those become useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share tasks, then build on common components
The Environments Hub provides a place to create, discover, and share RL environments and evaluations. Brown reports hundreds of builders and environments. Contributions include reimplementations of papers, benchmarks adapted with new data or examples for RL, games, and original tasks. Packaging the task creates a starting point for experimentation even when the builder does not yet know how to improve the model.
This changes the data problem. Brown attributes some of the difficulty of adopting SFT to the work of collecting examples of the desired solutions. An environment can instead define situations the model will encounter and a way to measure its answers. If you can assess generated answers, you need not supply every solution upfront. The environment can become an engine for producing data as it runs. Brown connects this approach to Verifiers, the library he had released nine months before the talk.
Verifiers supplies components that can be composed into evaluations, question-answering tasks, games, math problems, tool use, sandbox execution, and agent frameworks, including CLI coding agents. The resulting environments are designed to support RL training. Covering that range requires extensibility: the toolkit cannot anticipate every application, and different applications introduce different patterns and special cases.
The hierarchy separates application-specific behavior from shared foundations. Brown’s coding example runs through Kleinbench → Harbor → CLI agent → multi-turn environment → environment. Other branches include TextArena and Wordle, search through MCP, and a Python REPL inside a sandbox. Each branch helps identify what belongs in the base environment and what should be added by a more specialized layer.
Editor’s note: the benchmark name is likely cline-bench, whose announcement connects it to Harbor and Prime Intellect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
WikiSearch: from a Python package to training
WikiSearch makes the abstraction concrete. An agent receives tools for searching Wikipedia pages and finding answers. Its Hub entry is a Python project with dependencies and versions, alongside facilities for uploading evaluation results. The Hub therefore serves as both code management and a package registry for environments.
The environment’s tools are asynchronous Python functions. A dataset supplies tasks, and a rubric manages the components of the reward. The rubric can also record metrics that contribute zero reward, preserving visibility into behavior without making every observation an optimization target.
For a search rollout, the following Python function expresses that separation. It takes an answer score supplied by the task’s evaluator and records the number of tool calls without rewarding that count:
python
def score_search_rollout(
answer_score: float,
tool_calls: int,
) -> tuple[float, dict[str, float]]:
metrics = {
"answer_score": answer_score,
"tool_calls": float(tool_calls),
}
weights = {
"answer_score": 1.0,
"tool_calls": 0.0,
}
reward = sum(
weights[name] * value
for name, value in metrics.items()
)
return reward, metrics
Here, changing tool_calls changes an observable metric but leaves the reward unchanged. The environment still has to define how it obtains answer_score.
Training adds a configuration for Prime RL, Prime Intellect’s stack for large-scale asynchronous RL. Its configuration exposes choices that bring the builder closer to the training algorithm while keeping the setup relatively self-contained. Brown describes the defaults as intended to be sensible starting points for many users. The environment definition and training configuration remain separate pieces of the experiment.
In the workflow Brown demonstrates, the final step is a command that identifies the environment. If it is available on the Hub, the trainer automatically installs it and starts the run. The progression is therefore straightforward:
- Package the tools, task dataset, and reward rubric as an environment.
- Select the training configuration.
- Launch the run with the environment identified in the command.
The automatic installation is part of the workflow described in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reward curve starts another round of experimentation
A reward curve that rises quickly is a successful example, not a guaranteed outcome. The work continues through iteration on the environment, rewards, data, tasks, and training parameters. Looking at the data and reconsidering what the reward measures are part of making the task work in practice. Brown emphasizes that this process can improve both small and larger models.
For WikiSearch, Brown reports the following change:
| Model and stage | Reported WikiSearch score |
|---|---|
| Qwen3-4B, before training | About 55% |
| Qwen3-4B, after training | 89% |
Brown describes the trained result as on par with GPT-4.1 and GPT-5 Mini in this WikiSearch setting. These are reported task-specific results; the talk does not specify the evaluation split, judge, or matched inference budgets for the comparison.
Customization can serve different product goals: a faster model, a cheaper model, or better performance on a task where the strongest available models remain insufficient. A small model improved for one application is especially useful when speed or cost matters. Building an environment preserves the option to pursue those improvements without requiring customization from the outset.
If a product already needs evaluations, expressing them as environments also supports prompt tuning, model selection, and investigating how the system behaves with many users in parallel. The design work forces a useful set of decisions: what is the agent, what is the product, what does the harness permit, and what outcome should be optimized?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stress-testing the stack with INTELLECT-3
The larger validation effort was INTELLECT-3, still forthcoming at the time of the talk. Brown describes INTELLECT-3 as a 100B+ parameter model trained on about 500 GPUs, with an end-to-end post-training process covering SFT and RL. This work exercised the full Prime RL stack at a much larger scale; Prime RL also supports SFT for users who need it.
The training effort also provides a way to test techniques from research papers, determine which work in practice, and incorporate useful implementations into the library. Keeping Prime RL and Verifiers open lets downstream users consume those practices without reproducing all the implementation work themselves. The intended result is a shorter path from an idea in the literature to an experiment that improves a model or product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Community feedback and a managed research workflow
A broader community exposes problems that a single research team may miss. Prime Intellect collects feedback about what builders want, what works, and what remains painful. Its programs include sponsoring small tasks, a research residency involving graduate students around the world, and manually reviewing a subset of Hub environments in the repository Brown calls Prime Environments. Brown reports hundreds of submissions to this review process. Those implementations reveal rough edges and force improvements in the tools.
The lessons also feed into Lab, the platform Brown describes as forthcoming. Its proposed workflow lets users browse environments and run evaluations, inference, and fine-tuning through one interface. The operational burden includes getting Torch, Flash Attention, and vLLM versions to work together. Prime Intellect offers to manage that infrastructure: users can inspect the open code without having to operate every component themselves.
The environment remains the entry point. Evaluation begins with an environment; synthetic data for SFT begins with an environment; RL begins with an environment. A managed platform handles execution around that shared definition, allowing the builder to concentrate on the task, the agent’s available behavior, and the measurement of success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different deployment goals, deeper understanding
Environment building can support several destinations as demand for model customization grows:
- Fine-tuning through model labs. Brown anticipates labs offering services that let customers adapt their models to specific needs.
- Specialized on-premises models. A team may care most about the smallest model it can run locally at the lowest latency, optimized for one task.
- Research for its own sake. Experiments can advance shared understanding without an immediate product objective.
These goals differ, but each benefits from being able to define a setting, try changes, and examine what happens.
That ability also changes the relationship engineers have with models. Treating a model as a black box limits how much its behavior can teach you. Inspecting it, modifying it, and sometimes breaking it provides a more direct understanding of how it works and how its capabilities developed. Shared experimentation makes it easier to discuss what is being built—and to prepare for where the models may go next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Library for building LLM training environments and evaluations, integrated with Prime Intellect's training tools.
Training framework with SFT, asynchronous RL, evaluation and examples ranging from small tasks to large models.
Community-contributed RL environments and evaluations with tooling for creating and publishing environment packages.
Further reading
- Introducing cline-benchArticle
An initiative to turn real open-source engineering tasks into reproducible coding evaluations and RL environments.
Updates since the talk
Current walkthrough for training a Qwen3 model to answer trivia using Wikipedia search and reading tools.
Release announcement describing INTELLECT-3's model, post-training recipe, environments and GPU infrastructure.
Current Lab guide covering baseline evaluation, training configuration, monitoring and evaluation of trained adapters.
Full transcript
Loading transcript…