# RL Environments at Scale

Will Brown, Prime Intellect

AI Engineer Code 2025

[Talk page](https://ai.engineer/talks/_IzZWeuTx7I-rl-environments-at-scale) · [Video](https://www.youtube.com/watch?v=_IzZWeuTx7I)

## RL Environments at Scale

A reusable environment connects evaluation, synthetic data, and model training. Will Brown’s WikiSearch example shows how that abstraction can make research part of everyday AI engineering.

Basic familiarity with language models, evaluation, and Python will help; no experience implementing reinforcement learning is assumed.

### How do you scale the ability to do research?

[0:25](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=25s)

Running thousands of parallel rollouts and sandboxes on hundreds of GPUs is one kind of scaling problem. Another is increasing data, compute, parameters, or inference time to improve model performance. But neither fully explains where better algorithms, useful tricks, and new research ideas come from. That raises a different question: how do you scale the community capable of producing those improvements?

Ideas develop through a cycle of applications and experiments. Someone builds an application, encounters a problem, and develops an idea; the application then becomes a testbed for trying it. Sharing the result gives the next person a starting point. Reusable tools and accumulated knowledge reduce the work each researcher must repeat, making more ambitious experiments accessible.

This is the talent bottleneck that [Will Brown](https://willcb.com/) wants to address. Competing to hire existing researchers is one response. **Expanding the pool of people who can do research** is another—and it depends on making the practice easier to enter.

![The Axes of Scaling slide has Laws and Practices columns and asks, “How can we scale talent?”](https://ai.engineer/api/talk-essay-media/5bd56163b03a155822a684d0d39bd984ea0aa774a02a6a08ef896ac94152ce47)

Axes of scaling: data, compute and parameters alongside community, applications and accessibility.

### An open stack for research as a practice

[2:09](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=129s)

Prime Intellect combines a research lab, a compute provider, a platform business, and an open-source ecosystem. These roles support a common goal: give engineers a research toolkit they can use while improving applications, systems, and products, without first joining a large lab, acquiring a massive cluster, or completing a PhD. Research becomes part of the engineering workflow rather than a separate institutional activity.

The useful analogy with open-source software is broader than releasing model checkpoints. Linux, Node, and Apache illustrate how ecosystems accumulate abstractions, tools, and working practices. Brown applies that analogy to **research as a practice**: better tooling makes iteration faster, reusable foundations lower barriers, and those improvements allow people to build increasingly complex systems.

Prime Intellect calls the supporting infrastructure the **open superintelligence stack**. Its layers include compute, orchestration, training and evaluation libraries, and services for code execution, evaluation, inference, and fine-tuning. Together, these provide the machinery for running research, not just accessing a finished model.

![A layered diagram places aggregated compute and the Prime Compute Platform below RL libraries and Lab, with arrows connecting them to training and deployment.](https://ai.engineer/api/talk-essay-media/3847f996e8883785898f1c0d2ea9ce22f37bca6c54dbb9119abd69a393723cee)

The Open Superintelligence Stack connects compute, RL libraries, Lab and model training.

### Train the model inside the product

[4:23](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=263s)

Giving product builders the ability to train models creates another way to improve the user experience. Customization might affect one component, one model, or a larger part of the product. It need not be the first step, but having the option expands what engineers can change when improvements around an existing API are insufficient.

Cursor’s **Composer** and OpenAI’s **Codex** are Brown’s examples of products whose experience depends directly on a model trained for that product. In his framing, the product increasingly becomes the model: using the model and using the product become closely connected experiences. The mechanism is to take a **harness** representing the product and train the model inside it. That harness becomes part of an RL environment.

**An environment is a harness together with tasks and rewards.** The abstraction applies beyond reinforcement learning:

| Use | Role of the environment |
|---|---|
| Evaluation | Run tasks through the harness and measure the results. |
| Synthetic data | Generate examples that can feed supervised fine-tuning, or SFT, and distillation. |
| Reinforcement learning | Train the model using rewards from its attempts. |
| Deployed agents | Handle incoming user tasks and monitor outcomes. |

The tasks may come from an offline dataset or a live stream of product requests. Keeping the harness, tasks, and rewards together gives these activities a common experimental foundation.

### A small project that can grow into research

[6:36](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=396s)

Brown calls environments the “web apps of AI research.” They can begin as simple, self-contained projects and grow to represent a full product. As complexity increases, the builder encounters new questions about systems scaling, hyperparameters, and algorithms. Those questions arrive through a concrete task, without requiring the builder to implement an entire training infrastructure first.

The distinction between an agent harness and an environment is the requirement to define **tasks and rewards**. A harness alone lets you try an agent and decide whether its behavior seems promising. An environment also requires you to specify what it should do and how its attempts will be assessed. That supports experiments comparing models and hyperparameters, and provides a path toward RL, distillation, or fine-tuning when those become useful.

### Share tasks, then build on common components

[8:04](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=484s)

The **Environments Hub** provides a place to create, discover, and share RL environments and evaluations. Brown reports hundreds of builders and environments. Contributions include reimplementations of papers, benchmarks adapted with new data or examples for RL, games, and original tasks. Packaging the task creates a starting point for experimentation even when the builder does not yet know how to improve the model.

This changes the data problem. Brown attributes some of the difficulty of adopting SFT to the work of collecting examples of the desired solutions. An environment can instead define situations the model will encounter and a way to measure its answers. **If you can assess generated answers, you need not supply every solution upfront.** The environment can become an engine for producing data as it runs. Brown connects this approach to [Verifiers](https://github.com/PrimeIntellect-ai/verifiers), the library he had released nine months before the talk.

Verifiers supplies components that can be composed into evaluations, question-answering tasks, games, math problems, tool use, sandbox execution, and agent frameworks, including CLI coding agents. The resulting environments are designed to support RL training. Covering that range requires extensibility: the toolkit cannot anticipate every application, and different applications introduce different patterns and special cases.

The hierarchy separates application-specific behavior from shared foundations. Brown’s coding example runs through **Kleinbench → Harbor → CLI agent → multi-turn environment → environment**. Other branches include TextArena and Wordle, search through MCP, and a Python REPL inside a sandbox. Each branch helps identify what belongs in the base environment and what should be added by a more specialized layer.

*Editor’s note: the benchmark name is likely [cline-bench](https://cline.ghost.io/cline-bench-initiative/), whose announcement connects it to Harbor and Prime Intellect.*

### WikiSearch: from a Python package to training

[11:14](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=674s)

**WikiSearch** makes the abstraction concrete. An agent receives tools for searching Wikipedia pages and finding answers. Its Hub entry is a Python project with dependencies and versions, alongside facilities for uploading evaluation results. The Hub therefore serves as both code management and a package registry for environments.

The environment’s tools are asynchronous Python functions. A dataset supplies tasks, and a **rubric** manages the components of the reward. The rubric can also record metrics that contribute zero reward, preserving visibility into behavior without making every observation an optimization target.

For a search rollout, the following Python function expresses that separation. It takes an answer score supplied by the task’s evaluator and records the number of tool calls without rewarding that count:

```python
def score_search_rollout(
    answer_score: float,
    tool_calls: int,
) -> tuple[float, dict[str, float]]:
    metrics = {
        "answer_score": answer_score,
        "tool_calls": float(tool_calls),
    }
    weights = {
        "answer_score": 1.0,
        "tool_calls": 0.0,
    }
    reward = sum(
        weights[name] * value
        for name, value in metrics.items()
    )
    return reward, metrics
```

Here, changing `tool_calls` changes an observable metric but leaves the reward unchanged. The environment still has to define how it obtains `answer_score`.

Training adds a configuration for [Prime RL](https://github.com/PrimeIntellect-ai/prime-rl), Prime Intellect’s stack for large-scale asynchronous RL. Its configuration exposes choices that bring the builder closer to the training algorithm while keeping the setup relatively self-contained. Brown describes the defaults as intended to be sensible starting points for many users. The environment definition and training configuration remain separate pieces of the experiment.

![The Example: wiki-search slide shows Python environment code on the left and a training configuration on the right.](https://ai.engineer/api/talk-essay-media/bbc5e1f7d19c58ffca38cdb7b09770f6af55fb28531243409e04fe4a80c3d081)

Wiki-search environment code alongside its training configuration.

In the workflow Brown demonstrates, the final step is a command that identifies the environment. If it is available on the Hub, the trainer automatically installs it and starts the run. The progression is therefore straightforward:

1. Package the tools, task dataset, and reward rubric as an environment.
2. Select the training configuration.
3. Launch the run with the environment identified in the command.

The automatic installation is part of the workflow described in the talk.

### The reward curve starts another round of experimentation

[12:43](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=763s)

A reward curve that rises quickly is a successful example, not a guaranteed outcome. The work continues through iteration on the environment, rewards, data, tasks, and training parameters. Looking at the data and reconsidering what the reward measures are part of making the task work in practice. Brown emphasizes that this process can improve both small and larger models.

For WikiSearch, Brown reports the following change:

| Model and stage | Reported WikiSearch score |
|---|---:|
| **Qwen3-4B**, before training | About 55% |
| Qwen3-4B, after training | 89% |

Brown describes the trained result as on par with **GPT-4.1** and **GPT-5 Mini** in this WikiSearch setting. These are reported task-specific results; the talk does not specify the evaluation split, judge, or matched inference budgets for the comparison.

Customization can serve different product goals: a faster model, a cheaper model, or better performance on a task where the strongest available models remain insufficient. A small model improved for one application is especially useful when speed or cost matters. Building an environment preserves the option to pursue those improvements without requiring customization from the outset.

If a product already needs evaluations, expressing them as environments also supports prompt tuning, model selection, and investigating how the system behaves with many users in parallel. The design work forces a useful set of decisions: what is the agent, what is the product, what does the harness permit, and what outcome should be optimized?

### Stress-testing the stack with INTELLECT-3

[14:14](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=854s)

The larger validation effort was **[INTELLECT-3](https://www.primeintellect.ai/blog/intellect-3)**, still forthcoming at the time of the talk. Brown describes INTELLECT-3 as a 100B+ parameter model trained on about 500 GPUs, with an end-to-end post-training process covering SFT and RL. This work exercised the full Prime RL stack at a much larger scale; Prime RL also supports SFT for users who need it.

The training effort also provides a way to test techniques from research papers, determine which work in practice, and incorporate useful implementations into the library. Keeping Prime RL and Verifiers open lets downstream users consume those practices without reproducing all the implementation work themselves. The intended result is a shorter path from an idea in the literature to an experiment that improves a model or product.

### Community feedback and a managed research workflow

[15:12](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=912s)

A broader community exposes problems that a single research team may miss. Prime Intellect collects feedback about what builders want, what works, and what remains painful. Its programs include sponsoring small tasks, a research residency involving graduate students around the world, and manually reviewing a subset of Hub environments in the repository Brown calls [Prime Environments](https://github.com/PrimeIntellect-ai/community-environments). Brown reports hundreds of submissions to this review process. Those implementations reveal rough edges and force improvements in the tools.

The lessons also feed into **Lab**, the platform Brown describes as forthcoming. Its proposed workflow lets users browse environments and run evaluations, inference, and fine-tuning through one interface. The operational burden includes getting Torch, Flash Attention, and vLLM versions to work together. Prime Intellect offers to manage that infrastructure: users can inspect the open code without having to operate every component themselves.

![The Lab slide highlights Environments above Fine-Tuning beside a dashboard with metric cards, a blue bar chart and three line charts.](https://ai.engineer/api/talk-essay-media/39eb698a67189600f5628e4d8afd5ae1d3c06c169efcd6486bffe73c9b6064a2)

Lab navigation and a training-run metrics dashboard.

The environment remains the entry point. Evaluation begins with an environment; synthetic data for SFT begins with an environment; RL begins with an environment. A managed platform handles execution around that shared definition, allowing the builder to concentrate on the task, the agent’s available behavior, and the measurement of success.

### Different deployment goals, deeper understanding

[17:06](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1026s)

Environment building can support several destinations as demand for model customization grows:

- **Fine-tuning through model labs.** Brown anticipates labs offering services that let customers adapt their models to specific needs.
- **Specialized on-premises models.** A team may care most about the smallest model it can run locally at the lowest latency, optimized for one task.
- **Research for its own sake.** Experiments can advance shared understanding without an immediate product objective.

These goals differ, but each benefits from being able to define a setting, try changes, and examine what happens.

That ability also changes the relationship engineers have with models. Treating a model as a black box limits how much its behavior can teach you. Inspecting it, modifying it, and sometimes breaking it provides a more direct understanding of how it works and how its capabilities developed. Shared experimentation makes it easier to discuss what is being built—and to prepare for where the models may go next.

## Resources

- [Verifiers](https://github.com/PrimeIntellect-ai/verifiers) — Library for building LLM training environments and evaluations, integrated with Prime Intellect's training tools.
- [PRIME-RL](https://github.com/PrimeIntellect-ai/prime-rl) — Training framework with SFT, asynchronous RL, evaluation and examples ranging from small tasks to large models.
- [Wiki Search training example](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/examples/basic/wiki-search/README.md) — Current walkthrough for training a Qwen3 model to answer trivia using Wikipedia search and reading tools.
- [INTELLECT-3 release and training infrastructure](https://www.primeintellect.ai/blog/intellect-3) — Release announcement describing INTELLECT-3's model, post-training recipe, environments and GPU infrastructure.
- [Introducing cline-bench](https://cline.ghost.io/cline-bench-initiative/) — An initiative to turn real open-source engineering tasks into reproducible coding evaluations and RL environments.
- [Prime Intellect Community Environments](https://github.com/PrimeIntellect-ai/community-environments) — Community-contributed RL environments and evaluations with tooling for creating and publishing environment packages.
- [Run a hosted training experiment](https://docs.primeintellect.ai/hosted-training/end-to-end-run) — Current Lab guide covering baseline evaluation, training configuration, monitoring and evaluation of trained adapters.

## Slides

### Axes of Scaling

Will Brown contrasts scaling data, compute and parameters with scaling talent through community, applications and accessibility. Shared ideas and accessible tools let researchers build on prior work.

[0:38](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=38s) [Scaling laws list data, compute and parameters; practices list community, applications and accessibility. The slide asks how to scale talent.](https://assets.ai.engineer/sites/aiecode2025/sha256/ede20296907b67ee383654012b23372d357a103765e9152cc9633edb1c9951b7.webp)

Axes of Scaling

Laws:
• Data
• Compute
• Parameters

Practices:
• Community
• Applications
• Accessibility

How can we scale talent?

2

Interface text:
AIE / CODE
PRESENTED BY
Google DeepMind
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### Prime Intellect

Prime Intellect combines research, compute, a platform and open-source work to make AI research more accessible. Brown describes these as connected pieces of a toolkit for researchers beyond large labs.

[2:09](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=129s) [Prime Intellect describes itself as a research lab, compute provider, platform company and open-source ecosystem, with a mission to increase access to AI research.](https://assets.ai.engineer/sites/aiecode2025/sha256/69ec35a4637382abdacd4cf9bed05e88801688a954634076ff8fd20951cf8196.webp)

Prime Intellect

We are:
• a research lab
• a compute provider
• a platform company
• an open-source ecosystem

Our mission:
• increase the accessibility of doing AI research

Chart:
Legend: [illegible]
Y-axis: Mean Reward
Y-axis ticks: 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45
X-axis: Steps
X-axis ticks: 200, 400, 600, 800, 1000, 1200

3

Interface text:
Search
Location
Any
B200
VRAM: 180 GB
Socket: SXM6
Spot $1.98
Secure $4.04
Select GPU
H200
VRAM: 141 GB
Socket: SXM5
Spot $1.25
Secure $2.64
Selected
AiderPolyglot
Prime Intellect
Pinned
verifiers
prime-rl
prime
prime-environments
[illegible]

AIE / CODE
PRESENTED BY
Google DeepMind
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### AI Research vs. OSS

Will Brown compares open-source software with AI research as a practice: shared abstractions and better tools can improve iteration and compound over time, even though research ideas are less tangible.

[2:55](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=175s) [AI research shares abstractions, best practices, tooling and compounding improvements with software, but is more conceptual and less tangible.](https://assets.ai.engineer/sites/aiecode2025/sha256/23b776fe24b09306dc5e539c70f37167263f6fde5a666f8e27c821d1d7516dca.webp)

AI Research vs. OSS

“Models aren’t like software”

True, but research very much is:
• abstractions and best practices
• better tooling <—> iteration efficiency
• compounding improvements over time

Key difference: more conceptual, less tangible

4

Interface text:
AIE / CODE
PRESENTED BY
Google DeepMind
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### Open Superintelligence Stack

The stack connects aggregated compute and the Prime Compute Platform to RL libraries and lab tools for training and deploying open agentic models.

[4:30](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=270s) [A stack connects aggregated compute and the Prime Compute Platform to RL libraries, lab tools, and training and deployment of open agentic models.](https://assets.ai.engineer/sites/aiecode2025/sha256/99dce7dc6e708052fa648338e0a646a0e854dac02a6e742c20fbcf13f689533f.webp)

Open Superintelligence Stack

Training + Deployment of Open Agentic Models

OSS RL libraries:
prime-rl + verifiers

"Lab"
Sandboxes, Environments,
Evals, Fine-Tuning,
Inference

Prime Compute Platform
Aggregated Compute from Global Providers

5

Interface text:
AI Engineer
Engineering the future of AI

### The Product is The Model

Cursor’s Composer and OpenAI’s Codex illustrate models trained for their products. The speaker describes training a model in a harness that represents the product, using an RL environment.

[5:07](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=307s) [Slide comparing Cursor Composer and GPT-5.1-Codex-Max as product-specific models.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c8552144c259d8b3e0e093ba5a1cf03a1dc3c6320ff91dabf3c49f904f6094f.webp)

The Product is The Model

Composer: Building a fast frontier model with RL
Oct 29, 2025 by Cursor Team

Composer is our new agent model designed for software engineering intelligence and speed. On our benchmarks, the model achieves frontier coding results with generation speed four times faster than similar models.

We achieve these results by training the model to complete real-world software engineering challenges in large codebases. During training, Composer is given access to a set of production search and editing tools and tasked with efficiently solving a diverse range of difficult problems. The final result is a large-scale model optimized for high-speed use as an agent in Cursor.

Composer combines strong coding intelligence with best-in-class speed

Chart labels: Best Open; Fast Frontier; Frontier 7/2025; Composer; Best Frontier
Intelligence (Cursor Bench score) ticks: 0, 20, 40, 60
Speed (Tokens per Second) ticks: 0, 100, 200

November 19, 2025
Product
Release

Building more with GPT-5.1-Codex-Max

$ npm i -g @openai/codex

Introduction

We’re introducing GPT-5.1-Codex-Max, our new frontier agentic coding model, available in Codex today. GPT-5.1-Codex-Max is built on an update to our foundational reasoning model, which is trained on agentic tasks across software engineering, math, research, and more. GPT-5.1-Codex-Max is faster, more intelligent, and more token-efficient at every stage of the development cycle—and a new step towards becoming a reliable coding partner.

6

Interface text:
Get started
Listen to article
7:10
Share
AI Engineer
Engineering the future of AI

### Environments

An environment combines tasks, a harness, and rewards. The same framing spans evals, synthetic data for SFT, RL, and deployed agents.

[5:57](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=357s) [Tasks, harnesses and rewards alongside evals, synthetic data, SFT, RL and agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/514542d42fb78e38762a96a40441571e3996b767a07e8de9c42bf74c7b3b9252.webp)

Environments

• Tasks
• Harnesses
• Rewards

Evals
Synthetic Data
SFT
RL
Agents

7

Interface text:
AI Engineer
Engineering the future of AI

### The “Web Apps” of AI Research

Environments combine tasks, harnesses and rewards into self-contained starting points for research. Building more complex environments teaches new concepts, while predefined tasks and rewards enable experiments.

[6:35](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=395s) [Environment components and their qualities: self-contained, pedagogical, practical, simple or complex, and requiring experimentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/c107fd4a756cbee93b63416d492454f54982163281f8206a0bde90b6081a6f0d.webp)

The “Web Apps” of AI Research

• Tasks
• Harnesses
• Rewards

✳ Self-Contained
✳ Pedagogical
✳ Practical
✳ Simple or Complex
✳ Requires Experimentation

8

Interface text:
AI Engineer
Engineering the future of AI

### Environments Hub

The open-source Environments Hub helps people discover and share RL environments and evals, with search and categories ranging from coding to science and medicine.

[7:56](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=476s) [Environments Hub interface with search and category filters for discovering environments.](https://assets.ai.engineer/sites/aiecode2025/sha256/5baff8a2c929fae7fda6405674829df01e155bde4ddc3b6ac23a60cecd15da3f.webp)

Environments Hub
A community hub for discovering and sharing environments, both for RL training and downstream evaluation

Interface text:
Explore
My Stars
Team Environments
Search by name, author, description, tags...
Coding
Data & ML
Games
Math & Reasoning
Multimodal
Science & Medicine
Search & Tool Use

9
AI Engineer
Engineering the future of AI

### Environment Listings

Builders share original tasks and reimplement benchmarks for RL, with listings ranging from 2048 and medical questions to GPU kernels.

[8:20](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=500s) [Grid of environment cards covering coding, games, reasoning and other tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ed3ecdfb093c1ea110f9d9523f8245350f92435aff21534ec9e50d78555d209.webp)

Interface text:
hud · 16 stars
hud-browser-2048
Browser-based 2048 game for evaluating agents using visual...
cua · game · +3
Updated 3 months ago · v0.1.0

hud · 15 stars
hud-vf-gym
Generic adapter that bridges HUD's MCP infrastructure with th...
mcp · cua · +1
Updated 3 months ago · v0.1.1

ibrahim · 15 stars
terminal-bench
Terminal-Bench wrapper environment for verifiers.
Updated 2 months ago · v0.4.0

salty-duck · 14 stars
MAPP
Multi-Agent Path Planning Environment for reinforcement...
eval · train · +3
Updated 2 months ago · v0.1.0

sileod · 11 stars
reasoning-core-env
Formally verifiable reasoning on general symbolic domains...
math · train · +5
Updated 29 days ago · v0.1.0

maziyar · 10 stars
OpenMed_MedMCQA
Single-turn medical MCQ environment for MedMCQA with...
eval · think · +6
Updated 2 months ago · v0.1.1

popfido · 10 stars
kernelbench
KernelBench environment for verifiers - GPU kernel generation...
gpu · cuda · +6
Updated about 2 months ago · v0.1.3

sinatras · 8 stars
pmpp
PMPP CUDA evaluation environment with local and...
gpu · cuda · +2
Updated about 18 hours ago · v1.1.3

primeintellect · 8 stars
aime2025
AIME-25 evaluation environment
math · eval · +1
Updated 20 days ago · v0.1.14

manishram · 7 stars
medarc-k-qa
K-QA environment with FACTSCORE-style evaluation
Updated 25 days ago · v0.1.0

toilaluan · 7 stars
vrcli
RL for next chapter prediction for a story.
plan-generation · +2
Updated 3 months ago · v0.1.1

whyphylabs · 6 stars
vf-multi-swe-bench-...
Multi-SWE-Bench adapter environment for verifiers
Updated 3 months ago · v0.1.0

rose · 6 stars
arc-agi
ARC-AGI 1 + 2 (Abstract and Reasoning Corpus)
arc-agi · puzzles · +2
Updated 3 months ago · v0.1.1

maziyar · 6 stars
OpenMed_PubMedQA
Single-turn biomedical QA environment for PubMedQA with...
eval · think · +6
Updated 2 months ago · v0.1.1

jack · 6 stars
meow
Updated 2 months ago · v0.1.3

kunumi · 5 stars
GoodSirMath8k
Just GSM8K with the added reward based on how...
math · gsm8k · +3
Updated about 1 month ago · v0.1.0

nguyen599 · 5 stars
backend-bench
BackendBench environment for LLM kernel benchmarking
eval · kernels · +1
Updated about 1 month ago · v0.2.0

le​loy · 5 stars
nanogpt-speedrun
Evaluate code-generation and pretraining capabilities of LLMs v...
eval · sandbox · +2
Updated 2 months ago · v0.1.0

siro · 5 stars
backend-bench
Environment to evaluate LLMs on the ability to generate correct an...
Updated about 1 month ago · v0.2.0

chakra-labs · 5 stars
non-refusal
A simple benchmark for determining how much a model...
eval · safety · +1
Updated 3 months ago · v0.1.0

10
AI Engineer
Engineering the future of AI

### Verifiers

Verifiers provides mix-and-match components for building RL environments, from Q&A and games to coding agents, sandboxes and math.

[9:28](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=568s) [Verifiers supports environments including games, tool use, coding agents, sandboxes, evaluations, and math.](https://assets.ai.engineer/sites/aiecode2025/sha256/286bed7298577507c19a40530f91db962a3ea41071fed422c62c10c9971426bf.webp)

verifiers

Games
Tool Use
Coding Agents
Q&A
Sandboxes
Frameworks
Evals
Math

Interface text:
verifiers  Public
Environments for LLM Reinforcement Learning
Python  3.5k  433

11

AIE / CODE
PRESENTED BY Google DeepMind
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### Patterns + Extensibility

Verifiers organizes environments hierarchically, building from shared foundations toward applications. For example, ClineBenchEnv builds on HarborEnv, CliAgentEnv, MultiTurnEnv, and Environment.

[10:08](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=608s) [Examples of environment hierarchies for coding agents, games, search, and Python sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/97bdd32dc17ad4d8b426ea7e4f0a216dd3451d8410cec7abb1eda78316de4d55.webp)

Patterns + Extensibility

ClineBenchEnv
|
HarborEnv
|
CliAgentEnv
|
MultiTurnEnv
|
Environment

WordleEnv
|
TextArenaEnv
|
MultiTurnEnv
...

SingleTurnEnv
...

SearchEnv
|
MCPEnv
|
ToolEnv
|
MultiTurnEnv
|
Environment

PythonEnv
|
SandboxEnv
|
StatefulToolEnv
|
ToolEnv
|
MultiTurnEnv
...

12

AIE / CODE
PRESENTED BY
Google DeepMind

RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### Example: wiki-search — Environment Hub

Wiki-search gives an agent tools to find answers in Wikipedia pages. Its Environments Hub page packages the environment as a Python project with versions and dependencies.

[11:13](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=673s) [The wiki-search environment page shows its files, description, version, and dependencies.](https://assets.ai.engineer/sites/aiecode2025/sha256/9478f184e1fbadab0b97e736f2f398d884bcb4bcafad3622dbedb1a5503df191.webp)

Example: wiki-search

wiki-search

Overview
• Environment ID: wiki-search
• Short description: Multi-turn tool-use QA over a small Wikipedia corpus using ChromaDB and OpenAI embeddings, with judge-based scoring.
• Tags: retrieval, tools, multi-turn, embeddings, judge

About
Agentic RAG over Wikipedia pages for trivia Q&A

Tags
wikipedia  multi-turn  agentic-search  rag  train  eval  llm-judge

Last Modified
14 Days Ago

Version
0.1.20

Python Required
>=3.11

Dependencies
verifiers>=0.1.7
chromadb
datasets
openai

13

Interface text:
wiki-search
primeintellect
Fork
Starred 1
Train with Environment
Install Environment
Code  Evals  Discussions  Actions
0.1.20 (latest)
PI Research  Updated wiki-search to version 0.1.20  [illegible]  14 days ago  10 Commits
outputs
pyproject.toml  527 B
README.md  3 KB
wiki_search.py  8.9 KB

AIE / CODE
PRESENTED BY Google DeepMind
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect

### Example: wiki-search — Code and Configuration

Wiki-search combines Wikipedia tools and a dataset with tool and judge rubrics for rewards. A separate Prime RL configuration sets the training parameters.

[11:41](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=701s) [Python environment setup appears beside a training configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/f83aa8d60593fe4cb8263131f6e27b87db1d0a081169bd51e44856f0be1da79b.webp)

Example: wiki-search

def load_environment(
    [rest of signature omitted]
    system_prompt = "Use the provided Wikipedia search tools to help answer questions."
    tools = [
        search_pages,
        view_sections,
        read_section,
    ]
    parser = vf.Parser()
    dataset = load_dataset("willcb/wiki-trivia-questions-v4", split="train")
    tool_rubric = vf.ToolRubric(tools=tools)
    judge_client = AsyncOpenAI(
        base_url=judge_base_url, api_key=os.getenv(judge_api_key_var)
    )
    judge_rubric = JudgeRubric(
        judge_client=judge_client,
        judge_model=judge_model,
        parser=parser,
        judge_prompt=JUDGE_PROMPT,
    )
    judge_rubric.add_reward_func(judge_reward_func, weight=1.0)
    rubric = vf.RubricGroup(rubrics=[tool_rubric, judge_rubric])
    vf_env = vf.ToolEnv(
        dataset=dataset,
        system_prompt=system_prompt,
        parser=parser,
        rubric=rubric,
        tools=tools,
        max_turns=max_turns,
    )
    return vf_env

[trainer.model.experimental.lora]
rank = 8
alpha = 32
dropout = 0.0
target_modules = [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
]

[orchestrator]
batch_size = 512
rollouts_per_example = 16
seq_len = 16384
mask_truncated_completions = false
zero_truncated_completions = true
oversampling_factor = 2.0

[orchestrator.sampling]
max_tokens = 4096

[orchestrator.buffer]
online_difficulty_filtering = true

[[orchestrator.env]]
id = "primeintellect/wiki-search"

14

Interface text:
wiki_search.py
wiki-search.toml
configs > prime-rl > wiki-search.toml
RL ENVIRONMENTS AT SCALE
WILL BROWN / Research Lead
PRIME Intellect
PRESENTED BY Google DeepMind

### Example: wiki-search — Training

A Prime RL command starts the wiki-search training run. Its mean reward rises from about 0.2 to 0.75; Brown notes that runs do not always go this smoothly and require iteration on environments, rewards, data, and tasks.

[12:33](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=753s) [A training terminal is shown beside a rising mean reward curve.](https://assets.ai.engineer/sites/aiecode2025/sha256/bde05f1c12cd7f5f09bdd50bf849e0d345fde86805cea98216d68eeef8b3d8d3.webp)

Example: wiki-search

~/verifiers (main) uv run prime-rl @ configs/prime-rl/wiki-search.toml

Trainer
[default0]:wandb:
[default0]:wandb: View run wiki-search-4b-trainer at: https://wandb.ai/primeintellect/wiki-search-debug/runs/[illegible]
[default0]:wandb: View project at: https://wandb.ai/primeintellect/wiki-search-debug
[default0]:wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
[default0]:wandb: Find logs at: outputs/wandb/run-20251118_093623-[illegible]/logs
12:47:37 SUCCESS RL training finished!
root@industrious-accomplished-serval:~/prime-rl#

Orchestrator
12:20:00 INFO Starting orchestrator step 498
Generating rollouts (train): 0% | 0/512 [illegible]
12:20:00 INFO Hit async barrier because we are >4 step(s) async. Waiting for checkpoint 494
Generating rollouts (train): 100% | 512/512 [11:24<00:00, 1.34s/it]
12:31:29 SUCCESS Step 498 | Time: 688.97s | Reward: 0.8555 | Throughput: 14522.9 tokens/s | Seq. Length: 19431.2 tokens/sample | Async Level: 4 | Max. Off-Policy Level: 0
12:31:29 INFO Starting orchestrator step 499
Generating rollouts (train): 0% | 0/512 [illegible]
12:31:29 INFO Hit async barrier because we are >4 step(s) async. Waiting for checkpoint 495
Generating rollouts (train): 100% | 512/512 [10:08<00:00, 1.19s/it]
12:41:42 SUCCESS Step 499 | Time: 612.48s | Reward: 0.8301 | Throughput: 14371.1 tokens/s | Seq. Length: 17092.3 tokens/sample | Async Level: 4 | Max. Off-Policy Level: 1
12:41:42 INFO Saving final summary to file
12:41:42 SUCCESS Orchestrator finished.
wandb:
wandb: View run wiki-search-4b-orchestrator at:

Inference

reward/mean
Legend: wiki-search-orchestrator
Y-axis ticks: 0.2, 0.4, 0.6, 0.8
Step ticks: 0, 100, 200, 300, 400

15

Interface text:
tmux - verifiers
AI Engineer
Engineering the future of AI

### Example: wiki-search results

After training, Qwen3-4B’s wiki-search average reward rose from 0.550 to 0.890, just above GPT-4.1’s 0.880. Each run used 100 examples and one rollout per example.

[13:04](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=784s) [A model comparison table highlights rewards of 0.890 for the trained wiki-search model and 0.550 for the baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/4dd84094b85b0189e32a171a92c6e784fbfda73d43173ae45f6673ff5b7b70f5.webp)

Example: wiki-search

# | Model | Status | Run ID | Parameters | Average Reward
#1 | gpt-5-mini | Completed | dc83cc41 | Examples: 100; Rollouts per example: 1 | 0.920
#2 | willcb/qwen3-4b-wiki-search | Completed | 094df25d | Examples: 100; Rollouts per example: 1 | 0.890
#3 | gpt-4.1 | Completed | 50519885 | Examples: 100; Rollouts per example: 1 | 0.880
#4 | gpt-4.1-mini | Completed | 052a81ae | Examples: 100; Rollouts per example: 1 | 0.780
#5 | gpt-5-nano | Completed | 9a37a8bd | Examples: 100; Rollouts per example: 1 | 0.770
#6 | gpt-4.1-nano | Completed | 8d8aecd4 | Examples: 100; Rollouts per example: 1 | 0.630
#7 | qwen/qwen3-4b-instruct-2507 | Completed | e762fd98 | Examples: 100; Rollouts per example: 1 | 0.550

16

Interface text:
AI Engineer
Engineering the future of AI

### prime-rl + INTELLECT-3

Training INTELLECT-3 with prime-rl stress-tests end-to-end SFT and RL on a 100B+ MoE across 512 GPUs, while distilling lessons into reusable best practices.

[14:12](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=852s) [A training architecture diagram accompanies claims about full-stack testing and post-training a 100B-plus model on 512 GPUs.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a2acc6bba15985f2a1abd8bc38bfd33273d095266c42653b6bae1e2cd9072ec.webp)

prime-rl + INTELLECT-3

Full Stack Stress-Test
End-to-End Post-Train (SFT + RL)
100B+ MoE trained on 512 GPUs
Validating Infrastructure at Scale
Distilling Best Practices

Trainer
Consumes batch, updates weights
GPU 0, GPU 1, GPU 2, GPU 3, GPU 4, GPU 5, GPU 6, GPU 7

Orchestrator
Core data and scheduling logic
CPU

Inference
OAI API server + weight updating
GPU 0, GPU 1, GPU 2, GPU 3
GPU 0, GPU 1, GPU 2, GPU 3

Notifies of new weights
Writes training batch
Schedules rollout requests
(+notifies new weights)
Collects rollout requests

17

Interface text:
AI Engineer
Engineering the future of AI

### Lab

Lab is an upcoming platform for browsing environments and running evaluations, inference, and fine-tuning. Its example dashboard shows reward generally rising over training.

[16:12](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=972s) [The Lab slide shows environment and fine-tuning navigation beside a training metrics dashboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/14979eecd870e0594b6d830020992e3ddfec4421cd7d4f907988a368c22deb4f.webp)

Lab

Lab
Environments
Fine-Tuning

RL Training > Run 691bae74
Run 691bae74 — completed
Environment: reverse-text · Model: PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT · Started about 1 hour ago
Metrics · Rollouts · Settings

Mean Reward
0.564
+350.3% from start

Total Tokens
152,941
Across all rollouts

Total Rollouts
1,280
Generated samples

Training Metrics

Reward Distribution
Histogram of all reward values across training
Y-axis ticks: 0, 3, 6, 9, 12
X-axis labels: [illegible]

Reward Over Time
Mean reward distribution per training step
Y-axis ticks: 0, 0.2, 0.4, 0.6, 0.8
X-axis ticks: 0, 3, 6

Entropy per Step
Model exploration vs exploitation balance
Y-axis ticks: 0.7, 1.05, 1.4

Sequence Length per Step
Average completion length in tokens
Y-axis ticks: 90, 135, 180

Interface text:
19
AI Engineer
Engineering the future of AI

### AI Engineer Code Summit

[18:27](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1107s) [AI Engineer Code Summit closing slate with the event website.](https://assets.ai.engineer/sites/aiecode2025/sha256/f198b6fab802875fb00a1b0d3141c81e52d58479de0b37332507e080e5a1614d.webp)

AI Engineer
Code Summit

https://ai.engineer

## Full transcript

[0:00](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=0s) [on-hold electronic music] Today, we're talking about RL environments and how to scale them,

[0:25](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=25s) but the title's a little bit of a red herring. We'll talk a bit about the engineering pieces and, like, running these with thousands of parallel rollouts and sandboxes on hundreds of GPUs, but I'm mostly gonna focus on a different notion of scale.

[0:37](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=37s) Uh, and m- but what I mean by scaling here is we-- there's a, a number of different ways we talk about scaling in the context of AI and research.

[0:46](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=46s) We know about scaling laws, and we talk about how much data you need, compute, and parameters, and that if you pour in more data and compute and parameters or inference time, all of these things make models smarter or more performant.

[0:58](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=58s) But there's also a fuzzier side of scaling, which is sometimes referred to as unhobbling or algorithmic tricks or talent. But where does this come from? It's not just pouring in resources, but it's something that is more intangible, harder to put a finger on.

[1:13](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=73s) But really it comes from a community of people, a company, an organization, universities, the world, the internet, talking about ideas and sharing them and working on different applications, having these applications inspire ideas, using these ideas as testbeds for different techniques, and building on top of these to increase the accessibility for other people in the future to not

[1:35](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=95s) have to reinvent the wheel and to be able to build from, uh, what has been done by those before them to, uh, do more effective research and accelerate the pace of innovation.

[1:46](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=106s) And so why do we have this talent bottleneck? There's a big issue that we hear all about with AI labs trying to, like, find more talent, and salaries are going through the roof, and everyone wants to hire the best and brightest AI researchers.

[1:58](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=118s) But one other approach besides trying to just pay the most is increase the pool. Uh, and so how do we increase the pool of AI researchers? How do we make doing AI research more accessible?

[2:09](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=129s) And I wanna talk a bit about who we are at Prime Intellect. If you haven't heard of us, we are a mu- a bunch of things. We're a research lab, we are a compute provider, we're a platform company, and we are an open-source ecosystem.

[2:19](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=139s) We do a lot of things, and they all fit together in a way that I'm gonna try to explain in this talk. But we see these as all different pieces of how we can build a business around doing exactly this, which is increasing the accessibility of AI research and making doing research more of a toolkit available to

[2:35](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=155s) people at organizations around the world without needing to be inside of a large lab or without needing to spend crazy amounts on massive clusters or go do a PhD.

[2:43](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=163s) We think that there's versions of doing AI research that really should be part of the bread-and-butter workflows of AI engineers around the world as we build applications and try to improve our systems and models and products.

[2:57](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=177s) And I think, I think people are kind of iffy about in terms of AI is whether open-source models are gonna work. And in my mind, that's not quite the right analogy to draw.

[3:06](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=186s) And so when we're comparing, like, AI to traditional software, there's lots of, like, great examples of open-source software ecosystems that have thri- been thriving in the past, things like Linux and Node and Apache.

[3:17](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=197s) But in my mind, the analogy in AI is not models as kind of these fixed checkpoints, but it's about research as a practice and research as a set of ideas, and it's one that's more intangible.

[3:28](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=208s) But there's a lot of parallels in terms of the goals of the best practices of growing a research ecosystem as well as a software ecosystem, where you want to, uh, compound abstractions and best practices and have better tooling and iteration efficiency and have these gains over time allow, uh, more advanced, powerful, complex things to be built by,

[3:48](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=228s) uh, decreasing barriers to entry for any given application and allowing this to become more accessible. And so one thing that we-- a term we'll use to describe some of what we're building at Prime Intellect is we like this phrase called the open superintelligence stack.

[4:01](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=241s) One, 'cause it's a fun acronym, but also I think the idea of the stack of, of all the pieces of the puzzle to build the engine to go do research.

[4:09](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=249s) Uh, there's a lot of layers to it. You need compute, uh, you need orchestration, you need libraries for doing, uh, training and evaluation, and you need platforms to support things like code execution and evals and inference and fine-tuning, and we're doing all of these things.

[4:23](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=263s) Uh, but really the goal of this is to give people the tools to be able to go train models. We want people, more people in the world, and we think...

[4:30](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=270s) I'll explain why in a bit. There's a lot of reasons why, uh, the best products are going to be the ones that are not just kind of taking the thing out of a, a box of an API and putting a thin wrapper on it.

[4:42](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=282s) There's ways you can kind of improve around APIs, but I think in many cases people are realizing that winning products are going to be the kinds of things that whether it's the, a part of the model or a part of the stack, a part of the product or the whole thing, the ability to do research and have

[4:56](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=296s) at least the option of deciding where in your product you might want to customize a model or improve a model gives you a lot more flexibility to really, uh, make the best user experience.

[5:07](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=307s) Um, and so we have heard the phrase in the past that the model is the product, and I think we're starting to see now this change a little bit to a lot of winning applications have the product kind of be the model.

[5:19](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=319s) And I think the two notable examples of this that I'm big fans of and heavy users of are Cursor's new Composer model as well as, uh, OpenAI's Codex. And I think these are both, both good examples of models that really are...

[5:31](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=331s) where the product kind of is the model very directly, where the, the model was trained to be the model for that product, and the experience of using the model is the experience of using the product.

[5:41](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=341s) And the way that this is done is by taking a harness that represents the product and training the model in the harness in essentially an environment, an RL environment.

[5:49](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=349s) And environments really are just a harness with a collection of tasks and rewards, but they also have many other parallels throughout the ecosystem. Environments are not just for RL.

[6:00](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=360s) Environments are also essentially the same thing as evals.

[6:02](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=362s) Environments can also be engines for synthetic data, which then you can use for SFT or distillation. You can do RL in them directly, but also the agents we're actually deploying and monitoring out in the world, these are environments.

[6:13](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=373s) The product of these things, the tasks, the harness, and the rewards, whether this is a dataset offline or the stream of user tasks coming into a product, is an environment.

[6:23](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=383s) And so this as an abstraction I think is a very useful way of framing what it might look like to start having, uh, research become more of a, a practice that is adopted more broadly beyond just large AI labs.

[6:36](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=396s) And I also think that there's a sense in which they're a really accessible entry point. Uh, and so I like the analogy of environments as kind of like the web apps of AI research, and what I mean by this is that they're very simple, they're self-contained.

[6:47](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=407s) They can ... They start simple, but they can also get quite complex. They can get very elaborate representing the full complexity of a large product. They're also pedagogical in nature in that you can start simple and as you build complexity, you start bumping into these walls where you have to start learning new concepts, understanding more about scaling

[7:03](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=423s) the system side, understanding more about the hyperparameters and the algorithms. And they kind of open this door where you can, by playing around with them, start enter- entering into a world of research without needing to kind of build a whole training infrastructure system from scratch.

[7:17](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=437s) Um, and they also require experimentation. And so I think the key differen- diff- uh, differentiation between just an agent harness and an agent environment is that the environment forces you to also have your tasks and your rewards predefined to be able to do this experimentation.

[7:32](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=452s) It's a proper eval. And what this means is that you can't just vibe check it. You can't just, like, build it and test it out a bit and say, "Hey, it's good.

[7:41](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=461s) We're gonna ship it." It forces you to say, "Okay, let's think about this a little more scientifically. Let's do some experiments. Let's try out different models, try different hyperparameters."

[7:50](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=470s) Uh, and it also gets you to the point where you can start doing more advanced research in terms of RL training or distillation or fine-tuning. And, uh, so to really facilitate this, we wanted to make the environment as an entry point much more accessible.

[8:04](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=484s) A few months back, we launched what we called the Environments Hub, which is a open source community platform for creating, discovering, and sharing RL environments and evals. And so far we've had a lot of fun kind of seeing everyone build here.

[8:16](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=496s) We've had hundreds of builders and environments come create either their own ideas or re-implement papers. Uh, there's a bunch of examples here I can show you, but really it's just a bunch of people who have wanted to do research and found this as an entry point to start digging a little deeper, whether this was investigating some benchmark

[8:33](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=513s) and figuring out how to re-implement it or modify it to be appropriate for an RL context in terms of, like, new data or new examples, or whether this is some game that they'd been thinking about or some other task.

[8:44](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=524s) But having this as an abstraction for encapsulating the, the thing you want the model to do is a way of allowing yourself to start experimenting with ways of improving it without needing to have the answers.

[8:54](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=534s) So I think people talk a lot about how fine-tuning never really took off in the SFT regime, and I think a big part of this is that getting data was really hard of the actual, like, solutions.

[9:04](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=544s) I think having labeled examples of what you want the model to do is a very difficult thing to ask someone to go create. But if you can just think about the s- the settings it might be in without having the answers up front, if you can measure the answers, now you kind of can start creating data on

[9:19](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=559s) the fly. And this engine is really what the environment is about unlocking. Um, and so actually nine months ago, I was right here in this room. I had just released a library called Verifiers, which I'm still working on today.

[9:32](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=572s) Um, it's come a long way, but it's a toolkit for building these things, and it's been a lot of fun over this past year just playing with it and extending it to support more features and kinds of environments.

[9:43](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=583s) But the idea with Verifiers is to give people a toolkit that is a, a, essentially a bunch of components that you can mix and match and compose to do things like from simple evals or QA or games to things like tool use or using sandboxes or agent frameworks or, uh, c- uh, like CLI coding agents or math

[9:59](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=599s) problems. There's all sorts of things you might want models to do or agents to do, and it's a toolkit for building environments that is then, uh, ready to be, uh, automatically trained with reinforcement learning.

[10:10](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=610s) And the way we thought about this design, it's been a lot of fun and also a big challenge to think like, "Okay, how do you make a toolkit for this stuff that actually covers all the bases?"

[10:19](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=619s) And I think there's a lot of different approaches I've seen people go about, and I, I think they all make sense depending on what sorts of things you're wanting to work on.

[10:27](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=627s) But we took a very kind of a general approach where we tried to say, "We are not going to know all the answers right away. There are going to be lots of patterns.

[10:35](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=635s) There's going to be lots of special cases. There's going to be hierarchies of complexity. There's going to be patterns." And we really wanted to prioritize extensibility, so we think about these things hierarchically, where let's say you wanna do a co- a coding agent environment for Kleinbench.

[10:49](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=649s) Uh, this, which is an instance of the Harbor framework, which is a example of a CLI agent, which is a multi-turn environment, which is an environment. Uh, similar for TextArena and Wordle or for search with MCP or for giving a model a Python REPL in a sandbox.

[11:03](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=663s) And so thinking of these things hierarchically allows us to kind of really de- de- determine, like, what are the foundational pieces, what is generic across all environments, and then how do you build up the stack towards applications?

[11:14](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=674s) And so for one ex- like example of this that I'll kind of walk through the whole process end to end, uh, we, we call this one WikiSearch, but it's basically a simple search setting where we give an agent the ability to, uh, call some tools to search over Wikipedia pages and find some answers.

[11:28](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=688s) And so here is the Environments Hub page. So the Environments Hub is a kind of full stack, uh, code management package registry. So every environment is a Python project where you can have dependencies and versions and uploading your evals and whatnot.

[11:41](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=701s) Um, but the environments are very simple. They start simple. They can get really complicated, but this one's pretty simple, where we just kind of define our tools as async Python functions.

[11:50](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=710s) We have our dataset, and we have what we call a rubric. And so a rubric is the abstraction for managing the different pieces of your rewards, where you can kind of compose different things.

[11:59](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=719s) You can also have metrics that are just a zero reward but are for im- uh, observability of what's going on. And then the other piece of doing training will be a config.

[12:08](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=728s) And so the config here is for our Prime RL trainer, which is our kind of large scale training stack, which has been our, uh, culmination of all the best practices from the research literature for large scale asynchronous RL training.

[12:19](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=739s) Um, but the config files are intended to expose kind of the pieces that people need to think about. In ways that are starting to get you more into the algorithm but are also still designed to be pretty high level, pretty self-contained, and with, with defaults that we think are going to be sensible for a lot of people.

[12:34](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=754s) And so running this is just kind of running a command line with, uh, you specify the environment, and if it's in the Environments Hub, it'll automatically install it and start your training run.

[12:43](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=763s) And then you can, if you're lucky, see your reward curve just shoot right up. Um, and sometimes it doesn't g- go this nicely, but the process of doing this is iterating on your environment, on your rewards, and your data and your tasks to understand what makes this task holistically actually tangible in practice.

[13:01](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=781s) How do you tune the parameters? How do you look at your data? How do you define your rewards? Uh, and if you do this right, you can get really good improvements, especially from really small models, but also from much larger, larger models.

[13:10](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=790s) And so in this example, for the, the WikiSearch one, we started with a, a Qwen3-4B model, which was about fifty-five percent, and after training, it was at eighty-nine percent on par with, uh, much larger models like GPT-4.1, as well as reasoning models like, uh, GPT-5 Mini.

[13:24](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=804s) And so I think this practice of taking small models and being able to make them much better is a big win for a lot of applications where you either you want a really fast model, you want a really cheap model, you want a really, really powerful model because the best models out there just aren't quite good enough.

[13:38](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=818s) These are all the different things you can do with model customization. And this practice of doing, of creating environments isn't only for customization, but it gives you this option.

[13:46](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=826s) And so if you need to do evals anyways, it's useful to think of them as environments because the environment opens a lot of doors for whether this is prompt tuning or whether it's model selection or whether it's just getting a better sense of how your system could work at scale with many, many users in parallel.

[14:04](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=844s) It's a design process that really forces you to kind of pin down what is the thing I care about. What is my agent, what is my product, what is my harness, what am I optimizing for?

[14:14](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=854s) Um, and so to kind of fully stress test this, we've been training a large model which will be out into the world quite soon called Intellect-3 with our full Prime RL stack.

[14:22](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=862s) And this has been us really kind of validating the efficiency and performance at a very large scale. So this is a hundred B+ model trained on five hundred GPUs, where we've kind of done the end-to-end, uh, post train of SFT and RL, which the Prime RL stack also has SFT if people wanna do that.

[14:37](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=877s) But it's also been about just understanding all the best practices. We love reading papers, and we try to kind of try out all the tricks and see which ones work and see which ones don't, and then distill this into a library with Prime RL that can then be kind of consumed by the end user without needing to

[14:52](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=892s) do all of this, uh, implementation themselves. And so for us, it being open is very important. So Prime RL is on GitHub. You can go find it, verifiers is on GitHub if you wanna check it out.

[15:02](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=902s) And for us, this is really about opening the door for more people to start learning about these things and for incorporating it into their workflows for optimizing their models and their products.

[15:12](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=912s) Um, and the only way to do this that we've... What we see as the best way to do this is through growing community. And so for us, it's been really important to really think about getting good feedback loops from the people who are building with this and understanding what they want, understanding what's going well, understanding what's painful,

[15:29](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=929s) and addressing those problems. And so we've done a number of community programs in terms of sponsoring different kind of small tasks to, uh, a research residency program with, uh, grad students around the world, uh, and collecting like a, a smaller subset of the Environments Hub ones where we'll actually review them manually.

[15:43](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=943s) And so this repo here, the Prime Environments repo, is the ones where we are doing these directly, where we're kind of offering to look over someone's kind of example implementation.

[15:52](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=952s) And so we've had hundreds of these come in, and there will be hundreds more, and, uh, it's been a great learning process because it's forced us to fix a lot of things.

[15:59](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=959s) We kind of understand the rough edges, we understand what we need to add, and we're kind of then distilling all of these learnings into what will be our kind of upcoming, uh, platform product, which we're calling Lab.

[16:11](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=971s) And the idea of Lab is to give people an interface, a platform where they can browse environments, they can run their evals, they can do their inference, they can do their fine-tuning, and they can have research be more accessible in a way that it hasn't been historically.

[16:25](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=985s) Because I think a lot of people find infrastructure very painful. They find dealing with Torch versions painful, Flash Attention, vLLM, and getting all these things to work. We are happy to do that, but we understand that a lot of people may not want to.

[16:40](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1000s) Um, and so the idea with this is that if you wanna go read the code, you can go read the code, but you don't have to run it. We can run it for you.

[16:47](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1007s) Um, and so this has been our version, which will be kind of out into the world in the near future, of trying to allow people to really focus on the environment where the entry point to Lab will be the environment.

[16:59](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1019s) If you wanna do synthetic data in SFT, build, let's build an environment. If you wanna do your evals, you build that as an environment. If you wanna do RL, you build an environment.

[17:06](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1026s) And I think building an environment is the kind of thing that

[17:11](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1031s) I imagine a lot p- more people are going to want to be doing as we start really seeing where models are headed. In some cases, this will be we're gonna use fine-tuning services from the labs because they're gonna offer this because people want it.

[17:24](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1044s) In some cases, this will be we really care about the smallest model we can run on-prem at the lowest latency, and we're really just gonna optimize for our one thing.

[17:32](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1052s) Or it could just be research for the sake of research and advancing our kind of collective understanding of how this stuff all works. And I think that's really our goal, is to have a world where there's gonna be a lot of AI and where we can all kind of talk about it and understand it and look at

[17:46](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1066s) it and poke at it and tweak it and have a better sense of what we're actually building, because I think there's a lot of times when it feels like we're just kind of the model is a black box.

[17:55](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1075s) And digging into the research and going under the hood and changing things and breaking things tells you a lot about how these models work. It tells you a lot about understanding where they came from, where they could be going, where they might be headed, and preparing for that future.

[18:10](https://www.youtube.com/watch?v=_IzZWeuTx7I&t=1090s) Thanks. [upbeat music]
