AI Engineer World's Fair 2026
ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs — Brendan Rappazzo
Loading slides…
Read the talk
ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs
AlphaLab turns prediction tasks into parallel research campaigns, then uses the failures of that approach to make evaluation environments the foundation for improving the system itself.
From a talk by Brendan Rappazzo
Before you start: Familiarity with model training, held-out validation and coding agents will help; no quantitative-finance background is required.
Where more research cycles could help
Given a time-series dataset, can you predict future values while keeping the model well calibrated? That is the shape of many applied problems facing Brendan Rappazzo’s team at Morgan Stanley. He describes a group of about 30 PhD AI researchers whose work spans academic publishing, open-source releases and internal applications. Some of those applications resemble Kaggle tasks: the input, prediction target and constraints are sufficiently explicit to support repeated experimentation. Rappazzo also suggests that the sell-side setting involves less adversarial selection, making these problems a promising place to automate research.
The opportunity starts with algorithms already in production. More hyperparameter tuning or better combinations of existing methods might produce improvements that a small team lacks time to pursue. It extends across trading desks: a model built for credit bonds might transfer to municipal bonds, provided someone does the adaptation and testing. Other desks have little existing machine learning, so even a reasonably trained first model could be useful. These are different research starting points, but all benefit from sustained cycles of implementation and evaluation.
Rappazzo dates the practical turning point to December 2025. With Opus 4.5 and coding harnesses such as Claude Code and Codex, long-horizon work began to feel feasible enough for the team to make autonomous research a major effort. The enabling combination was a capable model and an execution environment that let it continue working through a substantial task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One harness, different starting points
The business goals are increased P&L and more algorithms reaching production. To contribute to those goals, the system has to integrate with existing data access, backtesting and evaluation infrastructure while supporting two ends of the research spectrum.
| Starting point | Work delegated to the system |
|---|---|
| Data path and a natural-language prediction goal | Research the problem, build an evaluation and begin experiments |
| Data scripts, an evaluation and good existing models | Run additional experiments to seek improvements |
The second case matters as much as the first: autonomous research can be useful without inventing a modeling pipeline from scratch.
The harness also needs to work with different frontier providers and increasingly with open-source models. Its design should benefit as those models improve, rather than depend on elaborate machinery that stronger models make obsolete. A further requirement is harder to express as an interface: how should Morgan Stanley’s enterprise knowledge and its researchers’ expertise enter the system? That question eventually changes where the team concentrates its engineering effort.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a prediction request to trained models
The first public system, AlphaLab 1.0, came with a technical report and publicly released code. Rappazzo recalls an early-April release. That version establishes the research workflow; the failures encountered afterward motivate the richer evaluations and environments behind AlphaLab 2.0.
Consider an exchange-rate dataset. A user supplies either a file path or API access with the API specification, then asks the system to predict the exchange rate one day ahead. AlphaLab turns that request into three phases:
- Research: understand the data and relevant modeling approaches.
- Evaluation building: construct the evaluation or backtest against which experiments will be judged.
- Mass experimentation: implement and evaluate candidate approaches.
The output is a suite of trained machine-learning models for the requested prediction task, rather than just a written research recommendation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tools for research and cluster execution
AlphaLab uses a custom harness rather than an off-the-shelf agent framework. Rappazzo credits Claude with writing the implementation and values the freedom to change any part of it. Function-based tool calling provides the boundary between models and actions, allowing the harness to accommodate OpenAI, Anthropic and open-source providers.
Three tools supply most of the practical capability:
- Shell access: agents write Bash commands, create Python environments, edit code and run it.
- Web search: agents consult arXiv papers, technical blogs and other public material to identify relevant methods.
- Slurm submission: agents describe the hardware a training job needs and submit it to the GPU cluster.
The Slurm boundary lets an agent request resources without managing the underlying hardware orchestration. For example, a worker’s Bash submission for four H100s can take this form:
bash
sbatch --gpus=h100:4 --wrap='python train.py'
The worker still needs to prepare the training code and job configuration. The scheduler handles allocation; CPU requirements can be included in that configuration as well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Research context that survives a handoff
The research phase serves as an expanded CLAUDE.md or /init: its purpose is to gather enough context for agents to form useful hypotheses and test them. AlphaLab runs on the server, with a lightweight UI exposing its progress. The first substantial artifact is a to-do list whose completion should leave the system ready to experiment.
That list also gives the harness a way to detect unfinished work. If an agent tries to exit early, outstanding items provide a reason to prompt it again. Tasks include setting up the Python environment, loading data and performing statistical tests. Each item produces Markdown notes, so later agents can query the relevant findings dynamically instead of needing the entire research history in context. Most web search happens here, adding public research to the local understanding of the dataset.
Rappazzo estimates that the research phase takes roughly three to four hours, with substantial variation. Its lasting output is the accumulated context that evaluation builders and experiment workers can reuse.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building an evaluation, then trying to break it
Optimizing against a faulty evaluation undermines the entire research campaign. An LLM does not need malicious intent to introduce a mistake that makes a model look better than it is. AlphaLab 1.0 therefore assigns evaluation construction and review to separate agents; this is also the phase that changes most in the next version.
The original review loop divides responsibility into three roles:
| Role | Responsibility |
|---|---|
| Builder | Write the evaluation code and repair reported issues |
| Conceptual critic | Check the evaluation’s logic and forward information leakage |
| Tester | Write unit and integration tests |
The two reviewers write up issues for the builder. Repairs return through the review loop, which continues until all agents accept the evaluation. This separates questions about whether the experiment measures the right thing from questions about whether its code behaves as intended.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A research campaign on a Kanban board
Mass experimentation is organized as a Jira-style Kanban board. A strategist agent queries the context gathered in earlier phases, proposes experiments and places cards in a to implement column. Worker agents pick up those cards and carry each proposal through execution.
A worker’s responsibility extends beyond generating code:
- Implement the proposed strategy.
- Write the Slurm configuration, including its hardware requirements.
- Submit the training job and wait for completion.
- Inspect training curves for underfitting or overfitting and examine evaluation results.
- Write a postmortem and return it to the strategist.
The postmortem closes the loop. A completed experiment changes what the strategist knows, which should change what it proposes next. Rappazzo gives the example of three disappointing transformer variants alongside strong XGBoost results: the next useful move may be to explore more tree methods. That is a decision about where to spend further experiments, not merely which finished model ranks highest.
The board is also a human steering interface. Researchers can cancel cards, add their own experiments or chat with the strategist to supply domain intuition and encourage different approaches. Meanwhile, a growing leaderboard shows model performance against the evaluation and a held-out private validation set. Researchers can drill down from an idea’s inception to its implementation and extract the code for further work.
The later UI demonstration makes the worker lifecycle visible as experiments move through building, deployment and analysis. It is sped up; the animation illustrates how work flows through the system, not how quickly training jobs finish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Results from public tasks and internal models
The initial paper explores CUDA kernels, an academic traffic time-series dataset and Karpathy-style LLM training speedrunning. Rappazzo reports that AlphaLab found a better LLM training configuration than a single-agent loop, while acknowledging the difficulty of benchmarking the system. The comparison concerns the research process’s ability to find a training configuration, rather than the general intelligence of the model running that process.
Rappazzo reports a top-12% finish among submissions in an NVIDIA Kaggle competition to fine-tune Nemotron for reasoning, after only about ten iterations because the team entered late. He expects more iterations to help AlphaLab explore further, but additional improvement in that competition is a hope rather than a measured outcome.
Internally, he describes a handful of useful improvements to already decent models. Those improvements are working through risk review toward production; he does not report quantified internal gains or say that the review process is complete. This is the more constrained starting point from the original design: give AlphaLab a working approach and let it search for improvements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the failures left unanswered
Successful experiments did not resolve failures on harder problems. Those failures raised a measurement question: how could the team distinguish dependable research capability from a collection of encouraging outcomes? They also exposed uncertainty about the harness itself. A research phase followed by a strategist and workers sounds plausible, but plausibility does not establish that this is the best arrangement.
Harness design could itself be a verifiable optimization problem. If alternative arrangements can be measured, an LLM could help improve those arrangements. At the same time, the original data-plus-goal interface leaves much of the enterprise’s knowledge and the researchers’ expertise outside the system. Rappazzo locates the answer to all three concerns—capability measurement, harness design and expert knowledge—in deliberately constructed evaluations and environments. He describes the original approach as overeager to treat the agent like a human researcher before establishing a clear way to measure its work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A stricter contract for AlphaLab 2.0
AlphaLab 2.0 makes the evaluation boundary much more explicit. The arrangement resembles Kaggle: data and a task description enter the system, the harness submits containerized models, and the environment returns a public leaderboard score. The user separately sees performance on a private, held-out validation set. This makes a submission’s task performance measurable while separating the feedback available to the harness from the private results available to its operators.
Once tasks share this strict format, they can support both evaluation and learning. Rappazzo reports building roughly ten to twenty carefully designed environments to supply a reinforcement-learning signal. In this formulation, an evaluation becomes an environment when the system uses it to train and improve, rather than only to report a result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Using AlphaLab to improve AlphaLab
A stable measurement setup makes manual harness tuning more informative. Should there be two strategists? Should they debate? Those proposals can become experiments rather than permanent architectural commitments justified only by intuition. Rappazzo calls the next step “AlphaLab-ing AlphaLab”: an LLM examines execution traces and evaluation results, then modifies the harness itself. The object being optimized has moved from a prediction model to the machinery that conducts the research.
Weight training provides another route. The team is collecting good traces from open-source models and working with GRPO and on-policy distillation methods to improve model behavior. Rappazzo anticipates that the best system may combine open- and closed-source models. Orchestration and model training are therefore related optimization choices: the aim is to improve the performance of the complete research system, not to assume one fixed agent arrangement or model provider.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Encoding expertise in the environment
Building a useful environment requires more than proprietary data and a numerical metric. The team also develops qualitative rubrics that examine rollout traces: what constitutes good research, how should a researcher reason through a problem, and how closely does the agent follow the team’s research process? Grading those traces turns expert judgments about method into a learning signal alongside the measurable task outcome.
The durable investment is the environment that defines and recognizes good research. With a strict evaluation setup, the orchestration in the middle can change. AlphaLab 1.0 supplies an initial implementation, but the longer-term aim is a system capable of improving that implementation itself. Rappazzo forecasts that general automated research will become a commodity, citing GLM 5.2 as an example informing that expectation. Manual tuning remains part of the team’s current work; a fully self-improving research process is the destination.
That distinction also explains the plan to continue releasing the system publicly. AlphaLab 1.0’s code and research are already public, and Rappazzo intends further releases because he places the enterprise-specific value in the environments: the data, evaluation criteria and research rubrics that make improvement meaningful for the organization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original report details the research harness, evaluation construction, experimental results and ablations across optimization domains.
A workflow overview with a sped-up dashboard demonstration, benchmark summaries, and links to the paper and code.
Further reading
Morgan Stanley's public research repository indexes publications and projects, including an AlphaLab code link.
Full transcript
Loading transcript…