AI Engineer World's Fair 2026

Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

Sina Shahandeh· CTO/CAIO, RADiCAIT19:23

Loading slides…

Read the talk

Autonomous Agents for Scientific Tasks

A CT-to-PET research workflow shows how explicit problem hierarchies and specialist model review can help coding agents move beyond local optimization.

From a talk by Sina Shahandeh

Before you start: Familiarity with model training, convolutional networks, and coding agents will help; PET imaging and registration are explained as they arise.

When the optimization loop runs out of ideas

Give a coding agent a model, an error metric, and permission to change the code. It can run experiments and search for changes that reduce the error—the basic hill-climbing pattern Sina Shahandeh introduces through Karpathy’s autoresearch. The harder question is what happens when the next useful experiment requires a new idea.

In open-ended scientific work, implementation can continue after useful progress has stalled. An agent can edit code and run experiments over substantial data, yet exhaust the hypotheses that would make those experiments worthwhile. Shahandeh describes this as a shortage of research taste. His opening comparison shows agents plateauing while capable humans continue improving, with the strongest humans improving further; it is an illustration of the problem, not a benchmark result established here.

A paper title above a Human vs Agent line chart; blue human curves rise while orange and green agent curves flatten.Open full source frame
Human and agent performance over time, with annotations showing agents plateauing while humans continue improving.

The scientific method separates these activities: observe a situation, ask a question, formulate a hypothesis, implement an experiment, and learn from its result. Memory and organized records of activity help prevent repeated mistakes. Shahandeh regards that learning machinery as comparatively tractable. The central challenge is generating a hypothesis worth testing next.

0:120:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

A concrete research objective: CT to synthetic PET

At RADiCAIT, the motivating problem is in silico PET: generating a synthetic PET image from a CT scan. Consider a pulmonary nodule visible in CT. The clinical question is whether it is cancerous, and a PET scan can provide information about tissue activity. The demonstration places a chest CT beside a synthetic PET image, with the nodule marked in both.

Grayscale chest CT beside a colored in silico PET image, connected by an AI arrow, with green lung-nodule boxes.Open full source frame
Diagnostic CT and in silico PET shown side by side, with a lung nodule marked in each.

PET reflects radioactive-tracer uptake: tissues that absorb more tracer appear brighter, and tumors often exhibit that behavior. The image-translation model attempts to learn the relationship between anatomical structure in CT and the activity represented in PET. This is the research objective of the example, not a demonstration that a generated image can replace a clinical PET scan or establish a diagnosis.

Such an objective spans years of research, so it must be broken into smaller goals, each with its own experimental loop. One is model training. The described GAN architecture encodes CT and decodes it into PET; making that experiment useful also requires assembling a suitable dataset. Shahandeh estimates that preparing good data accounts for 80% of the work in this workflow. With architecture and data in place, fidelity metrics compare synthetic PET with real PET. Those measurements define a starting point for further optimization.

2:502:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:50 · section reference included

From local adjustments to a different architecture

The Codex run Shahandeh presents follows several branches: some changes improve the system, some do nothing, and others lead to dead ends. Eventually progress saturates. The existing model uses a 2.5D arrangement: it stacks CT slices as channels but processes them with 2D convolutions. That representation becomes the concrete opportunity for a broader hypothesis.

ApproachTreatment of CT slicesSearch implication
Existing 2.5D modelSlices stacked as channels; 2D convolutionsTune within the current architecture
Proposed 3D alternativeUse 3D convolutionsTest a different architectural choice

In this run, familiar adjustments such as hyperparameter changes did not naturally lead the agent to the larger architectural move. Shahandeh intervened with suggestions, including considering 3D convolutions and reading papers for alternatives. The limitation concerns the search behavior he observed, not a rule preventing autoresearch from changing architectures. The engineering problem is how to make broader hypothesis generation part of the loop instead of relying on those human interruptions.

5:165:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:16 · section reference included

Make decomposition an explicit action

The intervention begins before another training experiment: ask the agent to decompose the problem. The top-level objective remains specific—translate pulmonary-nodule CT patches into equivalent PET—but the search now has named components to inspect. Shahandeh compares this explicit decomposition to prompting a model to reason step by step. The key is to request the decomposition as an action, rather than assume it will happen implicitly while the agent edits code.

The component inventory reaches beyond the neural network itself:

  • Data preparation: the inputs and preparation work on which training depends.
  • Learning architecture: the model that performs the translation.
  • Training loss: the objective used to train it.
  • Operations: the machinery that runs the modeling workflow.
  • Metrics and evidence: how changes are evaluated.
  • Peripheral scripts: supporting code around the model.

This gives hypothesis generation several places to look, including components that a narrow request to improve the model might overlook.

The coding agent inspects the codebase and produces a hierarchy of linked documents describing those components. The documents make the problem’s structure available for subsequent reasoning: what matters, what can change, and where a change belongs. The purpose is not documentation for its own sake. The hierarchy defines the areas the agent should search for improvements.

6:497:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:49 · section reference included

Follow the hierarchy down to code

The Obsidian walkthrough shows how the generated documents connect a research goal to implementation:

  1. Start with a level-one README containing the problem statement and the documentation entry point.
  2. Follow links to level-two components, which lead to more specific level-three components.
  3. Enter the model-architecture branch and inspect the generator document, which describes the existing 2.5D generator.
  4. Continue to the actual implementation code at the leaves of the hierarchy.

Each document is simple; the value comes from the linked path between levels.

Build this structure before starting the optimization loop. The agent can then reason through the hierarchy step by step, moving from the objective to a component and finally to an implementation. That gives the next hypothesis a location in the system, rather than leaving it as an isolated suggestion.

8:388:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:38 · section reference included

Generate candidates, then challenge them

Once the hierarchy exists, ask for ideas that improve the objective using its components. A bare codebase-plus-goal instruction, like the role played by program.md in the autoresearch example, leaves the agent to choose its own search coverage. The hierarchy supplies an explicit scaffold. Shahandeh suggests generating 100 candidate solutions as an illustrative target, not a reported count of successful experiments. The displayed proposal table organizes hypotheses by component and pairs proposed changes with evaluation hooks.

A slide about generating hypotheses through an adversarial or collaborative loop, above a table of configuration, runtime, and data-labeling proposals.Open full source frame
Component-by-component hypotheses pair proposed changes with evaluation hooks.

The resulting search can consider modifications across the system. In Shahandeh’s example, this process surfaces a three-dimensional alternative to the initial architecture—a more substantial change than another local adjustment. It remains a hypothesis to test, not evidence that 3D convolutions have already improved the model. Another agent or model can review the plan collaboratively, developing the ideas, or adversarially, challenging their assumptions.

The experiment loop still starts from a codebase, metrics, and a goal. It now includes a more deliberate hypothesis stage before implementation: formulate the required change, implement it, evaluate the outcome, and repeat. Shahandeh reports that better hypotheses make this process more effective; the talk does not quantify the gain.

9:5510:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:55 · section reference included

Registration needs more than a numerical score

A second example returns to data preparation: image registration, the process of aligning scans. The demonstration overlays grayscale CT with colored PET activity. The liver shows high uptake, while the lung regions appear dark. Because the scans were acquired at different times, their initial positions do not match. Registration must bring them into alignment before the paired images can be used reliably together.

This is not merely a matter of shifting one image. Respiration, patient movement, changes in the body, different scanners, and different positioning can all complicate the correspondence. Registration is therefore another optimization problem with multiple components and multiple ways to fail.

Some of its quality judgments are qualitative, and a poor result does not necessarily identify the bug that caused it. A scientist may inspect images one by one and ask whether the result makes sense. Two concrete checks in the demonstration are whether a lung mask actually covers the lungs and whether scans have been cropped or truncated correctly. Each processing stage can require its own review-and-improvement loop.

12:0612:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:06 · section reference included

Bring specialist models into the loop

The coding agent need not perform every kind of review itself. A model with stronger multimodal capabilities can inspect an image and assess alignment. A stronger reasoning model can help generate hypotheses or critique changes. Shahandeh names GPT-5.5 Pro for the latter role, attributing its usefulness to greater inference-time compute. These are distinct jobs: judging what an image shows and deciding what experiment should follow.

Skills make those consultations available inside the agent workflow. Shahandeh describes an image-review skill using Gemini, then introduces Peter Steinberger’s Oracle CLI for packaging code and data and sending that context to an external model API. The relevant integration is the consultation step; current Oracle defaults or API examples should not be assumed to be the configuration used in the demonstration.

Close the loop by returning both the implementation and its outcome to the reviewer. The review must connect what changed with what happened, assess whether the result makes sense, and propose the next action. Shahandeh reports better hypothesis generation and more useful critique through this process, based on his experience rather than a quantified comparison. A second model is useful here because its judgment feeds the next experiment, not simply because it supplies another opinion.

14:4715:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:47 · section reference included

The remaining problem of scientific observation

This workflow gives the scientific loop a more rigorous hypothesis stage, but it does not make every part autonomous. Shahandeh considers implementation comparatively solved when the work can run inside a basic simulated environment. That qualification matters: a task executable in a computational box gives the agent a much more accessible experimental setting.

Better hypotheses still depend on reliable observations. Returning to the scientific images, Shahandeh argues that contemporary general multimodal models do not reliably recognize tiny details such as pulmonary nodules. He attributes that weakness to insufficient training on scientific images and data. The grid of axial scans and colored overlays illustrates the kind of material a reviewer must interpret; the presence of an image-review model does not itself establish that it can notice the scientifically decisive feature.

A four-by-four grid of grayscale axial scans with colored overlays and a vertical color scale.Open full source frame
PET/CT registration quality-control views across multiple axial slices.

Fine-tuning or other adaptation could help models observe those details more like a trained scientist. Until then, observation remains a major obstacle to closing the entire loop and building a fully autonomous scientist in a data center.

16:4116:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:41 · section reference included

Give additional computation a structure

Hypothesis quality remains the other limiting capability. Shahandeh compares hierarchy scaffolding to the way chain-of-thought prompting can elicit stronger reasoning from an existing model, using GPT-4 as an analogy rather than a claim about where that technique originated. His forecast is that models pretrained to decompose and compartmentalize problems more effectively may eventually need less explicit scaffolding.

For the present workflow, decomposition is a deliberate sequence: expose the hierarchy, inspect its components, and look for improvements within each one. That structure gives additional test-time computation somewhere useful to go. More generated tokens can support broader hypothesis search around a defined research objective, while implementation and observed outcomes keep the search connected to the experiment loop.

18:0318:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:03 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…