AI Engineer World's Fair 2026
The Loop Is the Product — Roland Gavrilescu, Introspection
Slide chapters
3 slide and demo groups · Timestamps are approximate. 27 additional groups were omitted from this pass.
Read the talk
The Loop Is the Product
Roland Gavrilescu’s blueprint for improving agents connects execution, human judgment and production experiments: capture what a system learns in portable recipes, then optimize for work users value at a cost that makes sense.
From a talk by Roland Gavrilescu
At a glance
Ideas worth remembering
Build the feedback path as carefully as the worker: signals guide execution, verifiers check success, and artifacts inform the next improvement loop.
Preserve operating knowledge in versioned agent recipes: failures become evaluations, repeated behavior becomes skills, and human judgment remains part of the system.
Let humans calibrate the desired judgment and agents implement the evaluations. Then test whether users value that judgment before promoting a recipe change.
Establish that the output is valuable, then make its economics viable. Efficient execution only becomes a useful product advantage when customers want the work.
A car negotiation gives the loop an outcome
An agent that negotiates a car purchase has a concrete job: find available cars, compare prices, ask dealers for better offers and recognize when to close. Roland Gavrilescu, co-founder of Introspection, uses this example to introduce a product question that grew out of his work on agent infrastructure at xAI: how do you turn agents running long tasks into services that improve with their customers?
The discussion around agents had moved from improving models to improving harnesses—the surrounding software that lets a model use tools—and then to building loops. “The loop is the product” makes that progression concrete. The product needs a process that can act, receive useful information and decide what to do next.
In Gavrilescu’s account, an OpenClaw user assembled that process around car buying. Reddit discussions supplied price information; inventory searches established what was available; conversations with dealers produced competing offers. Putting dealers head-to-head gave the agent another action to take when an offer was unsatisfactory. A verifiable way to recognize an acceptable price supplied the stopping condition, after which the buyer could lock in the purchase. Gavrilescu reports that it worked, without giving a purchase price or savings figure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Signals guide the work; verifiers judge the result
The next reference is the OODA loop: a way to organize observation and action in a changing environment. Gavrilescu connects it to agents calling tools and taking observations. Each action changes what the system knows, and the next decision depends on that new information. In the car example, a dealer’s response becomes input to the next negotiation.
Two parts of the loop deserve separate attention:
- Signals: The information that guides the agent’s next action. Gavrilescu identifies signal quality as a determinant of the loop’s success rate.
- Verifiers: The checks that establish whether an apparent success is actually correct. A result can look promising while failing the criteria that make it useful.
What changes when the result of one loop becomes input to another? The diagram shows the extra feedback path. The first loop does the work and checks its result. Its artifacts then give a second loop material for understanding what went well, what failed and what should change. Continuous improvement depends on making that return path useful.
Execution feeds an improvement loop
Read the diagram as text
- Signals. Execution — Information that guides the worker
- Agent work. Execution — Call tools and take observations
- Verify the result. Execution — Check whether apparent success is correct
- Learn from artifacts. Improvement — Understand successes and failures before the next loop
- Signals → Agent work: Guide execution.
- Agent work → Verify the result: Check the work.
- Verify the result → Learn from artifacts: Pass results and artifacts.
- Learn from artifacts → Signals: Feed learning into the next loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Store what the system learns as an agent recipe
“System distillation is the moat” names the second idea: preserve the knowledge accumulated while operating an agent. A run can teach you about its harness, profiles, evaluations, model, resources, tools or environment. If those lessons remain scattered across individual runs, they are hard to reproduce or improve deliberately. An agent recipe gathers them into something portable, versioned and able to evolve.
The analogy is to data recipes in reinforcement learning. As unwanted behaviors such as hallucination and reward hacking appear, researchers change the recipe that produces the system. Gavrilescu proposes applying that accumulated know-how to the broader agent system, including adjustments and human judgments discovered only after the agent begins acting in its environment.
Different observations become different parts of the recipe:
- Failure patterns become judges and evaluations. Recurring mistakes supply criteria for checking subsequent runs.
- Repeated behavior becomes skills and prompts. Useful ways of working become reusable instructions.
- User frustration informs harness extensions and memory. Problems encountered during use change the support the agent receives.
The proposed recipe belongs to the company using it and remains independent of a particular model or provider. Putting it in a Git repository makes its evolution part of the product’s ongoing development. Here, distillation includes evaluations, skills, prompts and operating choices; the accumulated system knowledge has value even before any of it moves into model weights.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pi recipes preserve the choices behind the agent
Introspection’s implementation uses the Pi harness, Harbor for evaluations and Git repositories for versioning. Gavrilescu describes an early release called Pi recipes, intended to be owned by the user and managed by agents. Version history gives those agents a way to track what changed and why. The release is presented as an early implementation of the recipe idea, rather than a demonstrated guarantee of reproducibility across every provider.
The knowledge worth carrying includes the maker’s taste: the judgments that distinguish a desirable result from a merely plausible one. Agents should calibrate themselves to those judgments. Reusing someone else’s recipe would therefore mean bringing along the choices that produced the system, including the reasons behind them.
That expands the recipe beyond a skill’s instructions for doing a task. It also needs ways to encode taste in evaluations, run those evaluations, improve them over time and choose useful signals. Tool choices and harness profiles may vary with the model. Provider independence therefore still leaves room for model-specific operating decisions inside the portable recipe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish value before optimizing its cost
The third idea is “valued work per watt.” Gavrilescu points to Cursor and Cognition as examples of a progression from building a product, to building evaluations for that product, to building models informed by both. He proposes extending this pattern beyond coding into domains such as customer support and legal research. The order matters: first establish what work has value, then determine whether delivering it is a good economic deal.
A base harness and initial evaluations provide a starting point. Running the system in production reveals the requirements needed to reach the desired level of performance. Once the system can produce valuable work, the next research problem is delivering that value without spending more than necessary. “Per watt” serves here as an optimization principle connecting useful output to resource use; the talk does not define an energy measurement or present a numerical efficiency benchmark.
Fine-tuning APIs and infrastructure make parts of this process accessible. The harder know-how is translating judgment into evaluations and validating it through experiments. Gavrilescu describes a route from a maker’s preferences into environments and evaluations, and potentially into model weights. But improving the surrounding system also requires interpreting the artifacts a worker produces and deciding which changes are worth trying.
Experiments then test the maker’s judgment itself. Offline evaluations can establish that an agent satisfies its creator’s criteria. Production use asks whether customers appreciate those criteria. An improvement loop needs both: a way to reproduce the intended judgment and a way to discover whether that judgment creates value for users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A sourcing agent learns that prominence is the wrong signal
The practical example begins with a talent-sourcing agent: web search, LinkedIn, subagents and a system instruction describing its recruiter role. Recruiting makes taste visible because different people leading the process have different ideas of a good candidate. The baseline can search competently while choosing a strategy its owner dislikes.
Its execution traces reveal the problem: the agent repeatedly reaches out to big-tech employees, while this recruiter wants to find hidden gems. From the agent’s perspective, a prominent candidate looks like an obvious good choice. From the recruiter’s perspective, that choice misses the purpose of sourcing. Grouping similar behaviors and user frustrations into a pattern turns a vague complaint into a specific target for improvement.
The next step is a judge that reads a trajectory—the sequence of actions in a run—and identifies the unwanted pattern. Gavrilescu’s concrete question is whether the sourcing agent contacted Google employees instead of looking for hidden gems on GitHub. That gives subsequent runs a repeatable check. The criterion expresses this recruiter’s strategy, rather than a universal rule that big-tech candidates are unsuitable.
The proposed division of labor puts a human in charge of calibrating that criterion. Does the recruiter agree that the agent should favor hidden gems over big-tech outreach? Agents can build the evaluations and put the judgment into code; the person checks whether they are encoding the intended preference. This is where the observed failure becomes an explicit product decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the recipe with users before promoting it
A calibrated judge makes it possible to create recipe candidates: changes to the system that can be compared with the existing version. In the sourcing example, the intended observable change is a move away from big-tech outreach toward discovering hidden gems. Offline evaluations test whether a candidate follows that preference. Production experiments test whether users value the resulting service. The example describes this improvement process without reporting an experimental result.
How does a recurring sourcing mistake become the next recipe version? The diagram follows the example through two distinct judgments: the human confirms the desired recruiting strategy, then users test its value in production. The path makes clear why a candidate that satisfies the maker still needs an experiment before promotion.
Gavrilescu proposes A/B testing, with a multi-arm bandit as another possible experimental approach, to check whether customers appreciate the maker’s taste. Once that preference is validated with users, the candidate becomes the next version of the agent recipe. Repeating the process lets the system reproduce an increasingly well-defined service for other people.
The ending raises the ambition of the second loop: automate more of the maker’s role as a higher-level judge, so improvement agents can apply the same judgment to workers headed for production. Gavrilescu sees the continued accumulation of that judgment, together with workers that self-verify and cooperate, as a defensible basis for a vertical AI company.
That company still has to earn the customer’s choice. The closing economic test is whether the work is valuable and whether its price makes switching from a general-purpose coding agent worthwhile. Introspection’s invitation to vertical SaaS companies and agent labs is to build these research loops around their own products: learn what good means, reproduce it and deliver it at viable cost.
From sourcing traces to a new recipe
Read the diagram as text
- Baseline sourcing traces. Repeated outreach to big-tech employees
- Identify the mismatch. The recruiter wants hidden gems
- Calibrate a judge. Human confirms the strategy; agents encode the evaluation
- Evaluate a recipe candidate. Test whether it follows the intended sourcing preference
- Experiment with users. Test whether users value the preference
- Promote the next recipe. Advance after user validation, then repeat
- Baseline sourcing traces → Identify the mismatch: Cluster recurring behavior.
- Identify the mismatch → Calibrate a judge: Turn the pattern into a criterion.
- Calibrate a judge → Evaluate a recipe candidate: Create changes to test.
- Evaluate a recipe candidate → Experiment with users: Take the preference into production.
- Experiment with users → Promote the next recipe: Promote after users validate the value.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
3 slide groups. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 27 additional groups were omitted from this pass.
- 0:12
Hello everyone. How's everyone doing?
- 0:17
Are you guys ready for some more loops?
- 0:21
Yeah. My name is Roland. My co-founder
- 0:24
and I were in this mythical place called
- 0:27
XAI working hard on agent infra and we
- 0:30
realized there's something new that has
- 0:33
to be done in a standalone way. So we
- 0:35
left a few months ago to really figure
- 0:38
out okay what's the next stage of how we
- 0:41
should deploy these always on longunning
- 0:43
horizon tasks. Um and I'm happy to
- 0:47
announce we have a few findings that we
- 0:49
would like to present you. Um, and this
- 0:52
talk it's all about um, how you should
- 0:55
productize these ideas in ways that can
- 0:58
scale with your customers. Um, you've
- 1:02
heard a lot about auto research. Um, we
- 1:05
think there's a blueprint for 2026 and
- 1:07
beyond on how you should think about
- 1:08
auto research and it really comes down
- 1:12
to three ideas.
- 1:14
Let's go through the first one. The loop
- 1:17
is the product.
- 1:21
We're all familiar with this. We've
- 1:23
started with everything goes down to RL
- 1:26
chief for models and how you should
- 1:28
train the model to become better and
- 1:30
better reasoning. We then quickly moved
- 1:32
to harnesses and how the model is a
- 1:35
commodity and it's all about the
- 1:36
harness. And now we're talking about
- 1:38
loops and how you should build these
- 1:40
loops uh and not touch code anymore. But
- 1:43
what does it really mean and why is
- 1:45
everyone saying that?
Clawdbot bought me a car
AJ Stuyvenberg’s car-buying story illustrates an early Clawdbot loop: it talked to Reddit users and dealers to seek a better price. Clawdbot is now known as OpenClaw.
Text in image
PART I Remember when... Clawdbot bought me a car Outsourcing the painful aspects of a car purchase to AI was refreshingly nice, and sold me on the vision of Clawdbot (now OpenClaw) January 24, 2026 AJ Stuyvenberg Introspection
- 1:47
Do you guys remember clawbot?
- 1:50
That was the original um original name
- 1:53
of what is now now now known as open
- 1:55
claw. And this guy AJ built the first
- 2:00
loop around cloudbot.
- 2:03
What he did was to find a way to talk to
- 2:06
dealers and talk to Reddit users to get
- 2:09
bigger discounts on a car. He followed
- 2:12
these four steps. Um and is really open
- 2:15
call the one that did it. Go on Reddit,
- 2:18
find prices, find inventory, talk to the
- 2:22
dealers,
- 2:25
put dealers head-to-head, and try to
- 2:27
figure out how to make them outbid each
- 2:29
other,
- 2:31
have a verifiable way to know when the
- 2:33
price is right, and then lock in, get
- 2:38
the car, and it worked. Um, probably
- 2:41
this was when all the Mac minis were uh
- 2:44
selling off the shelves, but this was
- 2:46
the first real example of loop is the
- 2:49
product and something that probably
- 2:51
should be a startup at this point. Um,
- 2:53
but we've seen how this became a recipe
- 2:57
for everyone to build loops. But let's
- 2:59
take a step back. Why are we here? Um,
- 3:03
we really think models have been trained
- 3:05
with this loop in mind. And it comes
- 3:07
from this idea of udala loops. It's a
- 3:10
terminology coined back in 1970s by the
- 3:15
US air force and is the idea of these um
- 3:19
jet fighters how to react in fast-paced
- 3:22
environments.
- 3:24
If you think of models calling tools and
- 3:27
taking observations, it's it's what
- 3:29
we've been trained on uh as humans but
- 3:32
also as as agents. Now, now what happens
- 3:35
when you put strong signals and
- 3:37
verifiable work uh at the other ends?
- 3:41
You get to these workers or cloud code
- 3:44
agents. Um and and what matters here is
- 3:48
the quality of the signal determines the
- 3:51
uh success rate of the loop and the uh
- 3:55
quality of the verifi verifier um um is
- 3:59
able to calibrate if that success is
- 4:02
actually correct or not. But there's
- 4:05
another loop here. Um what happens when
- 4:07
you take that and feed it back into the
- 4:09
signal? And this is what looping around
- 4:12
is all about is how do you generate
- 4:15
these artifacts at the end of the first
- 4:17
loop to then run a second loop on and
- 4:20
have a way to continuously improve. And
- 4:24
this goes to my second point. System
- 4:26
dillation is the mode and is really the
- 4:30
ability to understand what went well and
- 4:34
wrong in the first loop and know how to
- 4:37
process that in the second one.
- 4:40
So how do we tune these AI systems? Each
- 4:43
loop generates useful information around
- 4:45
harnesses, profiles, evals,
- 4:48
models, resources, tools, and the
- 4:51
environment. What you really want is
- 4:55
to have a way to keep this portable, to
- 4:57
have a way to version this and to evolve
- 4:59
it over time. If you think about data
- 5:03
recipes in research, this is how RL
- 5:07
started to work really well. you
- 5:09
understood the recipes and how to
- 5:11
continuously change the recipe to combat
- 5:14
some of the behaviors that may happen
- 5:15
around hallucinations around reward
- 5:17
hacking and then you get to a stack
- 5:20
which is your final data recipe. We
- 5:22
don't have that for harnesses. We don't
- 5:24
have that for like AI systems in the
- 5:26
general term. So we thought there's
- 5:28
space for something like that. something
- 5:31
that contains the evils and contains the
- 5:33
tweaks and the human judgment and all
- 5:36
these things that are not predetermined
- 5:37
at the beginning but they're defined as
- 5:40
you learn more about your agent acting
- 5:42
in in in the environment.
- 5:47
We think recipes can be applied to this
- 5:49
and we should use the same name. So an
- 5:52
agent recipe is really something that
- 5:54
enables you to create reproducible
- 5:56
frontier AI systems. It's something that
- 5:59
allows you to have a mode that keeps
- 6:02
getting better over time, which is not
- 6:04
tied to any platform or any provider.
- 6:07
It's something that you control lives in
- 6:10
your company and is agnostic to the
- 6:13
models and providers you use. And loops
- 6:16
should focus on this. Loops should be
- 6:18
the way you distill these systems into
- 6:20
recipes.
- 6:21
Failure patterns should become judges
- 6:23
and evals. Repeated behavior should
- 6:26
become skills and prompts. user
- 6:27
frustration, extensions and memories to
- 6:29
your harness and so on. You we're all
- 6:32
familiar with this, but we didn't have
- 6:34
the the the right like terminology of
- 6:36
how we should think about it and how we
- 6:38
should define it. And we think recipes
- 6:40
is a way to put everything together into
- 6:45
a git repo and treat it as your ongoing
- 6:50
um strategy for for uh building these
- 6:53
self-improving systems. So we are
- 6:56
introspection but you can think of
- 6:58
introspection as the way you generate
- 6:59
these recipes. So they're recipes for
- 7:02
introspecting on your on your system. We
- 7:05
wanted to build something that is
- 7:07
portable and provider agnostic. So we
- 7:09
built our um approach to recipes on the
- 7:13
pi harness and on harbor for evals.
- 7:17
We baked it into uh git repos so uh
- 7:21
everything could be versioned and agents
- 7:24
could have a way to continuously track
- 7:26
how this change and why and is meant to
- 7:29
be owned by you but managed by your
- 7:30
agents. And this is how products should
- 7:33
really be built going forward. It's
- 7:34
something that treats the owner as the
- 7:38
um almost like the the the higher taste
- 7:42
um personality in the room. But agents
- 7:44
should try to calibrate themselves to to
- 7:47
the taste of the of the maker. So we
- 7:50
think recipes should be basically
- 7:52
encoding the taste of the makers into
- 7:54
how you build these agents. And if I
- 7:57
want to use someone else's recipe, I
- 7:59
should be able to also bring that taste.
- 8:01
It's not just the harness, it's not just
- 8:03
the model, is how did you arrive at this
- 8:05
particular recipe and why? And that's
- 8:08
kind of like what uh what is behind uh
- 8:12
reproducible
- 8:13
um uh products and services around
- 8:16
agents. Um we have an early release of
- 8:19
recipes is called pi. Recipes. It's very
- 8:23
similar to what skills uh used to be in
- 8:25
2025 but is going a step forward. And
- 8:28
this is what do I need to have a
- 8:30
frontier agent is everything about how
- 8:33
do I codify paste into evals? How do I
- 8:36
run evals? How do we have the loops to
- 8:38
continuously improve those evals over
- 8:40
time? How do we process signals and know
- 8:42
what are the right signals to to use? Um
- 8:45
what are the right tools to work with
- 8:47
certain models? How do I have different
- 8:49
profiles of the harness to work with
- 8:51
different models? Um and everything in
- 8:53
between. So have a look at what we've
- 8:56
been building here. It's still early uh
- 8:59
but hopefully it's useful enough for you
- 9:01
guys to to get going. And we feel this
- 9:03
is going to grow into something that um
- 9:05
really allows you to to use uh different
- 9:08
um almost like different the to to be
- 9:11
able to use the taste of of different
- 9:13
makers as recipes for your agent.
- 9:18
And finally, the last point is valued
- 9:20
work per watt. And why is this the score
- 9:22
to really optimize for? Think of how um
- 9:26
cursor and cognition went from building
- 9:28
the best product to then building the
- 9:31
best evals for the product and finally
- 9:33
building the best models based on the
- 9:36
previous two artifacts. We think this is
- 9:38
like the recipe for everything going
- 9:40
forward. Um code was the first domain
- 9:43
where this um was successful. Um
- 9:46
everything beyond customer support,
- 9:48
legal research um everything is going to
- 9:51
come down to this idea. How much value
- 9:53
am I getting per watt? Um, how do I
- 9:56
measure the value is the first step and
- 9:59
how do I know I'm getting a good deal on
- 10:01
that value is the second. And maybe this
- 10:04
makes it a bit more clear. We've all
- 10:07
started from a base harness and a base
- 10:11
set of evals and we went to go to the
- 10:13
frontier. Um, and you only go through
- 10:15
that by running these systems in prod.
- 10:17
There's no way you you know what
- 10:19
frontier is before you uh you start. Um
- 10:22
but the the the last step here which is
- 10:25
what is requiring a lot of research um
- 10:28
is okay once you've reached frontier how
- 10:30
do we make this um uh economically
- 10:34
viable which is how do we not spend more
- 10:37
than than uh we need for generating this
- 10:39
amount of value um and we think we have
- 10:42
the building blocks now to make this
- 10:45
accessible and pretty efficient in the
- 10:47
sense of you've seen all these
- 10:49
fine-tuning APIs all the infrastructure
- 10:51
that has been abstracted away for you to
- 10:54
do do this process is just the knowhow
- 10:56
that uh is not there yet and this is
- 10:59
what we we we hope we can like push for
- 11:01
the knowhow for knowing how to codify
- 11:04
taste into evals and how to validate
- 11:06
that in experiments um and you you've
- 11:09
you've heard a lot about evals in
- 11:10
experiments before but you didn't really
- 11:12
think of them of like what are they is
- 11:14
it's not just tests is is really what is
- 11:17
the taste of the creator that agents
- 11:20
should be able to reproduce and
- 11:22
self-improve around. And no one has
- 11:25
thought of how do I make this as
- 11:27
portable enough? How how do I make my
- 11:29
taste as an artist or as a software
- 11:33
developer um something that anyone can
- 11:35
download in their brain and be able to
- 11:37
be a onetoone replica to me? And this is
- 11:39
kind of like what RL is is is about now
- 11:42
is how do we uh turn these um taste
- 11:47
makers into uh environments and evals
- 11:50
around them so then we can move them
- 11:52
into the weights. But um there's more
- 11:54
than that. Um you can think of the
- 11:56
worker as the inner loop and it
- 11:58
generates all these artifacts. But how
- 12:00
you look at the artifacts and know what
- 12:03
to change is the taste. Uh and this is
- 12:06
what creates candidates of what you
- 12:09
should change and how you should adapt
- 12:10
based on that. And experiments is what
- 12:13
how you self-calibrate that okay my
- 12:15
taste is actually validated in
- 12:17
production with users and we make sure
- 12:20
that not only the maker is happy through
- 12:22
the um offline evals but the end users
- 12:25
are happy as well and they agree with
- 12:27
what we consider good.
- 12:30
Let's go through a practical example of
- 12:32
how this works.
- 12:36
Let's take a baseline um agent which
- 12:38
could be a talent sourcing agent. Um and
- 12:41
this is a very classical case of
- 12:45
everyone is doing recruiting differently
- 12:47
and is very much
- 12:49
about not what is good recruiting but
- 12:53
who is leading that recruiting that
- 12:55
considers recruiting as good. So in this
- 12:57
case we're starting with something
- 12:58
pretty simple. um a a bunch of tools,
- 13:01
web search, LinkedIn, uh a bunch of sub
- 13:04
aents that have been pre-popularized by
- 13:07
harnesses like codeex and cloud code and
- 13:09
uh system instruction which is about
- 13:11
your recruiter.
- 13:14
First step is really understand the
- 13:16
signals. So you can think of patterns as
- 13:20
being a way to look at the traces,
- 13:22
extract some common um behaviors or
- 13:25
common user frustrations and turn them
- 13:28
into like a cluster. So let's say this
- 13:30
idea of uh the agent is going uh and
- 13:33
reaching out to a lot of big tech
- 13:35
employees. As a recruiter, you don't
- 13:37
really want that. You want to find
- 13:38
hidden gems. You don't want to try to
- 13:40
hire John Carmarmac. But an agent would
- 13:44
think that's, oh, John Carmarmac is
- 13:45
great. why would I not reach out to him?
- 13:47
Um, so so this is a behavior that you
- 13:50
you'd never think of codifying, but you
- 13:53
discover the agent tends to do that. Um,
- 13:57
patterns is how you discover these
- 13:58
signals and inform you what you should
- 14:01
do next.
- 14:03
um calibration judges and evals is how
- 14:07
we used to think about how do we codify
- 14:10
these these behaviors into um something
- 14:13
that can try to uh apply the same
- 14:15
judgment across traces and across uh
- 14:17
execution. So let's say we we build an
- 14:20
agent that looks at a trajectory and um
- 14:24
identifies exactly that pattern. Hey,
- 14:26
did did this agent reach out to Google
- 14:28
employees instead of trying to uh find
- 14:31
hidden gems on GitHub? Um, and the
- 14:34
calibration bit and the eval generation
- 14:36
bit is not that hard. It it it should be
- 14:39
doable by agents to build. You just need
- 14:41
a human in the loop to say, hey, um,
- 14:43
this is the approach we're taking. Do
- 14:45
you agree with this
- 14:48
judgment? Do you really agree that we
- 14:50
should look more towards hidden gems
- 14:52
rather than reach out to um um big tech
- 14:56
employees? And that's about it. You
- 14:58
don't need the human to actually build
- 15:00
the evals. You need them to calibrate
- 15:02
the evals. And agents should be the ones
- 15:05
that really take the the the taste of
- 15:07
the maker and and put them in into code.
- 15:09
Once you have this, it's pretty easy to
- 15:12
create recipe candidates. And this
Recipe candidates
The candidate patch adds a hidden-gem scouting skill and favors public proof of work over employer brand. For each prospect, it captures an artifact, role fit, and a plausible reply path.
Text in image
candidate patch talent-search-recipe@r3-hidden-gem recipes/talent-search/agents/research.yaml 11 @@ 11 + skills: 12 + - hidden-gem-scout 21 @@ system_instructions: 22 mode: append 23 content: | 24 - Find the strongest matching profiles and summarize why they fit. 24 + Prefer under-noticed engineers with public proof of work: 25 + maintainers, issue authors, benchmark contributors, niche talks. 26 + Do not advance candidates mainly because of employer brand. recipes/talent-search/skills/hidden-gem-scout/SKILL.md 1 + # Hidden Gem Scout 2 + Look for evidence before prestige. 3 + 4 + For each prospect, capture: 5 + - artifact: repo, issue, benchmark, talk, design note 6 + - fit: why the work maps to this role 7 + - reply_path: why they might plausibly answer Interface text: Baseline Patterns discovered Calibration and evals Recipe candidates A/B tests Promote View Diff .introspection talent-search-… agents agent.yaml research.yaml M outreach.yaml extensions search-tools.ts skills hidden-gem-s… A SKILL.md A SYSTEM.md package.json … Introspection- 15:13
should be the the diffs that you really
- 15:15
want to taste. Um, and
- 15:19
you can have a pretty good offline evil
- 15:21
set around this, but the the the test
- 15:23
here is when you go to prod. So, do the
- 15:27
end user agree with your taste of not
- 15:30
hitting up um big tech uh employees,
- 15:34
right? And this is kind of like what you
- 15:36
want is you build a product that really
- 15:38
emphasizes your taste and then you you
- 15:40
make sure that your users appreciate and
- 15:43
value that taste. and AB tests have been
- 15:46
a way to to to make sure that that's the
- 15:48
case. Um so with a multi-arm banded um
- 15:51
scenario for example you you'd be able
- 15:53
to do that pretty well. So once you
- 15:55
validate okay I have great taste and my
- 15:57
users believe uh I have great taste as
Promote the recipe
After users validate the agent’s taste, promotion advances the recipe. This research recipe encodes a preference for public proof of work over employer brand.
Text in image
Baseline Patterns discovered Calibration and evals Recipe candidates A/B tests Promote name: research description: Candidate research and evidence collection. model: name: openai/gpt-5.5 reasoning_effort: medium tools: - WebSearch - WebFetch - mcp:github/search_repositories - mcp:linkedin/search_people skills: - hidden-gem-scout system_instructions: mode: append content: | # Research Bar Prefer under-noticed engineers with public proof of work: maintainers, issue authors, benchmark contributors, niche talks. Do not advance candidates mainly because of employer brand. ... Introspection Interface text: View Recipe .introspection talent-search-... agents agent.yaml research.yaml outreach.yaml extensions search-tools.ts skills hidden-gem-sco... SKILL.md SYSTEM.md package.json- 15:59
well that's when you promote and that's
- 16:01
kind of when you go to to the next
- 16:03
version of an agent recipe. The secret
- 16:06
is you keep doing this over and over
- 16:08
again and you know how to continuously
- 16:10
codify your taste and your um what what
- 16:13
what good is to you into an agent that
- 16:16
can reproduce the same service or
- 16:18
product uh for other people and they
- 16:20
also agree you have great taste and you
- 16:23
have great execution. And this is really
- 16:24
kind of like the the secret of building
- 16:26
good loops is okay can can someone
- 16:28
iterate on my um system in a way as uh
- 16:33
you know um a good example here is like
- 16:35
Miranda from the Delor product right
- 16:38
what would Miranda do uh in certain
- 16:40
cases and you kind of want to codify
- 16:42
that that thinking into like agents that
- 16:44
can do the same stuff at a higher level.
- 16:48
So the takeaways are this. Um the loop
- 16:50
is the product. You try to automate
- 16:51
yourself as the u as a um higher level
- 16:56
judge and you want to make sure your
- 16:59
second loop agents are able to apply the
- 17:02
same judgment to to the agents you're
- 17:04
trying to to to push to prod. Second bit
- 17:07
system dissolation is the mode. So, how
- 17:09
do you continuously inject that taste
- 17:11
into these uh workers and they how how
- 17:13
they continuously self-verify and work
- 17:15
together is uh the biggest thing that
- 17:18
you should focus on and the faster you
- 17:20
do it uh the the the faster you you
- 17:23
build a defensible
- 17:25
um approach to to becoming a vertical AI
- 17:27
company. And finally, valued work per
- 17:30
watt is how you should measure um am I
- 17:33
making progress or not. So first make
- 17:36
sure that uh the the the work you're
- 17:38
generating is valuable. Second make sure
- 17:40
that the economics makes sense and the
- 17:44
um the the difference in price is is
- 17:45
basically what um people would would
- 17:48
switch away from cloud code to to
- 17:50
something you provide.
- 17:53
We've been thinking a lot about these
- 17:54
ideas and we're building some very
- 17:56
interesting products around how to
- 17:57
deploy this in production. We'd love to
- 17:59
hear from you. would love to get um to
- 18:02
to understand more about how how certain
- 18:05
um vertical SAS companies are are
- 18:07
looking to go to prod with um or how
- 18:10
agent labs have been thinking about this
- 18:13
idea of um um creating these like auto
- 18:16
research uh labs around their their own
- 18:19
products. um get in touch. Uh we're
- 18:21
gonna be around the block for for
- 18:23
chatting more about this. And thank you
- 18:25
very much.