AI Engineer World's Fair 2026
From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer
Loading slides…
Read the talk
From 36% to 100%: How Self-Improving Agents Write Their Own Skills
Rafal Wilinski explains how Runlayer turns agent traces into reusable playbooks, distributes them through MCP, and uses the resulting feedback loop to preserve company knowledge across sessions and models.
From a talk by Rafal Wilinski
At a glance
Ideas worth remembering
Skills use progressive disclosure: a short description helps an agent find relevant knowledge, then the full procedure enters its context to guide the attempt.
MCP can distribute skills from a central server that selects procedures by query and role. Runlayer uses tools because MCP resource support is uneven.
Successful traces supply reusable routes; failed traces expose missing guardrails, dependencies and edge cases that can improve the procedure.
In the reported Chromium-on-AWS-Lambda evaluation using Sonnet 4.6, success rose from 9/25 runs (36%) without a skill to 25/25 (100%) with the distilled skill.
Company-wide learning requires more than distillation: candidate skills pass checks, enter a shared library and reach future agents through the MCP gateway.
Keep the solution after the session ends
A hard problem gets solved, the session closes, and the next person starts from scratch. The missing artifact is often the useful part: the debugging steps, rejected approaches and reasoning that made the solution possible. Rafal Wilinski, founding engineer at Runlayer and previously a leader of Zapier’s agent work, opens with a practical ambition: make every successful investigation leave a playbook the rest of the company can use.
That ambition extends beyond one agent remembering its own work. An engineer, salesperson or lawyer who cracks a difficult problem can improve the next attempt by sharing the procedure. Automating that handoff creates the possibility of company-wide learning: one agent’s discovery becomes another agent’s starting point.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Longer tasks make the starting trajectory more expensive
A skill is a playbook an agent reads when its knowledge becomes relevant. It uses progressive disclosure: the client or harness initially sees a short description; when the agent identifies a useful match, it reads the full skill into its working context. The procedure can then change which actions the agent takes. Better models, prompts and tools remain other ways to improve an agent, but skills offer a direct way to supply knowledge at the moment it is needed.
The stakes grow as models become capable of longer, multistep work. Wilinski points to a task-horizon benchmark measuring how long agents can work independently while retaining a chance of completing the task. That word—chance—matters. A longer reasoning window also lets an agent spend hours and hundreds of tool calls pursuing a wrong hypothesis. Without a useful starting procedure, greater persistence can make a mistake more expensive.
The most valuable skills often contain what Wilinski calls “the rituals that nobody wants rediscovered.” These are local details that general model capability does not automatically supply:
- Production rollback: the sequence of scripts and feature flags needed to reverse a change safely.
- Customer retention: the wording and approach appropriate for an enterprise customer close to churning.
Both examples depend on how a particular company works. Asking an agent to reconstruct that knowledge on every attempt adds exploration precisely where mistakes can be costly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Useful playbooks still need authors, access and trust
Skills have three separate adoption problems. Solving one does not automatically solve the others:
- Developer-oriented packaging: Markdown, JSON, CLI commands and Git repositories are familiar to developers. They make creating and sharing skills harder for people in marketing, HR, legal or finance.
- The cost of writing: everyone likes finding an existing playbook, but producing one requires time and deliberate reflection after the difficult work is finished.
- Uneven client support: a useful skill cannot guide an agent if the employee’s LLM client has no way to discover or load it.
Installation introduces another practical concern. Wilinski jokes that a skill installer can bring along prompt injections, a script requiring root permissions and somebody else’s API key. The joke comes with a concrete observation: Runlayer scans thousands of skills every day and encounters suspicious content. A playbook influences the agent’s decisions, and its package may also include executable material. Making installation easy therefore increases the importance of checking what is being installed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use MCP to deliver knowledge as well as capabilities
Skills and MCP serve complementary purposes. A skill explains how to do something; a tool supplies the capability to do it. Putting a command or API key in a Markdown file does not remove the need to connect agents to systems. MCP can also carry the playbook itself, giving knowledge a shared distribution channel alongside tools.
In this design, clients connect to a remote MCP server and search for relevant skills instead of installing their own local copies. The server selects what to return using the query and policy, including the requester’s assigned role. A central source can supply the appropriate version of a procedure and give an AI adoption lead one place to manage the knowledge being shared. Wilinski reports that Runlayer had used this approach internally for six months.
The implementation choice is about client reach. The Skills over MCP working group was developing an API, with the specification direction leaning toward MCP resources. Runlayer chose tools because resource support was mixed. The API was not final at the time of the recording, and practical reach still depends on the client’s MCP support. Within that constraint, distributing skills through tools lets more employees retrieve procedures through the clients they already use.
Central distribution addresses access and governance. It still leaves the authoring problem: somebody has to turn an investigation into a useful skill. The next step is to let completed agent work supply the material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn exploration into reusable procedural knowledge
Voyager, the 2023 Minecraft agent work, provides the model for this kind of learning. In Wilinski’s account, the agent explored and pursued goals while accumulating a skill library, without changing the underlying model weights. When it discovered how to do something, it saved a code recipe containing the tool calls that accomplished it.
A later goal—making a crafting table or a diamond pickaxe—could then retrieve the recipe rather than reconstruct the procedure. The accumulating library changes what the agent has available for its next attempt. This is the idea Runlayer carries into workplace tasks: exploration should produce reusable procedural knowledge instead of ending as a discarded trace.
Runlayer lets agents perform real work, then takes an interesting successful run containing multiple tool calls and asks a frontier LLM to distill a skill from its trace. The saved artifact is a procedure for future work. Authoring happens after the run, asynchronously, rather than depending on someone remembering to write documentation.
Failures supply a different kind of information. A successful run demonstrates a route that worked; a failed run can expose a missing guardrail, an unavailable library or an overlooked edge case. Both feed the distiller. Successes support the existing procedure, while failures provide reasons to revise it. Wilinski describes this update loop as autonomous, without a human in the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A Chromium-on-Lambda task moves from rediscovery to reuse
The concrete test was getting a Chromium fork to run inside AWS Lambda using Sonnet 4.6. The agent sometimes completed the task, but its initial success rate was 36%. Runlayer enabled its self-improvement mechanism. When one invocation solved the problem, the distiller generated a skill from that run and included it in all subsequent runs. Wilinski reports that the success rate rose to 100%. The slide below answers the immediate question: how many attempts sit behind those percentages? It labels the comparison a 25-run evaluation, showing 9/25 successes without the skill and 25/25 with it. The recording does not give the full evaluation protocol or contents of the skill, so this result describes the reported test rather than a general guarantee of reliability.
The comparison makes the observable change clear: occasional success became consistent success within the reported evaluation. Initially, separate invocations had to find a working route. One successful invocation supplied a trace; the distiller converted that trace into reusable instructions; later invocations received those instructions before attempting the task. The successful run stopped being an isolated event and became input to future behavior—“a developer with better attitude towards documentation,” as Wilinski puts it.
Why might the same change also reduce cost? A good skill narrows the search space. It steers the agent toward a known procedure, reducing the branches it explores and the tool calls it spends on rediscovery. Wilinski presents fewer tokens and lower spending as consequences of that mechanism; the numerical result here measures task success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share the learning loop across the company
There is another reason to retain procedures: access to frontier intelligence is rented. A provider can deprecate a model, and users may experience changes in its behavior. Wilinski’s response is to use capable models while they are available to discover and persist useful procedures. A catalog of how work gets done keeps something the company can carry forward when its model changes.
The organizational version combines three pieces already introduced: skills retain procedures, a central library stores them, and MCP distributes them. Runlayer groups similar runs, distills a candidate skill, and passes it through checks that include prompt injection and personally identifiable information. A skill that passes goes into the library and becomes available through the MCP gateway to clients and agents across the company.
Runs → candidate skill → safety checks → shared library → MCP clients. Each new attempt contributes traces, including failures that can reveal where the procedure needs to change. Checks happen before a skill becomes available to other agents.
The shared library is what extends learning beyond the original agent. When another employee’s agent encounters the same database restart or production rollback problem, it can retrieve the accumulated playbook. As more runs expose useful procedures and edge cases, the library can improve the starting point for subsequent work throughout the organization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The advantage is the knowledge accumulated in doing the work
The ending turns to competitive advantage. If competing companies can access the same models and reproduce similar product interfaces, where does differentiation come from? Wilinski proposes the accumulated learning loop: governed procedures drawn from a company’s own hard-won experience. That is a strategic argument about the value of internal knowledge, rather than a demonstrated barrier to competition.
The proposed advantage has a practical form. Each solved problem can raise the starting point for the next person; each useful failure can add a missing restriction or edge case. Retained procedures can reduce repeated exploration without changing the model and preserve useful knowledge through model transitions. The company’s particular way of working—its internal culture and identity—becomes something agents can retrieve and apply, rather than something lost when a session closes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Develops the complementary optimization mechanism: use diagnostic execution traces to revise prompts, programs and repository skills, then evaluate candidates and preserve useful alternatives.
Full transcript
Loading transcript…