AI Engineer Summit 2025

Building Self-Coding Agents

Colin Flaherty· Founding Researcher, Augment Code17:23

Loading slides…

Read the talk

Building an Agent That Helps Write Its Own Code

Augment’s self-coding agent connects repository context, tools, human corrections, and test feedback to implement integrations, diagnose bottlenecks, and improve its own code.

From a talk by Colin Flaherty

Before you start: Familiarity with repository workflows, subprocesses, unit tests, and basic concurrency will help you follow the examples.

What does it mean for an agent to build itself?

How much of a coding agent can the agent itself help implement? At Augment Code, Colin Flaherty’s team explored that question while building tools for software engineering organizations. The backdrop was a rapid change in interaction: autocomplete in 2023, broader adoption of chat in 2024, and, in Flaherty’s 2025 forecast, agents becoming the next major focus. The team had spent the preceding few months building its own agent. Flaherty reports that over 90% of its approximately 20,000-line codebase was written by the agent with human supervision. Here, self-coding means supervised development of the agent’s software, not autonomous retraining of its foundation model.

The first concrete example is adding integrations. To work within an engineering organization, the agent needed access to Slack, Linear, Jira, Notion, Google Search, and the repository itself. Humans implemented the first few integrations, giving the agent examples of where integrations belonged and which interfaces they followed. Asked to add Google Search, it could inspect those patterns, locate the appropriate file, and implement another integration.

Slide titled “Implementing core features” with a lines-of-code chart and the annotation “Agent writes its own 3rd party integrations.”Open full source frame
The agent writes its own third-party integrations.

That new capability then helped build the next one. The foundation model did not know the Linear API, so the agent used the Google Search integration it had previously written to find documentation and implement the Linear integration. The useful feedback loop was concrete: an implemented tool expanded what information the agent could obtain for subsequent development.

The team also asked it to add unit tests for Google Search. Writing the tests required more than file editing: the agent needed tools to start subprocesses, interact with them, read their output, and avoid hanging indefinitely if a generated test entered an infinite loop. Execution tools turn generated tests into usable feedback.

0:170:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Using the agent to diagnose its own slowness

Feature generation and test generation were familiar demonstrations. Optimization posed a different question: could the agent investigate behavior before deciding what to change? When the team noticed that its agent was slow, it asked the agent to profile itself. The investigation followed a short loop:

  1. Add print statements to its own codebase.
  2. Run subcopies of itself.
  3. Inspect the resulting output.
  4. Identify repository file loading and hashing that happened synchronously.
  5. Introduce a process pool and add a stress test to check the change.

The agent used its editing and execution tools to move from an observed symptom to a specific implementation change. Flaherty describes the optimization but supplies no measured speedup.

Slide titled “Optimizing Itself” shows lines of code against days, with a rising green line and the annotation “Agent profiles itself, adds optimizations.”Open full source frame
The agent profiles itself and adds optimizations.
3:023:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:02 · section reference included

Instrument, execute, clarify, and remember

The walkthrough begins with deliberately simple tasks that the agent handles reliably. Asked whether it can search Google, the planning agent finds a Google Search tool, sends the query tests, and reports seeing the first ten results. This separates two responsibilities: the agent decides what to do, while tools let it act on external services and repository files.

The next request asks it to instrument its Google Search tool with logs and generate an example. Repository retrieval resolves that natural-language request to services/integrations/thirdparty/google_search_tool.py. The file-editing tool adds print statements there. The agent then runs a subcopy of itself to exercise the instrumented integration and obtain the requested output.

Execution exposes a missing piece of context: the agent believes Google credentials are unavailable. Instead of continuing with an unsupported assumption, it invokes a clarification tool and offers two alternatives—stub the Google API or guide the user through credential setup. Flaherty supplies a third answer: credentials already exist at .augment/googleapi.json; the agent simply missed them.

The agent then uses a memory tool to retain the credential location for future work. This is a concrete form of learning from interaction: a user correction becomes retrievable information, rather than a change to model weights. With that context, the agent produces logs for the query Python programming language, including returned domains python.org and wikipedia.org. The full task combines search, repository retrieval, editing, execution, clarification, and memory. No single tool performs the whole job; the planning agent connects them as new information arrives.

3:574:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:57 · section reference included

The infrastructure behind the tool loop

Augment did not begin by building an agent. Its earlier completion and chat work concentrated on a scalable enterprise context engine, alongside interface design that made human collaboration practical. Those investments supplied the foundation for building the agent in a couple of months. The clarification step in the logging demonstration illustrates why both matter: retrieval helps find relevant information, while the interaction design gives the user a way to repair missing context.

Flaherty identifies three essential capabilities:

  • Context access: retrieve relevant information from the codebase and organizational sources such as Slack.
  • Foundation-model reasoning: use a capable model to plan and decide how to act on that information.
  • Code execution: safely run commands within a customer’s environment and observe what happens.

Context informs an action, reasoning selects it, and execution supplies evidence for the next decision.

6:517:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:51 · section reference included

Measure capability by complexity

A polished demonstration of an agent generating an entire website can suggest senior-engineer, or L5, capability. But professional development rarely starts from an empty repository. Existing systems contain conventions, dependencies, and messy environments that a greenfield demonstration can avoid. Agents can be useful in that setting without yet matching the broad competence implied by a senior engineering role.

That distinction changes how to organize the capability roadmap. Rather than assuming an agent will first solve back-end work, then front-end work, then testing, Augment found it more useful to think about levels of task complexity across domains. A general-purpose agent can handle some front-end, back-end, security, and testing work, with improvements raising its capability across those areas together.

Human job categories also encourage misleading expectations about strengths and weaknesses. Flaherty gives the example of an agent that struggles with mathematics yet implements a front-end feature very quickly. A weakness in one task does not imply the same cluster of weaknesses that it might suggest in a human engineer. Evaluate the work the agent can actually perform, rather than assigning it a human-shaped skill profile.

7:548:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:54 · section reference included

Teach organizational conventions through retrieval

An unfamiliar development tool exposes the limits of relying on model knowledge alone. Flaherty asks the agent to create a stack of two pull requests for a new reasoning module using Graphite. The model does not know how Graphite works, so it substitutes Git and invokes git checkout. That produces an action, but not the requested workflow.

Waiting for a foundation-model provider to retrain on the organization’s tools would not solve the immediate problem. Instead, the team adds a searchable knowledge base. A Markdown document explains Graphite, common commands such as gt create, and actions to avoid. Other documents describe the tool stack, test procedures, and style guide. The agent can dynamically retrieve these documents when it encounters something unfamiliar.

With that documentation available, the same request leads the agent to read about Graphite and invoke gt create. The demonstrated improvement is selecting the intended tool after retrieval. In the current Graphite CLI, gt create creates a branch and commit; gt submit publishes pull requests, with --stack available for a stack. The visible command invocation therefore does not establish that both requested pull requests were completed.

Projected terminal example asks for two stacked PRs using Graphite, then displays “Reading about Graphite” and a tool invocation beginning with “gt create.”Open full source frame
The Graphite example shows knowledge retrieval followed by a command invocation.

Onboarding is part of agent engineering. A new human hire would receive explanations and documents, not just instructions to stare at the repository until everything made sense. An agent likewise benefits from being able to ask questions, with the resulting answers feeding back into organizational documentation.

9:489:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:48 · section reference included

Cheaper implementation expands what is worth trying

The integrations also changed the team’s product decisions. Initially, it was unclear which integrations, such as Linear or Slack, deserved priority. Ordinarily, the team would make educated guesses, implement a few, and learn from them. Agent assistance made it possible to build the integrations together, reducing the pressure to choose their implementation order. As implementation becomes cheaper, more ideas become practical to explore, and product insight and design can become the limiting factors.

Repository awareness lowers the cost of specifying that work, too. The earlier logging request did not name a file or give precise editing instructions. The agent resolved the intent to the appropriate integration through its knowledge of the codebase. The user could describe the desired change at the level of an engineering task.

The same context supports work beyond code editing. In another example, the team asks the agent to inspect recent pull requests and draft an announcement, which the team then posts to Slack. The announcement describes new CLI-agent tools, including Slack notifications and Linear integrations. The agent also creates the plot of interactive-agent lines of code against date shown earlier in the talk. These tasks draw on different sources and operations: repository changes, communication context, and code history.

The broader point is that context sources can complement one another. Understanding a repository helps identify what changed; access to organizational communication makes those changes useful in another workflow. Flaherty describes combined codebase and Slack access as four times as useful as either alone, an illustration of complementary context rather than a measured multiplier.

11:4511:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:45 · section reference included

A write lock can still lose updates

The testing example exposes a more subtle failure than choosing the wrong tool. Multiple agents store information for their runs in a shared cache location. The agent-generated save function locks around the JSON dump, so concurrent agents do not write simultaneously and cause an obvious write failure. But it does not read the current shared cache before saving. Each writer can therefore overwrite information another agent has already saved. Serializing writes does not make stale state current.

Consider a small illustration of that failure. Agents A and B both start with an empty cache. A adds {"a": "ready"} locally and saves it. B adds {"b": "ready"} to its own still-empty snapshot, then saves after A releases the lock. The writes happen one at a time, yet the final shared cache contains only B’s entry. The vulnerable Python operation has this shape:

python

import json


def save_cache(path, local_cache, lock):
    with lock:
        with open(path, "w", encoding="utf-8") as cache_file:
            json.dump(local_cache, cache_file)

For this example, the critical section would need to protect reading the current shared state, incorporating the new entry, and writing the result. A lock around only the final dump cannot preserve A’s entry when B’s input never contained it.

A serialized save overwrites another agent’s entry

Constructed example: Agent labels A and B, an initially empty cache, the keys a and b, the value ready, and the exact save order are teaching values illustrating the reported lost-update mechanism.

Agent B’s stale local cache — unchanged
{"b": "ready"}

Operation: Agent B acquires the lock after Agent A finishes, dumps its local cache to the shared file, and releases the lock.

Agent A’s local snapshot

Before: After A saves, before B saves
{"a": "ready"}
After: After B saves · Unchanged
{"a": "ready"}

Agent B’s local snapshot

Before: After A saves, before B saves
{"b": "ready"}
After: After B saves · Unchanged
{"b": "ready"}

Shared cache file

Before: After A saves, before B saves
{"a": "ready"}
After: After B saves · Changed
{"b": "ready"}
The lock prevents overlapping writes, but B still replaces shared state with a stale snapshot.
14:0114:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:01 · section reference included

Tests provide the feedback that autonomy needs

Unexpected runtime behavior exposed the cache bug. The team had no test for the relevant concurrency case, so the agent received no test signal that its implementation was wrong. A useful regression test for the illustrated case would give both agents the same initial snapshot, let each save a distinct entry, and assert that both entries survive. Testing only that a save completes or that the resulting JSON parses would miss the lost update. Caches and parallel execution need tests that check preservation of state, not just absence of exceptions.

Augment’s internal bug-fixing benchmark gives a second view of why feedback matters. Flaherty compares changing the foundation model with enabling repeated test-guided attempts:

ChangeReported score improvement
Foundation-model upgrade spanning about six months4%
Four iterations of proposing a fix, running tests, and using feedback20%

These are reported gains on the internal benchmark. The talk does not identify the models, baseline scores, or whether the percentages mean relative changes or percentage-point increases. The mechanism behind the larger reported gain is explicit: the agent can inspect test failures, revise its proposed fix, and try again.

Better tests enable more autonomy because they give the agent actionable evidence and give engineers a stronger basis for trusting the result. A test harness is therefore part of the agent’s problem-solving environment, not merely a final gate after it finishes writing code.

14:5415:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:54 · section reference included

Code remains; the engineering work changes

Flaherty closes by contrasting the agents’ recent usefulness with their earlier limitations. When agents help implement their own tools and improvements, development can compound: a new capability becomes available for the next task. Code still matters as a specification of the system, even as engineers spend less of their effort producing every line directly.

That shift makes strong test harnesses more valuable and poorly tested parts of a codebase more consequential. It also changes product development: cheaper implementation increases the importance of deciding what to build, gathering customer feedback quickly, and turning that feedback into insight. At the time of the talk, Augment’s agent release was still forthcoming. The working prototype already showed the engineering pattern behind it—context to understand the task, tools to act, human input to resolve gaps, and tests to expose mistakes.

15:5816:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:58 · section reference included

Resources

From the talk

Updates since the talk

  • The April 2025 announcement introduces Augment Agent, persistent Memories, terminal access and external integrations.

Transcript download
Full transcript

Loading transcript…