AI Engineer Summit 2025

How Coding Agents Change Software Development Forever - Hailong Zhang

Loading slides…

Read the talk

How Coding Agents Change Software Development Forever

Hailong Zhang’s Gru case study shows how an asynchronous agent can turn code changes into tested pull requests, and what evaluation, context, and shared infrastructure make that workflow possible.

From a talk by Hailong Zhang

Before you start: Familiarity with Git pull requests and basic unit testing is helpful; no prior agent-building experience is required.

Which work should an agent take on?

If an AI agent takes over routine coding, debugging, testing, and operations, what remains the developer’s responsibility? Hailong Zhang begins with this division of work. His forecast leaves product design, architecture, and difficult issues with humans, while agents handle more of the routine implementation. The resulting workflow depends on human–agent collaboration: automating a task still leaves decisions about what to build and whether the result is right.

Workflow from product design and architecture design, labeled creative work handled by humans, to coding, debugging, testing and operating, labeled routine work mostly handled by AI.Open full source frame
Human and AI roles across the software development workflow.
0:320:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:32 · section reference included

Live assistance and asynchronous delivery

The first collaboration pattern is familiar from GitHub Copilot and Cursor: an assistant lives inside the IDE and works while the developer types. Zhang places the broader category of synchronous assistance around 2020, with maturation in 2023 and rapid growth in 2024; that chronology is not a launch date for each named product. In the talk’s 2025 setting, he describes this pattern as widely adopted.

The second pattern moves the agent into the surrounding workflow. Zhang dates the emergence of this asynchronous category to 2024. A GitHub-style bot receives a manual or automatic trigger, completes a task without continuous human attention, and returns a deliverable.

CollaborationWhere it worksHuman involvementResult
SynchronousInside the IDEDeveloper and assistant work togetherAssistance during editing
AsynchronousInside a repository workflowDeveloper delegates and later reviewsA completed deliverable

Both patterns remain useful because they serve different moments in development. An engineer may want immediate assistance while editing and separately delegate a task that can run to completion.

At repository scale, this becomes a collection of small agents operating at different stages: writing unit tests, fixing bugs, producing documentation, submitting code reviews, and handling releases. Each agent takes responsibility for a bounded piece of work, freeing human attention for the creative parts of the project.

1:111:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:11 · section reference included

Fast generation makes tests more valuable

Unit testing provides a concrete place to start. In Cursor, accepting a suggestion with Tab can generate multiple lines in multiple parts of a file. The developer can move quickly while overlooking individual changes. Zhang’s concern is the gap between generation speed and attention: code arrives faster, but bugs can arrive with it. Tests provide one way to catch those issues.

Slide titled “Unit Tests are More Important in AI Era,” with a unit-test meme beside a code editor overlaid with three Tab labels and text warning about overlooked issues.Open full source frame
Unit testing concerns alongside repeated Tab-based code generation.

Writing those tests still requires effort. Zhang points to developers’ reluctance to do that work as the motivation for Gru, an agent for generating and managing unit tests. The case study concerns the historical Test Gru workflow demonstrated in 2025.

2:553:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:55 · section reference included

From a code change to a test pull request

Gru’s unit of work begins with a repository change and ends with a reviewable pull request:

  1. Trigger the agent. Invoke Gru manually or let a developer’s pull request trigger it automatically.
  2. Inspect the change. Determine whether the changed code needs additional tests or changes to existing tests.
  3. Write and run the tests. Generate the test code and execute it to check that it works.
  4. Submit a test pull request. Include a summary of the tests and their coverage improvement.
  5. Ask for a human decision. The reviewer judges the tests and decides whether to merge them.

The screenshot Zhang presents is the deliverable from this process: a test pull request with information a reviewer can use to assess it. Autonomous task execution ends at a human merge decision.

For a small TypeScript/Vitest illustration of that review boundary, suppose a change makes an order eligible for free shipping when its subtotal reaches 50. A proposed test can make the boundary explicit:

typescript

import { describe, expect, it } from 'vitest';
import { qualifiesForFreeShipping } from './shipping';

describe('qualifiesForFreeShipping', () => {
  it.each([
    [49.99, false],
    [50, true],
    [50.01, true],
  ])('returns %s eligibility for subtotal %s', (subtotal, expected) => {
    expect(qualifiesForFreeShipping(subtotal)).toBe(expected);
  });
});

Running this proposed test checks the implementation against the encoded expectations. Reviewing it asks the further question: is 50 actually the intended inclusive threshold? That distinction explains why Gru’s workflow includes both test execution and human judgment before merging.

3:584:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:58 · section reference included

What production use shows

Zhang reports that over 50% of Gru’s pull requests in the displayed production GitHub list were accepted and merged by humans. He treats that level of acceptance as useful in practice while acknowledging substantial room for improvement. The talk does not establish a denominator, observation window, or model configuration for the figure, so its scope is the production example he presents.

Zhang also reports that Gru handles around 80% of the unit tests in its own repository, where the team uses it daily. This describes the agent’s share of unit-test work, not a code-coverage percentage or a pull-request acceptance rate.

By commit count, Zhang reports that Gru is the team’s leading contributor. Commit count measures repository activity separately from the quality or acceptance of the resulting tests. From that internal experience, he predicts that more agents will become repository contributors during 2025.

4:495:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:49 · section reference included

Start with a task that can be evaluated

Building such an agent begins with defining a clear, concrete, feasible problem. Unit testing gives the team a bounded task; software engineering as a whole is too broad to serve as the same kind of problem definition. The scope determines what the agent should do and what its evaluation must assess.

Zhang’s construction sequence puts evaluation before model and context work:

  1. Define the problem. Choose a task specific enough to solve.
  2. Build evaluation datasets and a harness. Establish the infrastructure for assessing candidate approaches.
  3. Work on models and context. Develop the components that perform the task.
  4. Orchestrate the components. Use a framework or Agent OS to bring them together.

This order gives model and context decisions an evaluation setting instead of leaving success defined only by a plausible-looking output.

5:505:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:50 · section reference included

Select models and build context for the task

Gru is designed to work across frontier models from OpenAI, Anthropic, DeepSeek, Google, and other providers. The team evaluates models in different scenarios to choose an appropriate model for each one. That choice can vary between stages of a single job: a job does not have to use one model throughout.

The team also adapts models. Zhang describes fine-tuning GPT-4o with human-labeled unit-test code to improve test generation. Scenario-based selection and fine-tuning address different decisions: which model to use for a stage, and how to adapt a model to the desired output.

Three diagrams show Gru connected to OpenAI, Anthropic and DeepSeek; scenarios paired with different models; and human-labeled unit-test code flowing into GPT-4o.Open full source frame
Gru’s model-agnostic, multi-model and fine-tuning approaches.

Context is also task-specific. For unit testing, Gru builds context around the programming language and testing framework, then gathers information from the repository environment. Relevant sources include GitHub issues, code reviews, commits, pull requests, README material, and the code itself. Gathering is followed by filtering and selection so the information fits into the model’s context. The mechanism is therefore more deliberate than simply passing the entire repository to an LLM.

6:366:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:36 · section reference included

Share infrastructure across specialized agents

The next targets extend beyond unit tests to refactoring and end-to-end testing. Building every agent from scratch would repeat much of the same engineering, so the team abstracts common infrastructure into a framework Zhang calls Agent OS. Different tasks can share runtime facilities, tools, and context infrastructure while retaining their own task-specific behavior.

The architecture slide places UnitTest Gru, Refactor Gru, and E2E Test Gru above shared layers. Those layers include runtime and tools; planning, decision making, context building, and environment grounding; and fine-tuning, RAG, and prompt engineering. The intended benefit is faster construction of additional agents within software engineering: each new task can build on an existing foundation instead of recreating the machinery needed to operate in a repository.

Layered diagram places UnitTest Gru, Refactor Gru and E2E Test Gru above runtime and tools; planning, decision making, context building and environment grounding; and fine tuning, RAG and prompt engineering.Open full source frame
Agent OS shares infrastructure across engineering agents.
7:578:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:57 · section reference included

Resources

  • Example repository documenting Test Gru's configuration, test execution, and automatic or manual pull-request workflow. Service setup instructions reflect an earlier product.

  • Historical announcement of GPT-4o fine-tuning, now carrying a notice about the platform's 2026 wind-down.

Transcript download
Full transcript

Loading transcript…