AI Engineer Summit 2025
How Coding Agents Change Software Development Forever - Hailong Zhang
Loading slides…
Read the talk
How Coding Agents Change Software Development Forever
Hailong Zhang’s Gru case study shows how an asynchronous agent can turn code changes into tested pull requests, and what evaluation, context, and shared infrastructure make that workflow possible.
From a talk by Hailong Zhang
Before you start: Familiarity with Git pull requests and basic unit testing is helpful; no prior agent-building experience is required.
Which work should an agent take on?
If an AI agent takes over routine coding, debugging, testing, and operations, what remains the developer’s responsibility? Hailong Zhang begins with this division of work. His forecast leaves product design, architecture, and difficult issues with humans, while agents handle more of the routine implementation. The resulting workflow depends on human–agent collaboration: automating a task still leaves decisions about what to build and whether the result is right.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Live assistance and asynchronous delivery
The first collaboration pattern is familiar from GitHub Copilot and Cursor: an assistant lives inside the IDE and works while the developer types. Zhang places the broader category of synchronous assistance around 2020, with maturation in 2023 and rapid growth in 2024; that chronology is not a launch date for each named product. In the talk’s 2025 setting, he describes this pattern as widely adopted.
The second pattern moves the agent into the surrounding workflow. Zhang dates the emergence of this asynchronous category to 2024. A GitHub-style bot receives a manual or automatic trigger, completes a task without continuous human attention, and returns a deliverable.
| Collaboration | Where it works | Human involvement | Result |
|---|---|---|---|
| Synchronous | Inside the IDE | Developer and assistant work together | Assistance during editing |
| Asynchronous | Inside a repository workflow | Developer delegates and later reviews | A completed deliverable |
Both patterns remain useful because they serve different moments in development. An engineer may want immediate assistance while editing and separately delegate a task that can run to completion.
At repository scale, this becomes a collection of small agents operating at different stages: writing unit tests, fixing bugs, producing documentation, submitting code reviews, and handling releases. Each agent takes responsibility for a bounded piece of work, freeing human attention for the creative parts of the project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast generation makes tests more valuable
Unit testing provides a concrete place to start. In Cursor, accepting a suggestion with Tab can generate multiple lines in multiple parts of a file. The developer can move quickly while overlooking individual changes. Zhang’s concern is the gap between generation speed and attention: code arrives faster, but bugs can arrive with it. Tests provide one way to catch those issues.
Writing those tests still requires effort. Zhang points to developers’ reluctance to do that work as the motivation for Gru, an agent for generating and managing unit tests. The case study concerns the historical Test Gru workflow demonstrated in 2025.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a code change to a test pull request
Gru’s unit of work begins with a repository change and ends with a reviewable pull request:
- Trigger the agent. Invoke Gru manually or let a developer’s pull request trigger it automatically.
- Inspect the change. Determine whether the changed code needs additional tests or changes to existing tests.
- Write and run the tests. Generate the test code and execute it to check that it works.
- Submit a test pull request. Include a summary of the tests and their coverage improvement.
- Ask for a human decision. The reviewer judges the tests and decides whether to merge them.
The screenshot Zhang presents is the deliverable from this process: a test pull request with information a reviewer can use to assess it. Autonomous task execution ends at a human merge decision.
For a small TypeScript/Vitest illustration of that review boundary, suppose a change makes an order eligible for free shipping when its subtotal reaches 50. A proposed test can make the boundary explicit:
typescript
import { describe, expect, it } from 'vitest';
import { qualifiesForFreeShipping } from './shipping';
describe('qualifiesForFreeShipping', () => {
it.each([
[49.99, false],
[50, true],
[50.01, true],
])('returns %s eligibility for subtotal %s', (subtotal, expected) => {
expect(qualifiesForFreeShipping(subtotal)).toBe(expected);
});
});
Running this proposed test checks the implementation against the encoded expectations. Reviewing it asks the further question: is 50 actually the intended inclusive threshold? That distinction explains why Gru’s workflow includes both test execution and human judgment before merging.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What production use shows
Zhang reports that over 50% of Gru’s pull requests in the displayed production GitHub list were accepted and merged by humans. He treats that level of acceptance as useful in practice while acknowledging substantial room for improvement. The talk does not establish a denominator, observation window, or model configuration for the figure, so its scope is the production example he presents.
Zhang also reports that Gru handles around 80% of the unit tests in its own repository, where the team uses it daily. This describes the agent’s share of unit-test work, not a code-coverage percentage or a pull-request acceptance rate.
By commit count, Zhang reports that Gru is the team’s leading contributor. Commit count measures repository activity separately from the quality or acceptance of the resulting tests. From that internal experience, he predicts that more agents will become repository contributors during 2025.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with a task that can be evaluated
Building such an agent begins with defining a clear, concrete, feasible problem. Unit testing gives the team a bounded task; software engineering as a whole is too broad to serve as the same kind of problem definition. The scope determines what the agent should do and what its evaluation must assess.
Zhang’s construction sequence puts evaluation before model and context work:
- Define the problem. Choose a task specific enough to solve.
- Build evaluation datasets and a harness. Establish the infrastructure for assessing candidate approaches.
- Work on models and context. Develop the components that perform the task.
- Orchestrate the components. Use a framework or Agent OS to bring them together.
This order gives model and context decisions an evaluation setting instead of leaving success defined only by a plausible-looking output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Select models and build context for the task
Gru is designed to work across frontier models from OpenAI, Anthropic, DeepSeek, Google, and other providers. The team evaluates models in different scenarios to choose an appropriate model for each one. That choice can vary between stages of a single job: a job does not have to use one model throughout.
The team also adapts models. Zhang describes fine-tuning GPT-4o with human-labeled unit-test code to improve test generation. Scenario-based selection and fine-tuning address different decisions: which model to use for a stage, and how to adapt a model to the desired output.
Context is also task-specific. For unit testing, Gru builds context around the programming language and testing framework, then gathers information from the repository environment. Relevant sources include GitHub issues, code reviews, commits, pull requests, README material, and the code itself. Gathering is followed by filtering and selection so the information fits into the model’s context. The mechanism is therefore more deliberate than simply passing the entire repository to an LLM.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Example repository documenting Test Gru's configuration, test execution, and automatic or manual pull-request workflow. Service setup instructions reflect an earlier product.
Historical announcement of GPT-4o fine-tuning, now carrying a notice about the platform's 2026 wind-down.
Full transcript
Loading transcript…