Coding agents and automated change
Repository actions and feedback
In a 2025 demonstration, Anthropic’s Boris Cherny invoked Claude Code from a terminal. The editor displayed a diff, a comparison of two file states. Look at the first argument to the function, then at the decision still open in the terminal.
- File —
index-in-card.js - First argument to
presentMany— Currentusers→ proposedusers || [] - Permission prompt — “Do you want to make this edit to index-in-card.js?”
The proposed expression passes users through when it is truthy and supplies an empty array otherwise. The decision hinges on what a missing list means. If there are no users, an empty array represents that state. If loading failed, the same fallback could hide the failure. The prompt is still open, so the developer can decline and inspect the caller. Approving the write would leave a separate question for a test: what does the application do when the list is missing?
The model proposes text and tool calls from patterns it learned during training. The harness is the application software around it. In Bill Chen and Brian Fioca’s account it includes “a collection of prompts and tools combined in a core agent loop.” The application checks a requested operation, performs it when permitted, and returns the result. A rejected write and a successful write therefore give the next model call different evidence. A script can also retry an operation, but its author fixes the path through the script. An agent can use the result to choose between reading a file, testing a hypothesis, or asking for a decision. Chen and Fioca on the harness · Harness responsibilities · Agent action selection
Where the programmer intervenes
| Handoff | What the agent provides | What the programmer decides |
|---|---|---|
| Editor diff | Cursor Agent displays edits for inspection and rejection. In Cherny’s Claude Code session, the editor shows the proposed change while the terminal awaits approval. | Does the changed expression make sense in this caller? |
| Terminal session | OpenAI Codex CLI reads files, edits them, and runs commands in a chosen directory. OpenCode lets developers configure and extend the file and shell tools. Pi starts with a minimal harness and lets developers package extensions, reusable instructions called skills, and prompt templates. | Which checkout should the agent use? Which operations and recurring procedures should be available there? Pi creator Mario Zechner on that choice |
| Delegated task | Cognition’s Devin accepts assignments such as a reproducible bug or a bounded migration. At OpenAI’s May 2025 cloud Codex launch, each task ran in its own environment with the repository already loaded. | What starting conditions and expected outcome will let me check the work when it returns? |
| Pull request | GitHub Copilot can submit changes for a delegated issue. Cloud Codex at launch could also propose a pull request. | Do the proposed changes and their checks satisfy the issue well enough to merge? |
Behavior, scope, and authority
An acceptance criterion describes an outcome someone can observe under stated conditions. In Kiro’s feature specification example, submitting a form with invalid fields should produce field-specific errors. A generic rejection prevents the submission but leaves the user unable to tell which field to correct. A preservation check would submit valid data and confirm that it still succeeds. Together those checks say what should change, what should keep working, and why redesigning the rest of the form is outside the assignment. Behavioral contracts
The kind of change determines the promise. A repair removes a defect. A feature adds an observable capability. A refactoring improves internal structure while preserving observable behavior. For a migration, name the consumers that must work during the transition as well as at its end. Kiro can record requirements, design, and implementation tasks separately. A discovery during implementation may require updating all three. The agent can inspect existing forms to find the project’s field-error component. If the product has no convention for reporting an error that belongs to several fields, the owner may need to decide what the user should see. Independent work can continue in the meantime. Fowler’s definition of refactoring
Permission to act
The task describes the desired software. Authority determines what the agent may do to produce and deliver it. Claude Code distinguishes reading files, modifying them, invoking a shell, and accessing the web. An external service applies its own credential rules. GitHub’s cloud agent can push to a designated branch and submit a draft pull request, while GitHub reserves approving and merging that proposal for humans. Deploying the accepted code requires permission in the destination environment. Claude Code permissions · GitHub’s cloud-agent restrictions
How automated software change developed
One early approach to automatic programming started with what the programmer could state precisely. In 1969, Cordell Green described constructing a program in the course of proving that available operations could produce a specified result. The proof justified the construction relative to its formal description. Even in a restricted form of LISP, finding such a proof could be expensive. Green supplied the proof for a recursive sorting example that the system could not synthesize in reasonable time. Green’s paper
A programmer often knows part of the implementation or can recognize the desired output more easily than formalize the entire job. In 2006, SKETCH let the programmer leave holes in a partial program. The synthesizer searched for fillings that satisfied a behavioral specification within bounds on inputs and execution. Sumit Gulwani’s 2011 spreadsheet system took example input and output strings instead. If two transformations fit every example, the system could ask for the intended output on a cell where they disagreed. The human answer narrowed the search by supplying a missing requirement. SKETCH · Programming by example
For a repair, much of the desired program already exists. GenProg’s 2009 method used a failing test to focus modifications on relevant execution paths, borrowed code structures from the program, then compiled and tested variants. Finding a variant that satisfied both the failing and previously passing tests gave the search a stopping condition. In 2015, Qi and colleagues showed how much depended on that condition. Some checkers accepted incorrect output when the program exited successfully. Limited test coverage also let patches pass after deleting required behavior. GenProg · Analysis of patch plausibility
Code completion drew guidance from regularities in programs people had already written. Hindle and colleagues used frequencies of short token sequences to suggest Java completions in 2012. In 2021 the Codex model generated function implementations, and HumanEval used executable tests to judge candidates. The likelihood of a continuation could guide generation even when its correctness had to be determined by another means. The Codex in that study was a model. The cloud product introduced in 2025 also supplied a repository and execution environment. On the Naturalness of Software · Codex and HumanEval
SWE-bench in 2023 made an issue in an existing repository the assignment. Locating the implementation and understanding its callers became part of solving it. SWE-agent in 2024 investigated how the interface for navigation, editing, and execution affected that work. OpenHands supplied a platform for running such actions. Robert Brennan of All Hands AI described its editor for inspecting and changing files, terminal for running programs, and browser for consulting documentation. These tools matter when the next step depends on a discovery. At a known cursor position, completion can still suffice. For a precisely specified recurring rewrite, a programmed transformation can apply the rule directly. SWE-bench · SWE-agent · Brennan’s introduction
Investigating and changing a repository
The working baseline
Identify the directory the agent will use and the revision it contains. A commit records a saved snapshot. HEAD usually identifies the current branch, or points directly to a commit in a detached checkout. git status adds information a commit identifier cannot supply. It distinguishes changes staged in the index for the next commit, further edits in working files, and untracked files. Ignored files are normally absent from the listing. Inspect existing modifications before assigning credit for an edit or reverting an unsuccessful attempt. Git glossary · Git status
Read the project instructions that apply to the files. Under the AGENTS.md convention, the closest instruction file can specialize a subproject’s commands. Confirm the package, build, and test commands against project and continuous integration configuration. Then check what is actually installed. With pnpm, --frozen-lockfile makes installation fail when the dependency record is missing or inconsistent with the manifest. By contrast, --lockfile-only can update the record without installing the packages. Two checkouts with identical lockfiles can differ in their ability to start the application. pnpm install
Separate files can lead to the same port
ExampleThe boundary that separates edits ends before the servers bind their sockets.
Read the diagram as text
- Repository history.
- Checkout A. Own working files and index.
- Checkout B. Own working files and index.
- Server A.
- Server B.
- Host address and port. Shared host networking in this example.
- Repository history → Checkout A: Supplies A’s starting snapshot.
- Repository history → Checkout B: Supplies B’s starting snapshot.
- Checkout A → Server A: Loads A’s files.
- Checkout B → Server B: Loads B’s files.
- Server A → Host address and port: Holds exclusive bind.
- Server B → Host address and port: Second bind fails.
A test suite groups checks for execution together. Run a relevant portion before editing when feasible. Save the input, outcome, and any prerequisite that prevented execution. If a database failed to start, the baseline tells you about setup but leaves the application’s response unknown. If the test reached the application and observed the wrong response, the same input can later distinguish the proposed repair from the starting behavior. Test cases, fixtures, and suites
Concurrent checkouts
A linked Git worktree gives a task its own checked-out files, HEAD, and index while sharing repository history. In Liam Hampton’s demonstration, entering the background agent’s worktree selected the changed application, but starting it exposed a port conflict. Follow the two processes to the same port in the figure. Package caches, credentials, and databases can likewise be shared according to their own configuration. A sandbox imposes separate limits on what a process can read, write, or contact. Git worktrees · Filesystem boundaries · External effects
Locating behavior and its consumers
The model’s context is the information visible to its current call. An agent can have permission to read a million lines while needing only a few to decide where to look next. Aider’s repository map shows selected filenames, symbols, and defining lines such as function signatures. Aider ranks a graph of file dependencies and chooses entries that fit the current conversation’s token budget. A signature can answer which function accepts the relevant fields. Reading its body can answer which fields it rejects. Selecting sufficient information · Revealing detail on demand
| Question | Operation | Evidence it returns |
|---|---|---|
| Where does the error message occur? | Search for its literal text. | Matching lines, possibly in fixtures, comments, and unrelated code. Ignore rules affect the search. |
| Which declaration does this name denote here? | Go to definition. | Locations resolved by the language server, if it can resolve the symbol. |
| Who depends on this declaration? | Find references. | Source locations using the symbol. A separately built consumer or dynamic lookup may need another search. |
| How could execution reach this code? | Inspect incoming calls, then read the caller’s conditions. | Possible call sites. The selected input and configuration determine which path runs. |
Two inputs determine the validator’s behavior
ExampleFollowing calls locates the function. Following configuration reveals which rule it uses.
Read the diagram as text
- Submission test.
- Public handler.
- Test configuration.
- Field validator.
- Submission test → Public handler: Invokes with form input.
- Public handler → Field validator: Passes submitted fields.
- Test configuration → Field validator: Selects validation mode.
For example, a public submission handler might always call the same validator while a configuration setting selects strict validation. Finding the validator answers where the rule lives. Following the test through the handler and reading its configuration answers whether that test could expose the strict-mode defect. Run the relevant entry point to confirm what happened. Even the set of files edited by a known repair can omit an unchanged consumer that explains the requirement.
When a saved summary disagrees with a changed file, reread the file. When observed behavior disagrees with documentation, preserve the observation and check which behavior the owner intends. A fixture or search result can contain text phrased as a command. It remains input to the investigation and cannot grant permission to execute it. Refreshing context · Keeping instructions and data distinct
Planning around dependencies
Divide a change into pieces with a recognizable outcome, an owner, and a way to check the result. Suppose a catalog lookup that returned a title string will now return a record with a title and an author. The new return shape is a decision both the lookup and its callers depend on. Once the fields are agreed, one developer can update the lookup while others adapt a receipt and a packing slip to print the title. If the callers import generated types, the revised declaration must reach the generator before those callers can be checked against it. Brennan on decomposing refactors
Unit tests can let each caller substitute a catalog that always returns the expected record. That leaves a gap where the changes meet. Print both documents using the real lookup for a sale with a known title. Compare each title line with that title. Comparing the documents only with each other would accept two matching author names. The work can proceed on separate files, but the check needs the changes together. Exercising real integration boundaries · Multi-Agent Systems
Three edits meet at a real sale
ExampleThe return contract lets three owners proceed. Only their combined code exercises both consumers against the new producer.
Read the diagram as text
- Agreed title and author fields.
- Catalog lookup. Produce the record for the requested book.
- Receipt generator. Print the purchased title.
- Packing slip generator. Print the purchased title.
- Known-sale check. Run both generators with the real lookup.
- Agreed title and author fields → Catalog lookup: Specifies return value.
- Agreed title and author fields → Receipt generator: Specifies field to read.
- Agreed title and author fields → Packing slip generator: Specifies field to read.
- Catalog lookup → Known-sale check: Provides real lookup.
- Receipt generator → Known-sale check: Produces receipt line.
- Packing slip generator → Known-sale check: Produces packing slip line.
A plan earns its cost when investigation can prevent a costly wrong edit. For a known correction at a unique location, an edit and focused test may be sufficient. For an interface change, locating consumers early can save rewriting several dependent patches. Add a newly discovered consumer to the plan and identify the check it requires. Dividing every operation into its own assignment merely adds handoffs if none can be verified independently. Claude Code’s planning guidance
Edit formats and their limits
An edit format determines what the model must generate and what the editor must locate. Robert Brennan’s OpenHands slide lists three strategies. With full-file replacement, even a one-line correction requires the model to reproduce every line it wants to keep. A diff-based patch specifies the lines to add or remove, with surrounding context to locate them. Find-and-replace supplies an existing passage and its replacement. Both localized formats leave the rest of the file in place, provided the editor can identify the intended site. The editor reports whether it applied the change. A terminal check can then exercise the changed code. Brennan’s editor slide

Anthropic’s text editor requires a unique exact match for a replacement, including whitespace. A repeated fragment needs more surrounding text to identify it. A Git patch hunk uses context lines around additions and removals. git apply --check tests whether the hunks fit without writing or reserving the files. By default a failed hunk rejects the patch as a whole. With --reject, some edits can already be on disk and the rejected hunks appear in .rej files. Inspect the new state before retrying. Also check the tool’s vocabulary. Aider calls its search-and-replace format diff and offers a separate simplified unified-diff format. Text editor contract · Git apply · Aider edit formats
For repeated structural changes, a codemod transforms parsed source constructs. LibCST’s concrete syntax tree retains Python comments, parentheses, and whitespace, allowing a transformation to preserve those details around a selected node. It can still select a call whose behavior should stay as it was. After any kind of edit, inspect the actual changed regions and inventory staged, unstaged, and untracked files. A one-line change to a default can affect every caller. Updating several callers together can be the smallest coherent patch. LibCST · Making changes reviewable
Execution and diagnosis
Reading a command result
A terminal tool request contains an operation to attempt. Evidence begins with what the application actually launches. Record the command and arguments, working directory, candidate revision and local edits, relevant configuration, and build loaded or produced. If the executable cannot be found, the application code may never run. If a server starts successfully, it can continue running while a separate client checks its behavior. Tool dispatch · Node.js child processes
| Action | Recorded result | Claim supported |
|---|---|---|
| Build the form application. | Build exits successfully and produces the bundle. | The configured build completed for those inputs. |
| Serve that bundle. | Server prints a listening address and remains active. | A client can attempt a connection to that address. |
| Probe the submission endpoint. | Invalid input returns a response identifying the field. | The API identified the field for this request. |
| Submit through the browser. | The browser check finds the error beside the invalid field and exits successfully. | The exercised page presented the required error in that location. |
Standard output and standard error are distinct streams. A line on either can arrive before the check finishes. An exit code or terminating signal describes how the process ended. In Node.js the close event also waits for its standard I/O streams to close. Check the final assertion and retained output before reporting success. For a background job, track the process through termination. Sending a stop signal alone establishes neither that it exited nor that a shell’s descendants stopped. Stopping owned work
Repair and preservation checks
Run the same defect check on the starting revision and the proposed repair under comparable conditions. If it fails before the edit and passes afterward, the comparison shows that the outcome changed. A pass on the edited version alone cannot establish that. Rerun checks for behavior that should remain intact as well. A new failure there can reveal a regression, a break in previously working behavior. In either case, the test oracle is the rule that decides whether the observed result is acceptable. If that rule checks only part of the requirement, a faulty implementation can earn a pass. Matching checks to claims · Executable outcomes
| Candidate output | Contains exactly one 3 and one 7 | Equals the required sequence `[3, 7]` |
|---|---|---|
[7, 3] | Pass | Fail |
[3, 7] | Pass | Pass |
In automated repair, a patch that satisfies the validation tests is called plausible. It overfits the tests if it passes them while violating the intended behavior on other inputs. In a libtiff repair analyzed by Qi and colleagues, generated patches removed a faulty integer-overflow guard. The suite accepted the change, which reintroduced a vulnerability. Repairing the guard and deleting the protection could both satisfy the observations available to the search. Patch plausibility and correctness
Choose a check that could detect the specific mistake. Type checking can expose an incompatible declaration. A unit test can distinguish two output sequences. Integration tests exercise real callers and dependencies, while a browser test can inspect what appears after a user interaction. Property-based tests search generated inputs for a violation of a stated rule. Differential tests compare implementations on valid inputs and use disagreement to expose a possible defect. Both depend on the choice of inputs, and two implementations can agree on a wrong answer. QuickCheck · Differential testing with Csmith
Inspect tests the agent changes alongside the implementation. Look for expectations derived from the same algorithm, deleted assertions, newly skipped cases, and edits to the mechanism reporting completion. In Nick Nisi’s Case workflow, an agent created the marker file the gate expected without running the tests. Even a hash of output would need a trustworthy connection to the actual run and candidate. Nisi’s account
Choosing the next debugging check
Preserve the failing input, revision, setup, and result. Then look for a boundary where two explanations predict different observations. At Zed, a repository editing task failed. Inspecting what the model received from a search for fn run exposed a problem upstream of the edit. The search returned a match without enough of the surrounding construct. Zed used Tree-sitter, a code parser, to expand the match to its syntax boundaries. The team could check the expansion on a fixed source file without calling a model. Changing the editing prompt alone would have left the missing source context unexplained. Nathan Sobo’s investigation
For a stale display, inspect the failing interaction before reloading the page. If no request went out, check the code that should have triggered it. If the request went out but produced no usable response, check whether it failed or is still pending. Once a response arrives, compare its value with the displayed value for the same item. An old value in the response sends the investigation upstream. A current value beside a stale display points to the code that consumes the response. A reload that restores the display cannot tell you which step went wrong.
Trace a stale value across the response boundary
ExampleA captured response can separate a producer problem from a display problem without rerunning the whole interaction.
Read the diagram as text
- Inspect failing refresh.
- Inspect trigger.
- Inspect request outcome. Find out whether the request is pending or failed.
- Compare the same item. Read the captured response and visible value.
- Trace producer.
- Trace display update.
- Inspect failing refresh → Inspect trigger: No request issued.
- Inspect failing refresh → Inspect request outcome: Issued without usable response.
- Inspect failing refresh → Compare the same item: Response available.
- Compare the same item → Trace producer: Response already stale.
- Compare the same item → Trace display update: Response current, display stale.
Controlling what changes
A successful rerun after several changes rarely identifies the cause. Suppose a date test passes alone on the base revision but fails after a language-setting test on the patch. Run the date test alone on the patch and the two-test sequence on the base. Use a fresh process for each run and hold the rest of the setup fixed. If both new runs pass consistently, only the combination of the patch and that test order fails among the four conditions. If the sequence also fails on the base, investigate the pre-existing dependency on process state before attributing the failure to the patch. pytest on flaky tests · Testing explanations
With a repeatable failure and known good and bad revisions, git bisect narrows the search by selecting intermediate revisions to rebuild and test. Skip a revision that fails to build for an unrelated reason, even if the introducing commit remains uncertain. In a live session, set aside a speculative edit if it obscures the reproduction. When another retry cannot distinguish any remaining explanations, inspect a different boundary or ask for the missing information. Git bisect · Bounding retries
Human direction and integration
Steering and handing off work
Interactive work lets a programmer correct an assumption before the agent builds more code around it. In an Aider session, the programmer wanted tests to simulate terminal input. After Aider changed the tests, the programmer reported that the application’s main function rejected the new input argument. Changing the test alone could never deliver that input to the component that read it. The programmer identified where input and output needed to go, supplied the library signatures, and interrupted an approach that used the wrong interface. Aider switched to PromptSession, and the programmer reported that it worked. Aider’s debugging session
For background work, choose an assignment whose outcome the returning reviewer can recognize. A performance fix should arrive with the affected workload, evidence of where time went, the change, and a rerun checking its effect on that workload. Interrupt the programmer if a choice would change the goal or require more authority. For a discoverable fact such as a library signature, inspect the installed version first. A progress update should say which observation changed the plan and what decision, if any, remains with the owner. Verifying background performance work
A handoff identifies the current goal, checkout, revision, working edits, active processes, completed checks, and outstanding work. If direction changed, record the superseded decision and which checks depended on it. For example, an API test could still establish that the response identifies the invalid field after a change to error placement, while the browser assertion needs updating. On resumption, inspect the files and process state before continuing. A summary can be stale and a stopped conversation can leave both edits and a server behind. Reconstructing context · Handling waits and approvals
Reviewing the complete change
Name the states being compared. git diff A B compares the two commit snapshots. git diff A...B compares their common merge base with B, showing what the branch contributes since divergence. In a working checkout, plain git diff compares working files with the index. git diff --cached compares staged content with a commit, normally HEAD. Check status as well. An untracked helper or unstaged correction might explain why a local run passed while the proposed commit is incomplete. Git diff
Follow the requirement through the entry point, changed code, and unchanged consumers. Inspect configuration, dependencies, generated files, and modifications to the tests. For the libtiff patch, a reviewer could ask where the changed program now prevents the overflow. That question connects a visible deletion to a protection the suite failed to exercise. An AI reviewer can raise such a hypothesis. Ask for its consequence, the source path that permits it, and a check or independently established requirement. Agreement between two models still permits both to share the same mistaken expectation. Google’s code review guidance · Presenting evidence for review
A review also needs someone responsible for answering it. Brennan reported that early OpenHands pull requests appeared under the bot’s identity. Under his team’s rules, the initiating programmer could then approve the request, bypassing the intended second human review. Failed checks could sit unanswered. The team changed attribution so the initiating human owned the proposal and its consequences. Brennan on pull-request ownership
Which candidate passed?
The candidate is the software proposed for acceptance. Tie each result to the source revision, relevant working files, dependency and configuration state, and generated or compiled artifact the check actually loaded. A browser test pointed at an old bundle can pass after an edit to the source. A completion report should identify the command, the candidate it exercised, the behavior it checked, and its outcome. Identifying what was tested
In the figure, a title assertion passes on A. Editing the receipt creates B. The expected title has stayed the same, but the code producing the line has changed. Exercise the receipt built from B and record its own result. Evidence about an unaffected property can carry forward if you can identify the relevant inputs and show they still apply. Review approval can also expire. GitHub branch protections can dismiss an approval when a new push or a changed base alters the reviewed diff. Protected branches
One assertion, two executions
ExampleEditing the receipt opens a gap in the evidence even when the expected title stays the same.
The title assertion passes on A.
Read the diagram as text
- Candidate A.
- Receipt title assertion. Compare the title printed for a known sale with the purchased title.
- Pass on A.
- Candidate B. Receipt implementation edited.
- Title unchecked on B.
- Pass on B.
- Candidate A → Pass on A: Version exercised.
- Receipt title assertion → Pass on A: First execution.
- Candidate A → Candidate B: Receipt edit creates.
- Candidate B → Title unchecked on B: Needs new result.
- Candidate B → Pass on B: Version exercised.
- Receipt title assertion → Pass on B: Fresh execution.
- Print the receipt on A. The title assertion passes on A. Active: Candidate A, Receipt title assertion, Pass on A. New: Candidate A, Receipt title assertion, Pass on A.
- Edit the receipt. The earlier observation still describes A. B has different receipt code. Active: Candidate A, Receipt title assertion, Pass on A, Candidate B, Title unchecked on B. New: Candidate B, Title unchecked on B.
- Print the receipt on B. The check now records an outcome for B as well. Active: Candidate A, Receipt title assertion, Pass on A, Candidate B, Pass on B. New: Pass on B.
Distinguish a pass and a failure, both observed outcomes, from a skipped check or an unavailable assessment. A runner can intentionally omit a case. A service that failed to start can prevent the runner from judging it at all. Some GitHub status rules accept skipped or neutral statuses. Before telling a reviewer the behavior passed, inspect what ran on the final candidate. Completion checks
Checking the integrated result
Merging combines versioned changes. A textual conflict leaves edits Git cannot reconcile automatically. Inspect the common base and both contributors’ changes to preserve their respective intentions. A semantic conflict can survive a clean text merge. In a reported Microsoft Edge merge, Chromium had renamed a member from IsIncognito to GetIncognito. Downstream code still called IsIncognito. Git could combine those files, but the build failed when the compiler tried to resolve the old name. Git merge · Microsoft’s Edge merge study
If the target branch has changed, merge the current target and check the resulting code. Inspect and exercise the combined result. A runtime conflict can be subtler than a missing member. If reading a saved value can silently fall back to a current default, first save one value, then change the default. Exercise the combined code and assert the saved value. With equal values, the assertion would pass along either path.
A clean merge with an unresolved member
The name supplied by the upstream declaration differs from the name the downstream caller needs.
Read the diagram as text
- Upstream declaration. GetIncognito
- Downstream caller. IsIncognito
- Combined source. Git reconciles the files.
- Member lookup fails. The compiler cannot resolve the downstream call.
- Upstream declaration → Combined source: Defines GetIncognito.
- Downstream caller → Combined source: Calls IsIncognito.
- Combined source → Member lookup fails: Build looks up IsIncognito.
Continuous integration (CI) runs automated checks on shared changes. GitHub’s merge queue constructs a temporary candidate from the proposal, current target, and preceding queued work. Its commit identifier differs from the pull-request head. Required GitHub Actions workflows need to handle merge_group to report checks on that combination. A submission opens a proposal, a review records judgment, a merge changes the target history, and deployment changes what runs in an environment. Verify and authorize each transition separately. Reverting a commit afterward can leave database writes or other external effects intact. GitHub merge queues · Restoring software after effects · Software Factories
Evaluating software work
Designing a repository evaluation
Verification asks whether a particular change satisfies a requirement. Evaluation measures how a coding system performs across the work you intend to delegate. Each case specifies an assignment and its starting state. Each trial is one attempt at that case. The starting information matters in maintenance work. Handing the agent the commit that introduced a defect narrows its search. Asking it to find defects first introduces a selection effect: if it finds mostly simple bugs, its repair rate describes that easier subset. Every bug it missed is absent from the repair count. The SM100 maintenance evaluation · Cases and trials · Representative tasks
| Role | Materials | Use in a comparison |
|---|---|---|
| Solver inputs | Issue, repository at its base commit, installed environment | Give each system the same starting problem and allowed information. |
| Trial outputs | Candidate patch and execution logs | Retain what each system produced, including unsuccessful attempts. |
| Assessment materials | Reference fix, test patch, designated repair and preservation checks | Keep the reference fix out of solving. Use the same grading procedure on each submission. |
Define success according to the assignment. Check a repair against the failing case and preserved behavior, a feature through its public entry point, a refactoring against its preservation promise, and integration against the named target. Specify the model and harness, allowed tools and lookup, context policy, human help, time and resource limits, number of attempts, stopping and selection rules, and grader version. Retain enough evidence to distinguish an incorrect submission from a broken environment, a grader mistake, and a case whose cause is still unknown. Ibragim Badertdinov on executable issue tasks · SWE-bench evaluation guide
Two controls solve different problems. Resetting the environment prevents one attempt from inheriting another’s files or processes. Reserving untouched cases prevents earlier grades from shaping the system selected for final evaluation. Even an aggregate score can guide tuning. Keep discovered failures as regression cases, freeze the configuration, then evaluate on fresh representative tasks. Match resource limits between systems. Anthropic found that extra memory could prevent an infrastructure failure or enable a different solution strategy even with the model, tasks, and harness fixed. Infrastructure noise · Paired comparisons
Interpreting coding benchmarks
| Benchmark | Given | Submitted | What passes |
|---|---|---|---|
| HumanEval | A function prompt | A generated implementation | The function satisfies the executable tests. |
| Original SWE-bench | An issue and a repository revision | A repository patch | The patch applies. Designated fail-to-pass tests that exposed the problem now pass, and designated pass-to-pass tests protecting existing behavior keep passing. |
Resolved rate counts the fraction of instances meeting the repository benchmark’s grading rule. Identify the dataset revision, split, and grader implementation. For example, the SWE-bench grader’s pass-and-fail mode treats a skipped fail-to-pass test as a failure yet can count a skipped pass-to-pass test as maintained. A different mode has different rules for missing results. Knowing the label alone does not tell you which code ran. Grader implementation
Pass@k measures coverage by at least one passing candidate among k samples. If only the fourth of ten patches passes, the ten contain a success. A system that returns the first patch still hands the programmer a failing change. To interpret the result for deployment, ask how much sampling was allowed, which checks the agent could consult, and which candidate its selection policy actually returned. Attempts and selection
Hidden requirements and exposed solutions
A grader can also reject a reasonable solution. An original SWE-bench issue said that TfidfVectorizer ignored its copy argument. The resolving pull-request discussion later specified a deprecation warning, and the hidden test required its exact text. A solver reading the issue lacked that decision. In August 2024, SWE-bench Verified selected 500 tasks after developers screened issues and tests for problems including hidden requirements. Introducing SWE-bench Verified
In February 2026, OpenAI reported further problems in Verified. Investigators concentrated on the 27.6% of tasks models often failed. They judged at least 59.4% of that selected subset capable of rejecting functionally correct solutions. That percentage describes the audited subset. On some tasks, models also reproduced distinctive problem text or reference fixes, which OpenAI interpreted as evidence of prior exposure. It stopped reporting Verified for frontier launches. Prior exposure during training and retrieving a public fix during a trial are different routes to a misleading success. DeepSWE uses newly authored tasks from contributors familiar with the repositories to reduce reliance on issues with already published solutions. OpenAI’s reassessment · DeepSWE task design
Measuring accepted delivery
A patch may arrive in minutes and wait hours for someone who understands the affected service. To measure useful delivery, define a common acceptance standard and follow comparable assignments through that point. Count abandoned work, review, corrections, integration effort, and operating expense. Track later repairs where they are observable. Separate elapsed time from human attention. A programmer who works on another issue while an agent runs incurs a wait on the first issue’s clock without necessarily spending that interval supervising it. A teammate who reviews the patch contributes time even if the author has moved on. Comparison baselines · AI Cost and Performance Engineering
METR randomized whether 16 experienced open-source developers could use AI on 246 predefined tasks in repositories they knew well. Allowing the early-2025 tools increased measured completion time by 19% relative to working without them. Participants estimated afterward that the tools had saved time. The discrepancy matters because a survey and a clock can give opposite answers about the very same assignments. The study gives a result for those developers, tools, and tasks. METR’s randomized study
By February 2026, METR reported that some developers declined participation or withheld tasks they were unwilling to do without AI. Completion and delivered work differed, and concurrent assignments made task time harder to interpret. METR judged the later data unreliable for estimating the current effect. In a local comparison, record which assignments enter the study, who already knows the repository, whether the work is supervised, who reviews it, and what counts as accepted. More pull requests can reflect smaller task divisions or more review work without settling whether useful delivery improved. METR’s experiment redesign
Practical reference
Frequently asked questions
How is an agent different from autocomplete?
Autocomplete suggests text at an editing position. The programmer decides what to do with it. A coding agent can choose to inspect a caller, request an edit, or run a test and use the returned result to choose its next operation. The programmer can intervene at any decision the interface exposes.
How much repository context is enough?
Gather enough to trace the relevant entry point, affected consumers, and a check that would expose a wrong change. Use a repository map to choose the next file. Read its implementation or configuration when the answer depends on details omitted from the map.
Should an agent write tests?
Yes. Derive the expected result from the requirement or an independently checked example. Ask whether a plausible faulty implementation would fail the assertion. A test that computes its expectation with the same faulty logic can pass consistently.
Can passing tests replace review?
A passing suite supports the properties it actually exercised. Review can reveal an omitted caller, a deleted guard, or an assertion weakened by the patch itself. Inspect those changes before accepting the result.
When should the agent ask for help?
Inspect project commands, library signatures, and existing conventions in the checkout first. Ask when the remaining choice changes what users receive, requires information only the owner has, or exceeds the assigned authority. A recurring procedure can be recorded for later sessions. Agent Skills
Does a worktree provide a sandbox?
A worktree supplies a separate checkout and staging area. A process launched there can still bind the host’s ports or contact shared services with available credentials. Containment depends on configured filesystem, network, and execution controls.
When does parallel coding help?
Parallelize when each assignment has a clear owner, its dependencies are settled, and its result can be inspected. Where the changes meet, run the real consumers against the combined candidate. Multi-Agent Systems
Will a benchmark predict results on our repository?
Transfer depends on how closely the tasks, tools, available information, assistance, and acceptance checks match your workflow. For a local estimate, fix the system configuration and assess representative assignments held out from development.
Can an agent merge or deploy?
Only when the task owner delegates the specific operation and the destination permits it. GitHub’s cloud agent can submit its work on a designated branch but cannot approve or merge its pull request. A separate destination grant governs deployment. Software Factories
Open questions
How can an agent tell when it needs a product decision before it writes both code and tests around the same assumption? Agreement between implementation and test can conceal the mistake. Progress would mean identifying a small distinguishing example that an owner can resolve, while allowing different implementations of the agreed behavior.
Which results remain valid after a target branch advances? An unchanged caller can depend on a renamed declaration, a configuration value, or a generated artifact. A useful method would explain why each retained check still applies and compare its predictions with checks actually run on the combined candidate.
When does unattended work reduce total human effort through acceptance and maintenance? Task selection changes with adoption, waiting overlaps other work, and review can move to another person. Comparable assignments with separate measures of elapsed time, active attention, rejected work, and later repair would reveal where the savings or costs accrue.




















































































































































































































































































































































































































































































































































































































