Foundations and history
What a reviewer can accept
Reyes’s request to Code Droid named the repository, the failure and a possible delay: “In firecrawl, if we get a no content error - retry with a waitfor of maybe 2000ms more in factory-mono”. The error tells an engineer when the retry should happen. The word maybe leaves the delay open to discussion. Code Droid can inspect the repository before it proposes the change. The coding-agent execution loop

Droid inspected the checkout, identified the branch, and consulted recent changes before proposing a plan and asking Reyes for clarification. As Reyes put it, “AI tools are only as good as the context that they receive.” The repository told Droid how the code worked. Reyes could clarify what he wanted it to do. After his reply, Droid ran checks and prepared a pull request (PR), a proposal to incorporate its change into the shared codebase. Reyes reported that the PR passed the project’s automated continuous integration (CI) checks.
A patch and a test report are artifacts, outputs a receiver can inspect. A maintainer can read the patch to see which error should trigger a retry. To confirm that the retry ran, the maintainer would need a test result or trace showing a no-content response followed by a second scrape. Confirming the fix in production would also require identifying the deployed version and observing how it handled that condition. Reyes’s demonstration ended at the PR.
In Lloyd’s proposed workflow, monitoring can uncover a bug for triage, where the team decides how to handle the report. The team chooses which repositories and stages to automate. Lloyd’s diagram marks three points where people review the work. View the original diagram at full resolution · Lloyd’s interview on factory design
- Reviewing the spec — Settle what the feature should do before implementation begins.
- Reviewing the code — Inspect the proposed change against the agreed behavior.
- Reviewing the product — Try the working feature and decide whether it is ready to ship.

| Account | What starts work | What gets checked | Human decision |
|---|---|---|---|
| Factory · demonstrated task | A request to retry a scrape that returns no content | Droid runs project checks on the patch. | Reyes clarifies the plan. A maintainer can then assess the proposed retry. |
| Warp · proposed workflow | A product idea or bug report | Specification, implementation, and working product | Settle intent, review the code, and decide whether the product is suitable to ship. |
| Uber · illustrated feature | A pickup idea discussed in Slack | Minion, Uber’s coding agent, attaches check results and a simulator screenshot to a draft PR. | Compare the rendered screen with the design and decide whether the draft needs more work. |
The July 2026 World’s Fair Software Factories track brought together different questions about this loop. The program listed Factory’s Tereza Tížková under Rise of the Software Factory. Her recorded talk is What It Actually Takes to Build a Software Factory. Lloyd’s session on self-improving factories was released as Software Engineering Is Becoming Factory Engineering. Dex Horthy of HumanLayer examined the review and testing burden in Why Software Factories Fail. These concerns also apply to a team using one agent, ordinary build tools, and human maintainers. The scope of the factory follows the work those participants undertake together. World’s Fair program · Defining completed work
Reuse and organized production
Software was easy to copy long before it was easy to produce. At the October 1968 NATO Software Engineering Conference, Douglas McIlroy asked why programmers kept constructing familiar routines when a supplier could offer dependable families of them. A programmer might need the same operation at a different precision or with a different tradeoff between memory and execution time. The supplier’s contribution would be to engineer, describe, and maintain those useful variants. A catalogue of identical copies would solve little of that problem. McIlroy’s component proposal, section 8.2
For organizations taking on large contracts, the scarce resource could instead be a dependable way of organizing people. In 1969 Hitachi separated its Software Works from its Kanagawa Works as orders grew for systems including railway reservations and online banking. System Development Corporation (SDC) tried a division of labor in 1976. Managers at customer sites retained responsibility for system design. A central factory took on detailed design, programming, and testing with shared procedures and tools. Michael Cusumano’s historical study describes the difficulty of directing work across those management lines while contract demand fluctuated. SDC ended the experiment in 1978, although standardized engineering practices survived. Sharing a method across projects proved possible even when pooling the work in one organization proved difficult. Hitachi’s history · Cusumano’s SDC study
A software product line makes a third choice about reuse. Its designers identify related products, implement what they share, and plan where one product can vary from another. In their 2003 Software Factories proposal, Jack Greenfield and Keith Short used domain-specific languages to express choices in terms familiar to people working in a particular application area. A template brought reusable implementation assets and development guidance into the programming environment. This approach requires someone to understand the product family well enough to anticipate useful variations. It gives the next developer a way to express those variations without rebuilding the common parts. Greenfield and Short’s proposal
Delivery pipelines and learned generation
Even a team making a single product has a recurring production problem. An edit that works in one developer’s checkout must still work alongside everyone else’s changes. Martin Fowler’s account of continuous integration (CI), first published in 2000, describes combining work at least daily and checking each integration with an automated build and tests. Continuous delivery extends those checks until the team can release a version on demand. Continuous deployment goes further by automatically releasing changes that pass the pipeline. Fowler on continuous integration
Fragile manual deployments at ThoughtWorks motivated the deployment pipeline documented in 2006. Its practical insight was to build a package once and promote that package through successive assessments. A staging test then describes the same package that production can receive. Environment differences still matter, but rebuilding between stages introduces a needless uncertainty about what was tested. DevOps also addressed the working relationship between those developing software and those operating it. Patrick Debois founded devopsdays, which held its first event in Ghent in 2009. Testing, automation, and organizational culture are recurring subjects in the conference series. Jez Humble on deployment pipelines · History of devopsdays
Learned generation changed the source of a proposed implementation. In the July 2021 Codex study, a model trained on public code generated Python functions from docstrings. Executable tests assessed each function on a specified task. In May 2025, GitHub introduced a Copilot coding agent that could take a repository issue, inspect the code in a GitHub Actions environment, push commits to its own branch, and request review on a draft PR. At launch, a human had to approve running the PR’s CI/CD workflows. The unit of delegation had expanded from writing a function to preparing a repository change. Existing review and integration decisions still applied. Codex study · GitHub’s launch announcement · Earlier approaches to automated change
Work and responsibilities
Work intake and admission
Work intake covers receiving requests, clarifying what they mean, ordering them by priority, and selecting work to start. Admission is the point at which the team commits capacity and an owner to a result. A ticket can be important and still lack the information needed to admit it. Lloyd distinguishes a straightforward issue that can go directly to implementation from one that needs a product or technical specification. For a disputed architecture, the useful first assignment might be a comparison of two options for the person authorized to choose. That investigation has a finish condition of its own. Warp’s triage workflow · Transferring responsibility
Readiness includes the means to finish. A dependency update may have a known scope and an agreed compatibility test, yet need repository access, a running consumer service, or time from its maintainer. Record which prerequisite is missing. Deferring that update leaves its priority intact and keeps a work slot available for something that can finish. In the intake diagram, clarification changes what the team knows about a request. Deferral changes when the team undertakes it. The Kanban Guide on selecting work
Which requests become commitments?
ExampleSettling intent and making capacity available resolve different reasons for waiting.
Read the diagram as text
- Received request.
- Decline. Record the disposition.
- Clarify the outcome.
- Refer to the decision owner.
- Check readiness and capacity. Confirm access, dependencies, checks, and a receiver.
- Defer with a reason.
- Admit the work. Assign responsibility for the agreed result.
- Received request → Decline: Outside mandate.
- Received request → Clarify the outcome: Intent unclear.
- Received request → Refer to the decision owner: Decision exceeds authority.
- Received request → Check readiness and capacity: Wanted and bounded.
- Check readiness and capacity → Defer with a reason: Prerequisite or capacity missing.
- Check readiness and capacity → Admit the work: Ready and capacity available.
Specifications and acceptance criteria
A specification records the behavior and constraints people have agreed on. It identifies the effect wanted, existing guarantees to preserve, relevant operating limits, and changes outside the assignment. It can leave the implementation open. An acceptance criterion states an observable condition a receiver can use to judge the result. Recording why a condition matters helps distinguish a necessary guarantee from an implementation preference. An open decision that could change what users receive needs an owner. Defining technical requirements · Specifying what an operation promises
Suppose an export API gains an option to include a timezone. Callers that omit the option must continue to receive the original fields and timestamp values. The developer can choose the serializer. For callers requesting the new field, there is a product decision: should an export use the account’s zone or the requester’s? Either value can satisfy a schema requiring a timezone string. Once the owner chooses the account zone, a revealing acceptance example is a Paris-based requester exporting a Tokyo account. Expect Asia/Tokyo in the new field. Run the same export without the option to check the separate preservation promise.
Prose can preserve the reason for choosing the account zone. A schema specifies the representation. A scenario, such as one written in Cucumber’s Gherkin language, connects a starting state and action to an observable result. These support verification, assessing conformity to the agreement. Validation asks whether the export serves its recipient’s intended use. A shared Definition of Done adds quality obligations across work items, such as documenting changes to public response fields. Gherkin reference · Verification and validation · The Scrum Guide
Tížková’s slide describes Droid’s plan for a 750 mm wall-mounted rotor. It divides the design into a 3×3 grid so each tile fits the printer bed. Droid produced the tile and joinery files and suggested pocketing the back. Checking the tile dimensions addresses one acceptance criterion. To assess the assembled fit, a reviewer would need evidence from joining the printed tiles.

Execution and acceptance
Design a stage by working backward from the receiver’s decision. What input does the executor need, what may it change, and what must it hand over? A harness gives an agent an environment and tools in which to act. A gate enforces a condition for advancing the resulting work. Access to a test runner permits an agent to gather evidence. The organization separately decides who can accept a contribution, expose it to users, or grant an exception. Those decisions can be delegated to automation under a policy with an accountable owner. Agent Runtimes and Harness Engineering · Allocating work and authority separately
| Stage and input | Who produces what | Who decides what happens next |
|---|---|---|
| Investigate a request | An engineer or agent identifies affected behavior and open choices. | The requirement owner settles the scope or requests more investigation. |
| Implement agreed behavior | A worker prepares a patch against a known revision. | A maintainer takes that revision for assessment. |
| Check a candidate | Tools report observations. Reviewers compare them with expectations. | The maintainer accepts the evidence, asks for another check, or returns the patch. |
| Integrate reviewed changes | A merge queue builds and checks a proposed shared revision. | Repository policy permits it to land or sends a failed combination for investigation. |
| Deliver a package | Release automation installs an identified build and records its operating signals. | The release owner or delegated policy expands, pauses, or stops exposure. |
In Uday Kiran Medisetty and Adam Huda’s stadium pickup walkthrough, Uber’s assistant Cortana helps turn a Slack discussion into requirements and two possible button designs. Minion takes over implementation. The build produces a pull request. During validation, Minion can still revise that draft before a person reviews it or shared CI runs. Comparing a simulator screenshot with the Figma design lets a reviewer judge the button’s appearance. Checking which riders see the button also requires exercising the condition that makes it appear. The demonstrated handoff is the draft and its check results. Uber’s walkthrough · Giving ownership real authority
Dividing work for recombination
A good division of work leaves each receiver with something small enough to understand and a way to check it. A dependency exists when a contribution requires another’s particular output or interface. Consider a service with web and command-line clients. Once their response format is agreed, all three implementations can proceed independently against it. The clients do not need each other’s code. Both eventually need to read a real service response. The diagram shows why drawing the interface early creates room for concurrency and creates a specific obligation at integration. Decomposing work for recombination
In Robert Brennan’s OpenHands refactor workflow, imports reveal dependencies between groups of files. A worker can start with a batch whose prerequisites are complete and submit a reviewable change to an integration branch. Later workers incorporate upstream changes, and maintainers can assess the migration as a whole before proposing it for the main branch. Smaller batches reduce the amount of code in any one review. Splitting further eventually adds handoffs faster than it simplifies those reviews. Brennan’s demonstration · OpenHands integration workflow
Separate clients share a service contract
ExampleThe clients can be built concurrently, but each owes a compatibility check with the producer.
Read the diagram as text
- Agreed response format.
- Web client.
- Service.
- Command-line client.
- Check web display.
- Check command-line output.
- Agreed response format → Web client: Fields to render.
- Agreed response format → Service: Fields to produce.
- Agreed response format → Command-line client: Fields to parse.
- Web client → Check web display: Web reader.
- Service → Check web display: Actual response.
- Service → Check command-line output: Actual response.
- Command-line client → Check command-line output: Command-line reader.
Multi-agent coordination distributes work and decisions across several agents. Separate Git worktrees give each writer its own checked-out files, but writers can still make incompatible design choices or affect the same external resource. In Luke Alvoeiro’s account of Factory Missions, one worker or validator at a time acts on a feature. Searches, research, and read-only reviews can run concurrently. That arrangement preserves concurrency where findings can be combined cheaply and serializes edits where one feature depends on the application left by its predecessor. Git worktrees · Alvoeiro on the Missions architecture
Evidence and acceptance
Artifact and evidence identity
A release operator finds a passing test report and a package ready to deploy. How can the operator tell whether they belong together? Start with the package’s identity, such as a digest computed from its contents. Build provenance records how an identified output was produced. The Supply-chain Levels for Software Artifacts (SLSA) specification connects the output to a builder, build definition, execution, and recorded inputs. A source revision records what the builder obtained from a repository even if the branch later moves. The statement depends on trust in the builder. Behavioral evidence comes from exercising the package. SLSA v1.2 build provenance · Identifying what was tested and released
Two paths meet at the behavioral report in the figure. The test run records which package ran, under which configuration and conditions, and what happened. The requirement supplies the expectation against which to judge that observation. If the requirement changes, the receiver can sometimes reassess the recorded result. If the new requirement concerns behavior the test never exercised, another run is needed. A dependency change can also require a new build and a compatibility check on that build. Keeping the links among requirements, software versions, and checks is called traceability. Requirements management and traceability
One package, two kinds of evidence
ExampleOnly the behavioral verdict depends on what the requirement says the package should do.
Read the diagram as text
- Source and build inputs. Revision, build definition, and recorded dependencies.
- Package. Identified by a digest of its contents.
- Builder’s provenance statement.
- Test environment and configuration.
- Test run.
- Agreed requirement.
- Behavioral report.
- Source and build inputs → Package: Builder produces package.
- Source and build inputs → Builder’s provenance statement: Inputs named.
- Package → Builder’s provenance statement: Output digest.
- Package → Test run: Candidate exercised.
- Test environment and configuration → Test run: Execution conditions.
- Test run → Behavioral report: Observation and candidate identity.
- Agreed requirement → Behavioral report: Expectation for verdict.
A handoff connects those records to the assignment. Give the next owner the work item and agreed specification, the source baseline and patch, the package and configuration tested, the reports, and the obligations still open. Missions workers record commands, exit codes, discovered issues, and unfinished work before their sessions end. These notes tell the receiver where to resume. Retaining the actual reports alongside them lets someone inspect what happened after a temporary CI log expires. Retaining execution artifacts
Checks and acceptance claims
A test oracle is the rule that decides whether an observed result is acceptable. If a developer and a test both interpret an exclusive limit as inclusive, the test can exercise the exact boundary value and pass the wrong behavior. In a 2015 study of automated program repair, researchers assessed patches using tests that the repair tools had never seen. Some patches passed every test available during construction and still broke desired behavior in the independent cases. The same distinction matters when an agent writes its own tests or a second agent reviews the change using the same assumptions. Independent tests in automated repair · Assessment independence
Choose evidence by the mistake it could expose. Static analysis can flag an incompatible type. A behavior test can show that a supported input still works. A dependency scan can identify a known vulnerability for someone to assess in its deployment context. A browser check can show whether a person can complete a flow across several services. A domain reviewer can judge whether that flow solves the intended problem. In Missions, the planner assigns assertions to features before implementation. Validators can then use the agreed assertions, code checks, and live application flows to challenge what the worker built. Matching checks to claims
Consider a patch that changes a failing assertion so that it passes. The change may be legitimate if the requirement changed. A receiver needs the new agreement and separate authority to approve the altered check. Keep enforcement outside the implementer’s unrestricted control. GitHub can dismiss approvals when a diff changes or require approval of the latest reviewable push by someone other than the person who made that push. Inspect the configured bypass permissions and check reports as well. A status accepted by branch protection can be skipped or neutral, so the green merge indicator alone cannot establish that the intended test ran. GitHub protected branches · AI Security
The combined candidate
Integration brings contributions into the intended shared version and checks how they interact. A textual conflict occurs when the version-control system cannot automatically reconcile the edits. A semantic conflict can pass through a clean merge. Suppose one patch reverses a service’s default list order to oldest first. Another patch adds a client that labels the first entry as the latest. The service author tests the new ordering. The client author tests against the original service, which returned the newest item first. Only when the two patches meet does the client label the oldest item as the latest.
Notice that the two branch reports in the figure keep their original subjects. The combination introduces a third subject needing its own report. If a combined failure is intermittent, replay the same operation from reset data on the baseline, each individual change, and the combination. Repeat under matched conditions. A failure reproduced with one change alone narrows the investigation. A failure observed only with both changes points toward their interaction. Substituting a unit test for one of these replays leaves open what that revision would do in the integrated scenario. Verifying the combined result
Where the wrong entry first becomes visible
ExampleOnly the combined candidate contains both the new default order and the new reader.
Each passing observation belongs to a different version derived from the common baseline.
Read the diagram as text
- Baseline · newest first.
- A · reverse default order.
- B · show latest entry.
- A report · oldest first. Matches the new ordering criterion.
- B report · latest displayed. Matches the expected client behavior on the baseline service.
- A and B · combined revision.
- Combined report · wrong entry. The oldest item appears under the latest label.
- Baseline · newest first → A · reverse default order: Change service order.
- Baseline · newest first → B · show latest entry: Add first-entry reader.
- A · reverse default order → A report · oldest first: Exercise service.
- B · show latest entry → B report · latest displayed: Exercise client on original service.
- A · reverse default order → A and B · combined revision: Changed producer.
- B · show latest entry → A and B · combined revision: New consumer.
- A and B · combined revision → Combined report · wrong entry: Exercise combined behavior.
- Check separate revisions. Each passing observation belongs to a different version derived from the common baseline. Active: Baseline · newest first, A · reverse default order, B · show latest entry, A report · oldest first, B report · latest displayed. New: Baseline · newest first, A · reverse default order, B · show latest entry, A report · oldest first, B report · latest displayed.
- Check the combination. The same client operation now reads a list ordered in the opposite direction. The earlier reports keep their original meaning. Active: Baseline · newest first, A · reverse default order, B · show latest entry, A report · oldest first, B report · latest displayed, A and B · combined revision, Combined report · wrong entry. New: A and B · combined revision, Combined report · wrong entry.
Frequent integration shortens the interval during which a branch can diverge. A merge queue handles a target that moves while work awaits acceptance. GitHub constructs a temporary merge group containing the PR, the latest target, and changes ahead of it. An Actions workflow that supplies required checks must handle the merge_group event to assess that combined commit. If a contribution leaves the group or the base changes, establish which interactions the new candidate introduces before relying on an earlier result. GitHub merge queues
Uber’s SubmitQueue builds likely combinations before it knows which ones will land. This can reduce waiting, but a discarded combination consumes build capacity. In a 21-week before-and-after observation spanning a July 2024 scheduling change, Uber reported that its Go repository’s mean weekly builds per change fell from 3.39 to 1.85. The authors tracked waiting time alongside build consumption because deferring every speculative build could save compute while delaying integration. The measurement ends at integration. SubmitQueue study · Gautam Korlam on combined builds
Release and learning
Release authority and delivery confirmation
Deployment installs a version. Release makes its behavior available to users. A release owner can authorize exposure, stop expansion, and arrange recovery. In Google’s account of Rapid, a project blueprint identified owners, build and test targets, and deployment rules. Authorized automation could promote a particular package according to the service’s policy. Building once makes the package traceable across environments. Checking installed identity and configuration tells the operator which version is actually serving requests. Google’s release engineering account · Deployment pipeline patterns
During a canary rollout, the candidate serves a small share of requests. Its error rate can rise sharply while the fleet average barely moves because most requests still reach the existing version. Canarying releases · Limiting exposure
How a canary informs exposure
Version-separated observations reveal differences that can be hidden in a fleet average.
Read the diagram as text
- Production traffic.
- Candidate version.
- Control version.
- Assess rollout criteria.
- Expand exposure.
- Hold exposure steady.
- Halt and recover.
- Production traffic → Candidate version: Limited requests.
- Production traffic → Control version: Control requests.
- Candidate version → Assess rollout criteria: Candidate errors and latency.
- Control version → Assess rollout criteria: Control errors and latency.
- Assess rollout criteria → Expand exposure: Representative results meet criteria.
- Assess rollout criteria → Hold exposure steady: Sparse or confounded evidence.
- Assess rollout criteria → Halt and recover: Unacceptable regression.
Operational acceptance transfers continuing responsibility to people equipped to exercise it. Google’s site reliability engineering teams examine dependencies, monitoring, capacity, and emergency procedures before taking on a service, sometimes after launch. Training and access make the handoff usable. Recovery also includes effects outside the software version. Restoring an earlier package can prevent another erroneous message from being sent. Someone still has to decide what to do for the customers who received the first one. Production readiness and onboarding · Restoring software after side effects
Exception ownership and recovery
An exception is work that needs a remedy or decision outside its normal path. Preserve the work item, candidate or operation identifier, completed steps, observations, and remaining obligation. Assign it to someone who can change the blocking condition. Specify when that person must acknowledge it and what would permit work to resume. Bound repair attempts by time or count, and stop earlier if repeated attempts produce the same failure without new information. Acknowledgment matters: in PagerDuty, notifying a responder, claiming an open incident, and resolving it are three different events. An unacknowledged incident can continue up the escalation path. PagerDuty incident lifecycle · Operating an exception queue
| Blocking condition | Receiving owner | Basis for resuming or closing |
|---|---|---|
| Competing interpretations of intent | Requirement owner | A recorded choice and updated acceptance criterion |
| Candidate fails a behavior check | Implementer | A corrected revision and a rerun of the failed check |
| Test environment unavailable | Environment operator | A working environment, followed by a completed check on the candidate |
| Producer and consumer disagree | Interface maintainers | An agreed contract and evidence from a compatible combination |
| Action exceeds delegated permission | Owner authorized to decide | Scoped authorization, a revised assignment, or a refusal |
| Deployment request times out | Release operator | The original operation reconciled with the observed running state |
A timeout means the caller stopped waiting. The remote system may still have installed the package. Look up the original operation and inspect the running version before submitting another deployment. If it completed, resume verification there. If it partly completed, preserve what succeeded and identify the remaining effects. Compensation is a new action that remedies a completed effect while respecting changes that have happened since. Recovery may itself need to be resumed, so retain its progress too. Uncertain external outcomes · Compensating transactions
Improving the production process
Repairing a candidate resolves one assignment. Changing how subsequent assignments are selected, implemented, or assessed requires a diagnosis of the production process. A rejected patch may reveal a missed requirement. An escaped defect may reveal that the integration environment omitted a dependency. A successful delivery can help identify the conditions under which a check was informative. In each case, preserve the observation, test the proposed explanation, and give someone responsibility for the resulting change. Postmortems and owned follow-up · Observability · Testing a failure explanation
Uber’s uReview illustrates two uses of a review finding. A developer can act on a comment to fix a PR. The team operating the reviewer can use ratings and comment metadata to adjust a filter, prompt, or model. A developer’s usefulness rating tells the team something about a comment that appeared. It says little about an issue the reviewer missed. Uber also benchmarks against changes with human-annotated issues. Its report identifies a practical limit: source code alone may omit a configuration or schema needed to understand system behavior. Uber’s uReview engineering report
Testing a repair and testing a reviewer
ExampleThe case that prompted a revision and the cases reserved to judge it answer different questions.
Read the diagram as text
- Investigated failure.
- Reserved representative changes.
- Original reviewer.
- Revised reviewer.
- Known-case check.
- Paired assessment. Apply the same expectations to findings from both reviewers.
- Investigated failure → Revised reviewer: Diagnosis guides revision.
- Investigated failure → Known-case check: Case to reproduce.
- Revised reviewer → Known-case check: Revised finding on known case.
- Reserved representative changes → Original reviewer: Evaluate reserved changes.
- Reserved representative changes → Revised reviewer: Evaluate identical changes.
- Original reviewer → Paired assessment: Original findings on reserved work.
- Revised reviewer → Paired assessment: Revised findings on reserved work.
When changing the reviewer, keep the motivating failure as a regression case and check behaviors that worked before. To test a claim about future work, reserve fresh representative cases from the revision process. This reserved sample is a holdout. Run the original and revised configurations on the same cases under the same criteria, preserving both versions and their findings. Once a score influences the next prompt or model choice, the cases behind it have helped shape that choice even if nobody saw the individual answers. Use fresh evidence for the next independent comparison. Holdout reuse · Revalidating a changed system
Capacity and design
Throughput and admission limits
Choose a work item and a finish event before measuring a factory. Throughput counts distinct accepted completions in a stated period. Lead time measures elapsed time from a stated starting point to that finish, including waiting. Work in progress counts admitted items still unfinished. Track their age as well as the duration of completed work. A fast stream of easy completions can conceal one long-blocked item. Multiple attempts on that item consume resources while leaving its completion count unchanged. Flow measures in the Kanban Guide · Measuring the complete process
| Candidates arriving | Review slots | Accepted that day | Left waiting |
|---|---|---|---|
| 4 | 4 | 4 | 0 |
| 8 | 4 | 4 | 4 |
The four candidates left waiting become part of the next day’s workload. A team can respond by starting fewer changes until reviewers have room. This is backpressure. In practice, a large change takes more review time than a small one, and a repair may need another review and another CI run. When choosing what to start, the team can use capacity for work that can finish while a higher-priority item waits for a scheduled check, provided it can still meet that appointment. The constraint can also lie beyond review. Uber’s practitioners describe being able to build more product variants than they can feasibly test in experiments.
DevOps Research and Assessment (DORA) measures change lead time from commit to production deployment. That excludes time spent clarifying the request before a commit exists. Its change fail rate captures deployments requiring immediate intervention, while deployment rework rate captures unplanned deployments caused by incidents. Uber’s managed-agent accounting uses still other boundaries, such as cost per review, merged PR, or resolved alert. To compare useful deliveries, follow comparable requests through acceptance and include reviewer effort, integration resources, reverts, escaped defects, and recovery. A migration needing five repository PRs may still fulfill just one request. DORA delivery metrics · Uber’s operating account · AI Cost and Performance Engineering
In early 2025, METR randomized AI availability across 246 tasks performed by 16 experienced open-source contributors in familiar repositories. Using the February–June tools tested, primarily Cursor Pro and Claude 3.5 or 3.7 Sonnet, contributors took 19% longer on tasks where AI was allowed than on tasks without it. They believed the tools had saved time. The unit measured was assisted task completion. In a February 2026 update, METR explained why a later experiment was harder to interpret: contributors withheld tasks or declined participation, some tasks remained incomplete, reported quality differed, and developers did other work during agent runs. Comparing factories requires following unfinished assignments too, and recording human labor separately from elapsed time. METR experiment · METR study-design update
Comparing factory boundaries
An automation boundary is the point at which the system hands a result to someone with the means and authority to act on it. Davis Palmie of Factory proposes a narrow starting point for incident response. An agent reads a Sentry alert, gathers traces of the operations involved, and posts a diagnosis in Slack. An engineer can inspect whether the traces support the diagnosis before committing anyone to a repair. That makes investigation a useful assignment even while implementation and release remain with the service team. Palmie’s signal-to-deployment account
| System and starting point | Artifact and assessment | Boundary of delegated authority |
|---|---|---|
| Factory · Reyes’s repository request | Droid returned a PR after clarification and project checks. | A maintainer could review the change. Release was outside the demonstration. |
| Warp · public issues and contributions | A public dashboard shows assignments and issue state. Lloyd’s design adds specification, code, and product reviews. | Maintainers can intervene at those reviews. The proposed loop leaves detailed release and exception permissions unspecified. |
| Uber · changes, CI failures, and operational signals | Managed agents perform reviews, repairs, triage, and maintenance. Uber tracks outcomes by assignment alongside quality signals. | People review work or receive escalations. The operating report leaves final release authority and ownership of uncertain deployments unspecified. |
| SDC · customer-site system design | Central factory staff performed detailed design, programming, and tests. | System design stayed with customer-site managers. Final release authority is unspecified. |
| Google Rapid · selected integration revision | A named build advanced through configured tests and deployment stages. | Project rules and role permissions governed promotion. Rollout could vary by service risk. |
Extending an alert investigation to a checked patch takes write access to a bounded workspace, a way to test the repair, and someone able to review it. Extending automation through release also takes evidence about the combined build, permission to expose that class of change, and an operator able to stop the rollout. Review capacity limits how many requests a team can undertake. One interface review may clear a shared dependency for several requests. Each request counts as delivered when its own recipient gets the promised result. Assigning decision rights
Reference questions
Frequently asked questions
How does a software factory relate to DevOps and CI/CD?
A factory organizes repeated software production. DevOps brings development and operations into a shared working relationship. Continuous integration checks frequently combined changes. Continuous delivery keeps a version releasable on demand, and continuous deployment releases passing changes automatically. A factory can use all of these practices. Continuous integration and delivery
Does a factory require several agents?
No. One agent can take an issue and return a draft for tools to check and a maintainer to review. Add workers when their assignments can proceed independently and the team can afford to check the combination. Comparing multi-agent designs
Can work start with incomplete requirements?
Yes. Scope the assignment to something the receiver can already judge, such as identifying consumers of an interface or comparing two designs. Ask the authorized owner to settle consequential choices before implementation depends on them. Factory Missions separates clarification and plan approval from execution. Missions planning workflow
What do passing tests establish?
A passing test shows that the version tested met the stated expectation under the conditions the test exercised. Its setup matters. A test that puts a saved draft in storage and reopens it checks that the editor can read the draft. To assess crash recovery, interrupt an actual editing session and check whether the writer can find and reopen the expected edits. Matching checks to claims
Which changes can run unattended?
An owner can delegate a defined action for an eligible class of changes. The policy should identify the evidence required, where automation stops, and who handles an exception. Permission to open a checked draft can cover a wider class of work than permission to promote it. Release policies in Rapid
Who owns a bad release?
The organization that authorized exposure remains accountable. Its release and service owners need permission to stop rollout, access to restore service, and responsibility for effects already experienced by users. A receiving team must explicitly accept its operating duties. Ownership and authority · Staffing for the required capability
Does more generated code demonstrate value?
More generated code establishes that implementation produced more text. To assess value, follow distinct requests to their agreed finish, observe whether the result serves its intended use, and count waiting, repairs, human effort, and operating failures. One migration spread over several repositories can increase PR volume while delivering a single outcome. Measuring the complete process
Open questions
Which classes of work benefit when automation extends from a checked draft to delivery? Selective assignment can hide difficult tasks, and concurrent agent runs separate elapsed time from human effort. Following comparable requests, including unfinished ones, through acceptance, operating cost, and subsequent repairs would help teams choose where to stop.
When can a factory safely reuse verification after a base revision or dependency changes? Testing every possible combination spends scarce integration capacity, but a missed consumer can break a shared release. Comparing selective checks with full combined runs on the same changes would expose both missed interactions and the cost in builds and waiting.
How can a reviewer improve detection without exhausting the people who act on its findings? Ratings describe visible comments, while omissions can surface much later. Paired reviews on fresh, independently annotated work, coupled with observed human review effort, would help distinguish better detection from a larger volume of plausible comments.


















































































