AI Engineer World's Fair 2025
Unlocking AI Powered DevOps Within Your Organization
Loading slides…
Read the talk
Making AI Useful Across the DevOps Lifecycle
AI adoption starts with better delivery, not more generated code: scope work carefully, share team knowledge, and extend automation through reviewed, recoverable changes.
From a talk by Jon Peck
Before you start: Familiarity with Git branches, pull requests, IDEs, and basic CI/CD workflows will help you follow the examples.
Does more code deliver more features?
An LLM can produce a huge amount of code. Does that actually increase the number of features a team delivers? That is the opening problem for Jon Peck, a GitHub developer advocate who has written software since the late 1990s and spent roughly a decade in developer advocacy. Once developers accept that models can perform some tasks faster, the organizational challenge becomes choosing workflows that make that speed useful. Generated code is an input; delivered functionality is the outcome.
Peck reports best-case feature-point throughput of up to 1.5× in the same period, with improvements nearer 30% in most companies he observes. These are his observations; he does not give a company sample, measurement period, or definition of feature points. He also describes more successful builds against existing pipelines and tests, attributing that improvement to catching problems earlier, alongside increased developer happiness. Those measures ask whether AI improves an established engineering process rather than merely increasing its output volume. The accompanying slides are available through the short link shown in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep context and model choice inside the workflow
The simplest way to try AI is to open a chat tool, ask for code, and paste the response into an IDE. It is also a cumbersome way to iterate: relevant context has to cross the same boundary repeatedly, and moving code into a separate service introduces exposure concerns. An integrated assistant makes it easier to select the files a task needs, including through drag-and-drop, without treating the entire repository as relevant to every request. Peck uses VS Code with GitHub Copilot, explicitly noting that GitHub is his employer.
Integration should preserve model choice. In the workflow Peck describes, orchestration is not yet reliable enough to select the right model every time, so the developer still chooses according to the task. A subscription that exposes several models makes that choice practical.
| Task | Model choice | Reason |
|---|---|---|
| Quick, inexpensive responses | Flash model | Favor speed and low cost |
| Explore a problem before coding | Reasoning model | Spend more effort on research and analysis |
The developer remains the operator, interacting with the IDE and deciding when a task needs a different kind of model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn on the work you already do
Greenfield experiments are tempting: ask for Minecraft built on the Doom engine and running on a dishwasher, then watch the model attempt something extraordinary. Peck's deliberately absurd example captures the appeal, but it does little to teach the patterns most developers need every day. Most work is brownfield development: changing an application that already has substantial code, conventions, and dependencies.
Start with an ORM version change, or generate tests for existing modules using a specified data source. These tasks force the useful pairing: a prompt that describes the intended change and context that makes the change possible. Repeated daily use teaches developers which files, examples, and constraints to supply. Once those habits are established, greenfield experiments become a way to apply them to building something new.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use agent mode to scope, execute, and recover
Agent mode helps with both new applications and existing ones, but the first useful action need not be writing code. For a greenfield build, Peck starts with a Markdown README and uses the model to develop the specification:
- Describe the kind of application and ask which specifications it needs.
- Review the initial proposal and replace weak assumptions with explicit technology choices.
- Define the directory structure and provide examples of expected API calls.
- Refine the document until it is clear enough to hand to another engineer, then ask the model to implement it.
The specification is the handoff. Asking the model to help shape it is different from accepting the model's first interpretation and immediately starting a build.
For brownfield work, the first task is narrowing context. In the ORM migration example, Peck selects the models folder, the Python Flask backend, and a few settings files. This serves two purposes: it keeps relevant material small enough for the model call and directs attention away from unrelated parts of the application. The agent should then narrow that selection further, explain its plan, and work through the plan's separate parts.
Supervision continues during execution. Pause an agent that is taking the wrong path, give it corrective instructions, and resume. If the approach itself is wrong, discard it—even if it has already made hundreds of changes. Peck uses the example of reverting 500 changes to make the point that cheap regeneration changes the economics of throwing code away. Commit working checkpoints so recovery returns to something known to work, and ask the agent to maintain a file-based change log so its edits remain easy to inspect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share institutional knowledge and solved failures
Repository instructions describe how the team works; institutional knowledge supplies details a model cannot infer from general training. Provide API specifications and examples of established implementation patterns. In the historical Copilot Enterprise workflow Peck describes, knowledge bases were named collections of repositories that an operator could add to a task. His examples include one to three repositories demonstrating accessible frontends for blind operators, Python practices for particular machine-learning tasks, and definitions of internal APIs. Adding the accessibility collection or internal finance API collection makes those examples available while the developer works. This named knowledge-base product was subsequently replaced by Copilot Spaces; the underlying practice is still to supply the relevant internal knowledge explicitly.
Knowledge also emerges from mistakes. Make it acceptable to admit that a model failed, ask a question, or bring an unresolved context problem to the team. Share the problem in Slack or Teams while it is still useful to discuss. Once solved, move the result into a centralized guide, repository, or team knowledge store. Otherwise, the solution remains trapped in a conversation that the next developer may never find.
Watch adoption with the same supportive intent. Peck points to Copilot metrics APIs, internal usage tracking, and the volume and type of submitted code as possible signals. The purpose is to notice colleagues who lack access, have not encountered a useful pattern, or feel hesitant—not to assume that everyone learns a new workflow at the same pace. Use the observations to decide where help is needed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate privacy guarantees from context controls
Organizational adoption needs explicit safety and privacy decisions. Peck recommends evaluating proxies, sanitization, and provider guarantees around reuse of submitted data. He describes GitHub enterprise protections in terms of indemnification, opt-in tracking, no retraining on enterprise data, and discarded prediction context. Those are distinct contractual and technical questions: no training does not mean no retention. Current model-hosting documentation confirms that Business and Enterprise customer data is not used for training, but describes provider-specific caching, retention agreements, and exceptions. Indemnification and telemetry should be checked against the applicable terms rather than inferred from the no-training policy.
Peck next describes repository-level exclusions intended to keep sensitive files, such as environment secrets, out of predictions. Content exclusion has important surface-specific limits: current documentation says editor Edit and Agent modes do not support it, and other limitations can allow indirect information to enter context. Treat it as a supported-surface control, not a universal secret boundary. At the organization level, centralize policies for model access and new features so each repository maintainer does not have to reproduce the same governance work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look beyond production-code generation
Application coding is only one part of DevOps, and Peck expects some other activities to yield larger early gains. Planning is the first: use a model to explore a feature or product, compare technology choices, and consider infrastructure before implementation commits the team to a direction. This extends the earlier specification workflow from the shape of a single application to the larger plan for delivering it.
Security is another entry point. Ask the assistant to inspect an existing codebase for problematic dependencies, unsafe patterns, and potential cross-site scripting. The selected slide shows a Copilot response listing potential vulnerabilities and suggested fixes. This gives the developer concrete findings to investigate and remediate within the existing application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the artifacts that coding leaves behind
Testing is Peck's second-largest personal speed gain. Coverage work often accumulates in a backlog, leaving existing modules without the cases developers intended to write. His workflow supplies three things together: selected modules, representative data, and the desired test framework. Drag the modules into context, add JSON samples or database examples through MCP, and ask for unit tests using the chosen framework. Peck reports getting a functioning test skeleton in about 10% of his usual test-writing time, with manual updates still needed. The reported result is a scaffold to refine, not a completed and validated test suite.
Deployment extends assistance to another set of artifacts. Once the application is ready, ask how to deploy it to the intended platform and generate the relevant GitHub Actions workflow or Terraform files. This is also a reason to include infrastructure specialists and other roles in AI adoption: they can benefit without spending their day implementing application features.
Documentation is Peck's largest personal speed gain. A brief documentation pass after coding gives both users and documentation specialists something concrete to work from. He suggests spending about 15 minutes asking for several complementary outputs:
- Application description: A Markdown explanation of what the application does.
- API specification: A fuller description of the exposed interface.
- Executive summary: A higher-level account for readers who do not need implementation details.
- Interaction examples: Concrete examples of using the application.
- Workflow diagrams: Graphs with a vision-capable model, or Mermaid charts, as Peck proposes.
These drafts give the people curating company documentation a head start instead of leaving them to reconstruct the application after the code is finished.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move execution off the developer's critical path
An IDE agent that pauses for terminal permission or plan approval is still semi-autonomous. Moving work into a more autonomous mode should save time without removing the human decision about whether the result belongs in the product. In Peck's GitHub example, the agent creates a new branch and runs in a protected environment. Its work stays isolated while a person reviews the branch and decides whether the pull request can be completed and merged. Autonomous execution produces a proposal; review authorizes integration.
A low-friction starting point is generating a pull-request description after the work is done. Copilot can supply a high-level summary followed by details about key files, giving downstream readers a more useful account of the change. From there, the workflow expands to code review. Peck describes that capability as having arrived roughly six months before the talk, without establishing a precise public launch date.
Request Copilot review on an existing PR, regardless of how the PR was created. It works asynchronously and leaves comments and suggested changes in the familiar review workflow. A proposed rewrite to make a function more efficient is not automatically inserted: the operator accepts or rejects it, just as with another developer's suggestion. The developer can work elsewhere and return later to inspect the comments. Peck's example of coming back half an hour later illustrates that asynchronous pattern, not a promised review latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delegate a bounded issue, then review the result
Assigning an issue to Copilot moves from reviewing existing work to producing a change. Choose the scope carefully. Missing documentation, missing tests, or one clearly defined function are more suitable starting points than a giant issue with many loosely related requirements. A useful issue resembles a useful prompt: identify the focus, exact actions, expected outcomes, and tests the agent should produce.
The issue-to-PR loop preserves the normal delivery process:
- Delegate: Assign the bounded issue. Copilot creates a branch and PR, then works toward a proposed solution.
- Inspect: Review the result the agent considers complete.
- Restart or revise: If the approach is wrong, discard the PR, refine the issue, and reassign it. If the problem is small, comment on the PR so the agent can modify the branch.
- Validate and merge: Run the standard CI/CD checks and merge only when the result is acceptable.
The same recovery principle used in the IDE applies here: a bad generated solution does not become worth preserving merely because it contains a lot of work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect the agent to the rest of the workflow
Model Context Protocol (MCP) adds connections to systems outside the immediate code context. Peck first describes configuring servers in VS Code and choosing integrations relevant to the team's technologies and tasks. He points to the MCP servers repository; today it emphasizes educational reference implementations and directs broader discovery to the MCP Registry.
With GitHub's MCP server, the agent can carry a finished local change into the collaboration workflow: create a branch, make a sensible commit, open a pull request, and request an approver. That avoids switching to the GitHub web interface to repeat the handoff manually. Peck then extends the example to repository-level MCP configuration for Copilot Enterprise agentic work: an agent assigned to an issue can use those configured connections to reach other systems while completing the task. This is the substantive expansion at the end of the talk—from generating code to carrying out connected engineering work under the same review boundaries.
The closing resource slide collects training material, cheat sheets, a FAQ, learning pathways, blogs, previews, research, and YouTube links. It provides a place to continue learning after the workflow examples, before Peck closes with an invitation to continue the conversation in the hallway.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Educational MCP server implementations and links to further server-discovery resources.
GitHub’s official MCP integration for connecting AI tools to GitHub workflows.
Further reading
Repository instructions and other ways to supply team conventions to Copilot.
Updates since the talk
Supported exclusion controls and their limitations, including lack of support in editor Edit and Agent modes.
Full transcript
Loading transcript…