AI Engineer World's Fair 2025

Unlocking AI Powered DevOps Within Your Organization

Jon Peck22:13

Loading slides…

Read the talk

Making AI Useful Across the DevOps Lifecycle

AI adoption starts with better delivery, not more generated code: scope work carefully, share team knowledge, and extend automation through reviewed, recoverable changes.

From a talk by Jon Peck

Before you start: Familiarity with Git branches, pull requests, IDEs, and basic CI/CD workflows will help you follow the examples.

Does more code deliver more features?

An LLM can produce a huge amount of code. Does that actually increase the number of features a team delivers? That is the opening problem for Jon Peck, a GitHub developer advocate who has written software since the late 1990s and spent roughly a decade in developer advocacy. Once developers accept that models can perform some tasks faster, the organizational challenge becomes choosing workflows that make that speed useful. Generated code is an input; delivered functionality is the outcome.

Peck reports best-case feature-point throughput of up to 1.5× in the same period, with improvements nearer 30% in most companies he observes. These are his observations; he does not give a company sample, measurement period, or definition of feature points. He also describes more successful builds against existing pipelines and tests, attributing that improvement to catching problems earlier, alongside increased developer happiness. Those measures ask whether AI improves an established engineering process rather than merely increasing its output volume. The accompanying slides are available through the short link shown in the recording.

Slide with four percentage-based metric cards and a purple horizontal bar chart of developer experiences.Open full source frame
AI development benefits across activity, productivity, efficiency, and satisfaction.
0:170:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Keep context and model choice inside the workflow

The simplest way to try AI is to open a chat tool, ask for code, and paste the response into an IDE. It is also a cumbersome way to iterate: relevant context has to cross the same boundary repeatedly, and moving code into a separate service introduces exposure concerns. An integrated assistant makes it easier to select the files a task needs, including through drag-and-drop, without treating the entire repository as relevant to every request. Peck uses VS Code with GitHub Copilot, explicitly noting that GitHub is his employer.

Integration should preserve model choice. In the workflow Peck describes, orchestration is not yet reliable enough to select the right model every time, so the developer still chooses according to the task. A subscription that exposes several models makes that choice practical.

TaskModel choiceReason
Quick, inexpensive responsesFlash modelFavor speed and low cost
Explore a problem before codingReasoning modelSpend more effort on research and analysis

The developer remains the operator, interacting with the IDE and deciding when a task needs a different kind of model.

Slide showing a Copilot model dropdown beside Free, Pro, and Pro+ model availability columns.Open full source frame
Choosing the right model for the right task in GitHub Copilot.
2:052:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:05 · section reference included

Learn on the work you already do

Greenfield experiments are tempting: ask for Minecraft built on the Doom engine and running on a dishwasher, then watch the model attempt something extraordinary. Peck's deliberately absurd example captures the appeal, but it does little to teach the patterns most developers need every day. Most work is brownfield development: changing an application that already has substantial code, conventions, and dependencies.

Start with an ORM version change, or generate tests for existing modules using a specified data source. These tasks force the useful pairing: a prompt that describes the intended change and context that makes the change possible. Repeated daily use teaches developers which files, examples, and constraints to supply. Once those habits are established, greenfield experiments become a way to apply them to building something new.

3:423:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:42 · section reference included

Use agent mode to scope, execute, and recover

Agent mode helps with both new applications and existing ones, but the first useful action need not be writing code. For a greenfield build, Peck starts with a Markdown README and uses the model to develop the specification:

  1. Describe the kind of application and ask which specifications it needs.
  2. Review the initial proposal and replace weak assumptions with explicit technology choices.
  3. Define the directory structure and provide examples of expected API calls.
  4. Refine the document until it is clear enough to hand to another engineer, then ask the model to implement it.

The specification is the handoff. Asking the model to help shape it is different from accepting the model's first interpretation and immediately starting a build.

For brownfield work, the first task is narrowing context. In the ORM migration example, Peck selects the models folder, the Python Flask backend, and a few settings files. This serves two purposes: it keeps relevant material small enough for the model call and directs attention away from unrelated parts of the application. The agent should then narrow that selection further, explain its plan, and work through the plan's separate parts.

Supervision continues during execution. Pause an agent that is taking the wrong path, give it corrective instructions, and resume. If the approach itself is wrong, discard it—even if it has already made hundreds of changes. Peck uses the example of reverting 500 changes to make the point that cheap regeneration changes the economics of throwing code away. Commit working checkpoints so recovery returns to something known to work, and ask the agent to maintain a file-based change log so its edits remain easy to inspect.

5:075:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:07 · section reference included

Make team standards available to the model

Team practices do not disappear when an assistant joins the workflow. Dependency files, manuals, standards, and playbooks still matter. A committed Copilot instructions file adds a way to supply those conventions to the model: .github/copilot-instructions.md. Use it locally and commit it so colleagues share the same guidance. The lead developer or manager should update it when the team changes linting practices, model accessor patterns, or checks for recurring problems. These instructions supply context; they do not guarantee compliance, so review and automated checks remain necessary.

For the ORM migration workflow, a concise instruction file could carry the relevant working conventions:

markdown

# Repository working conventions

- Follow the repository's committed dependency and lint configuration.
- Use the existing model accessor patterns.
- Before an ORM migration, inspect the models, Flask backend, and relevant settings.
- Explain the migration plan before editing.
- Keep changes focused on the requested migration.
- Check the affected behavior with the existing tests.
- Maintain a change log describing edits and remaining work.

The value comes from maintaining the file as team decisions change, rather than treating it as a one-time setup artifact.

8:048:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:04 · section reference included

Share institutional knowledge and solved failures

Repository instructions describe how the team works; institutional knowledge supplies details a model cannot infer from general training. Provide API specifications and examples of established implementation patterns. In the historical Copilot Enterprise workflow Peck describes, knowledge bases were named collections of repositories that an operator could add to a task. His examples include one to three repositories demonstrating accessible frontends for blind operators, Python practices for particular machine-learning tasks, and definitions of internal APIs. Adding the accessibility collection or internal finance API collection makes those examples available while the developer works. This named knowledge-base product was subsequently replaced by Copilot Spaces; the underlying practice is still to supply the relevant internal knowledge explicitly.

Knowledge also emerges from mistakes. Make it acceptable to admit that a model failed, ask a question, or bring an unresolved context problem to the team. Share the problem in Slack or Teams while it is still useful to discuss. Once solved, move the result into a centralized guide, repository, or team knowledge store. Otherwise, the solution remains trapped in a conversation that the next developer may never find.

Watch adoption with the same supportive intent. Peck points to Copilot metrics APIs, internal usage tracking, and the volume and type of submitted code as possible signals. The purpose is to notice colleagues who lack access, have not encountered a useful pattern, or feel hesitant—not to assume that everyone learns a new workflow at the same pace. Use the observations to decide where help is needed.

9:149:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:14 · section reference included

Separate privacy guarantees from context controls

Organizational adoption needs explicit safety and privacy decisions. Peck recommends evaluating proxies, sanitization, and provider guarantees around reuse of submitted data. He describes GitHub enterprise protections in terms of indemnification, opt-in tracking, no retraining on enterprise data, and discarded prediction context. Those are distinct contractual and technical questions: no training does not mean no retention. Current model-hosting documentation confirms that Business and Enterprise customer data is not used for training, but describes provider-specific caching, retention agreements, and exceptions. Indemnification and telemetry should be checked against the applicable terms rather than inferred from the no-training policy.

Peck next describes repository-level exclusions intended to keep sensitive files, such as environment secrets, out of predictions. Content exclusion has important surface-specific limits: current documentation says editor Edit and Agent modes do not support it, and other limitations can allow indirect information to enter context. Treat it as a supported-surface control, not a universal secret boundary. At the organization level, centralize policies for model access and new features so each repository maintainer does not have to reproduce the same governance work.

12:0012:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:00 · section reference included

Look beyond production-code generation

Application coding is only one part of DevOps, and Peck expects some other activities to yield larger early gains. Planning is the first: use a model to explore a feature or product, compare technology choices, and consider infrastructure before implementation commits the team to a direction. This extends the earlier specification workflow from the shape of a single application to the larger plan for delivering it.

Security is another entry point. Ask the assistant to inspect an existing codebase for problematic dependencies, unsafe patterns, and potential cross-site scripting. The selected slide shows a Copilot response listing potential vulnerabilities and suggested fixes. This gives the developer concrete findings to investigate and remediate within the existing application.

DevOps stages with Secure underlined beside a Copilot response listing potential vulnerabilities and suggested fixes.Open full source frame
Using Copilot to identify potential security vulnerabilities.
13:1913:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:19 · section reference included

Build the artifacts that coding leaves behind

Testing is Peck's second-largest personal speed gain. Coverage work often accumulates in a backlog, leaving existing modules without the cases developers intended to write. His workflow supplies three things together: selected modules, representative data, and the desired test framework. Drag the modules into context, add JSON samples or database examples through MCP, and ask for unit tests using the chosen framework. Peck reports getting a functioning test skeleton in about 10% of his usual test-writing time, with manual updates still needed. The reported result is a scaffold to refine, not a completed and validated test suite.

Deployment extends assistance to another set of artifacts. Once the application is ready, ask how to deploy it to the intended platform and generate the relevant GitHub Actions workflow or Terraform files. This is also a reason to include infrastructure specialists and other roles in AI adoption: they can benefit without spending their day implementing application features.

Documentation is Peck's largest personal speed gain. A brief documentation pass after coding gives both users and documentation specialists something concrete to work from. He suggests spending about 15 minutes asking for several complementary outputs:

  • Application description: A Markdown explanation of what the application does.
  • API specification: A fuller description of the exposed interface.
  • Executive summary: A higher-level account for readers who do not need implementation details.
  • Interaction examples: Concrete examples of using the application.
  • Workflow diagrams: Graphs with a vision-capable model, or Mermaid charts, as Peck proposes.

These drafts give the people curating company documentation a head start instead of leaving them to reconstruct the application after the code is finished.

14:2014:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:20 · section reference included

Move execution off the developer's critical path

An IDE agent that pauses for terminal permission or plan approval is still semi-autonomous. Moving work into a more autonomous mode should save time without removing the human decision about whether the result belongs in the product. In Peck's GitHub example, the agent creates a new branch and runs in a protected environment. Its work stays isolated while a person reviews the branch and decides whether the pull request can be completed and merged. Autonomous execution produces a proposal; review authorizes integration.

A low-friction starting point is generating a pull-request description after the work is done. Copilot can supply a high-level summary followed by details about key files, giving downstream readers a more useful account of the change. From there, the workflow expands to code review. Peck describes that capability as having arrived roughly six months before the talk, without establishing a precise public launch date.

Request Copilot review on an existing PR, regardless of how the PR was created. It works asynchronously and leaves comments and suggested changes in the familiar review workflow. A proposed rewrite to make a function more efficient is not automatically inserted: the operator accepts or rejects it, just as with another developer's suggestion. The developer can work elsewhere and return later to inspect the comments. Peck's example of coming back half an hour later illustrates that asynchronous pattern, not a promised review latency.

16:3516:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:35 · section reference included

Delegate a bounded issue, then review the result

Assigning an issue to Copilot moves from reviewing existing work to producing a change. Choose the scope carefully. Missing documentation, missing tests, or one clearly defined function are more suitable starting points than a giant issue with many loosely related requirements. A useful issue resembles a useful prompt: identify the focus, exact actions, expected outcomes, and tests the agent should produce.

The issue-to-PR loop preserves the normal delivery process:

  1. Delegate: Assign the bounded issue. Copilot creates a branch and PR, then works toward a proposed solution.
  2. Inspect: Review the result the agent considers complete.
  3. Restart or revise: If the approach is wrong, discard the PR, refine the issue, and reassign it. If the problem is small, comment on the PR so the agent can modify the branch.
  4. Validate and merge: Run the standard CI/CD checks and merge only when the result is acceptable.

The same recovery principle used in the IDE applies here: a bad generated solution does not become worth preserving merely because it contains a lot of work.

19:0819:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:08 · section reference included

Connect the agent to the rest of the workflow

Model Context Protocol (MCP) adds connections to systems outside the immediate code context. Peck first describes configuring servers in VS Code and choosing integrations relevant to the team's technologies and tasks. He points to the MCP servers repository; today it emphasizes educational reference implementations and directs broader discovery to the MCP Registry.

With GitHub's MCP server, the agent can carry a finished local change into the collaboration workflow: create a branch, make a sensible commit, open a pull request, and request an approver. That avoids switching to the GitHub web interface to repeat the handoff manually. Peck then extends the example to repository-level MCP configuration for Copilot Enterprise agentic work: an agent assigned to an issue can use those configured connections to reach other systems while completing the task. This is the substantive expansion at the end of the talk—from generating code to carrying out connected engineering work under the same review boundaries.

The closing resource slide collects training material, cheat sheets, a FAQ, learning pathways, blogs, previews, research, and YouTube links. It provides a place to continue learning after the workflow examples, before Peck closes with an invitation to continue the conversation in the hallway.

Resource slide listing quickstart, cheat sheet, FAQ, learning pathway, blogs, previews, research, and YouTube links.Open full source frame
GitHub Copilot resources for continued learning.
20:3620:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:36 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…