AI Engineer Summit 2025
The Devops Engineer Who Never Sleeps
Loading slides…
Read the talk
The DevOps Engineer Who Never Sleeps
Datadog’s agent workflows show how incident investigation, code repair, evaluation and observability fit together—and where human verification still matters.
From a talk by Diamond Bishop
Before you start: Familiarity with alerts, runbooks, logs, metrics, traces and pull requests will help you follow the workflows.
From observing a system to operating it
What changes when an observability platform can investigate the problems it exposes? Diamond Bishop approaches that question through Bits AI, Datadog’s assistant for DevOps work. His background spans Cortana at Microsoft, Alexa at Amazon, PyTorch at Meta and a startup building a DevOps assistant. The underlying job at Datadog connects two activities: understanding what is happening in cloud applications and taking action to make those systems safer and easier to operate.
AI already supports parts of that job: proactive alerting, root-cause analysis, impact analysis and change tracking. Bishop dates Datadog’s broader AI work to around 2015; that is a recollection of the program’s history, not a common release date for every capability. Stronger foundation models, emerging reasoning and multimodal capabilities, and rising expectations from products such as Cursor create an opportunity to move further up the stack.
The next product layer is an agent that uses the platform on the customer’s behalf. Bishop compares the potential shift to the microprocessor and SaaS eras, while acknowledging that its direction remains uncertain. Turning that opportunity into a product requires three kinds of work together: developing agents, evaluating their behavior and building observability for the workflows they execute.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An investigation starts with the alert
At the time of the talk, Datadog had two agents in private beta. The AI Software Engineer examines errors and recommends generated code changes. The AI On-Call Engineer investigates operational problems, aiming to reduce the occasions when an engineer must wake up and start from scratch. These are the historical product names used in the demonstration.
The concrete problem is the 2 a.m. alert. Normally, someone wakes up, opens a runbook and begins figuring out what happened. The On-Call Engineer starts proactively when the alert occurs, following the same broad investigative sequence:
- Orient to the incident. Read the alert context and relevant runbooks to establish the problem.
- Gather service knowledge. Collect the context needed to interpret the alert.
- Investigate iteratively. Query logs, metrics and traces, using what turns up to guide the next step.
The goal is to have useful investigative work underway before a human reaches a computer.
That work can support both automatic investigations and an engineer seeking a specific explanation. The agent gathers findings and summaries about why an alert fired or why a trace shows an error. Reducing overnight paging is the intended benefit; starting an investigation does not itself establish that the incident has been resolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the investigation inspectable
An agent working independently still needs a place where people can inspect its work. Datadog’s collaboration page exposes why a hypothesis was generated and what evidence the agent found. An engineer can follow the investigation and decide whether to agree with its conclusions. Trust depends on being able to verify the work, rather than merely receiving a confident summary.
The page also shows which runbook steps the agent actually took. A human can ask follow-up questions about why it chose a particular action, much as they would supervise a junior engineer. That analogy describes the review relationship: the agent does work, makes it inspectable and remains open to questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From hypotheses to remediation and a postmortem
Underneath that interface is a hypothesis-testing loop. The agent proposes explanations, reasons about how to test them and uses tools to query logs, metrics and other evidence. Bishop describes this as tool use in the Toolformer sense; the reference is an analogy for using tools, not a claim that Datadog implements the paper’s training method. Query results help the agent validate or invalidate each candidate explanation.
When the evidence supports a root cause, the agent can suggest a response: page another team, or offer to scale infrastructure up or down. Bishop distinguishes those suggestions from future plans to expand the available actions and discover workflows from a team’s behavior. Existing Datadog workflows already provide an integration point: the agent can understand a configured workflow and map it to a remediation need. A suggested response is not evidence that the change has been executed.
Incident work continues after remediation. Engineers reconstruct what happened, write a postmortem and share what they learned. The agent can assemble that account from the incident history, including both its own actions and the actions humans took, leaving a postmortem ready for morning review. This extends the workflow from immediate investigation into the team’s longer-term learning process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn observed errors into proposed code repairs
The AI Software Engineer addresses the preventive side of operations. It watches errors coming through error tracking, analyzes their causes and proposes solutions, including generated code fixes. The two agents can therefore complement each other: one investigates alerts, while the other works on defects that could otherwise produce future incidents.
In the demonstration, the assistant identifies a recursion issue and proposes a fix. It also generates a regression test intended to catch the problem if it returns. The engineer can then create a GitHub pull request or open the diff in VS Code for editing. The handoff is a proposed change and test that an engineer can review; the presentation does not establish that the patch was merged or deployed.
Bishop reports that this workflow reduces the time engineers spend manually writing and testing code, but provides no measured reduction or comparison baseline. The concrete contribution shown is the package of diagnosis, proposed repair, regression test and an entry point into the engineer’s existing development workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define work that people can evaluate
Building these agents involved revisiting early decisions. Bishop groups the lessons into task scoping and evaluation, team composition, changing UX, and observability. The first difficulty is the gap between a convincing demo and a task whose performance can be evaluated repeatedly.
Start with the human job: what would someone do, step by step, and how would another person judge whether they did it well? That framing leads Datadog toward vertical, task-specific agents rather than a generalized agent with an open-ended remit. Make each step measurable and verifiable wherever possible, so improvement has an observable meaning.
Domain experts belong in that process as design partners and task verifiers. They help define useful work and recognize a sound result. Bishop cautions against treating their expertise as a set of rules to transcribe directly into the agent: the way an expert reasons about a domain does not necessarily map cleanly onto how a stochastic model behaves.
Evaluation should precede the impressive demo. Bishop attributes avoidable mistakes to not thinking about it early enough. Even a small initial evaluation is useful, provided it becomes part of a continuing system:
- Offline evaluation: assess the agent against defined tasks outside live use.
- Online evaluation: instrument product use and gather human feedback.
- Living evaluation: keep updating the test set as real use exposes new cases.
Include end-to-end tasks and measurements as well as checks on individual steps. A sequence of plausible intermediate actions is not enough to establish that the whole job succeeded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a team that can experiment and design collaboration
Bishop’s suggested team starts with one or two ML experts and adds strong coding generalists who are optimistic about experimentation. A whole organization of specialists is neither necessary nor easy to hire. Front-end and UX skills matter because the product’s usefulness depends on how people collaborate with its agents, not only on the quality of the backend reasoning.
He also looks for people who use AI in their own daily work and want to explore new capabilities. In a field changing this quickly, willingness to learn and try things is part of the engineering requirement. A team reluctant to adapt can become stuck even while the available models improve.
The interaction design remains unsettled. Bishop describes UX as an area his team continues to revisit and says he currently favors agents that behave more like human teammates over adding more pages and buttons. That is a design preference, not a universal result: the underlying requirement is to make collaboration useful while accepting that established interface patterns may change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Observe the workflow, not just the model calls
An agent investigating someone else’s system is itself a system that needs debugging. Observability supplies the situational awareness required to understand these complex workflows, so it cannot be an afterthought. Datadog’s LLM Observability brings model interactions into the wider view of the application and its infrastructure, including GPU monitoring. Calls to models the team hosts or operates and models accessed through APIs can be examined together.
Collecting those calls does not automatically make the execution understandable. Bishop says Datadog’s agent can make hundreds of calls in a workflow, with repeated loops and decisions about tools. That is an account of their agent’s possible workload, not a benchmark. A flat list makes it difficult to see how those decisions connect.
The talk then previews an agent graph that exposes the workflow’s structure. In the displayed graph, a bright red node draws attention to an error; zooming into it reveals where the failure occurred. The useful change is the unit of inspection: instead of reading an undifferentiated sequence of calls, an engineer can navigate the execution as a connected workflow. Bishop presents this view as a sneak peek.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep improved models easy to try
Bishop calls his next lesson an application-layer adaptation of The Bitter Lesson: general approaches that can adopt better off-the-shelf models may outlast substantial task-specific investment. A team can spend time fine-tuning or constructing specialized behavior, only for a new foundation model to handle much of that reasoning directly. This is his engineering lesson, rather than a measured comparison supplied in the talk.
The practical requirement is to make alternative models easy to try and avoid becoming attached to the one around which the application was first built. Bishop does not claim new models already solve everything. The point is to keep the application able to benefit when capabilities improve, instead of making earlier investments a barrier to testing the next model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build for agents that use the platform
Building first-party agents raises a second product question: what should the platform provide to agents built by other teams? Bishop forecasts that agents could surpass humans as users of SaaS products in roughly five years, while acknowledging estimates ranging from one to ten years. This is a forecast, not an observed adoption result. Its design implication is to consider external agents alongside human users and the company’s own assistants.
His example is Claude using Datadog directly through MCP. Bishop reports setting this up relatively quickly, without presenting the implementation details. The platform-design work is to supply the operational context and API information an incoming agent needs to use the product effectively. An interface designed for a human browsing the application does not by itself answer those needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A team of DevSecOps agents
Bishop’s closing vision is a team of DevSecOps agents available for hire. Those agents would handle platform interaction, integration and on-call work that customers otherwise perform themselves. At the same time, external SRE and coding agents would become platform customers, using its tools much as a human engineer would. These are complementary directions: provide agents that do the work, and provide a platform other agents can operate.
For a small company, the imagined workflow combines automated development through products such as Cursor or Devin with agents handling operations and security. Bishop’s aspiration is that this could let an order of magnitude more ideas reach the real world; it is not a measured deployment outcome. He closes by inviting both builders of agents that could use Datadog and prospective users of Datadog’s own agents to participate in that work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original research on teaching language models when and how to call external tools.
Further reading
- The Bitter LessonArticle
Rich Sutton's essay on general methods that benefit from increasing computation.
Updates since the talk
Datadog's later account of its AI on-call teammate and incident collaboration workflow.
Explains how production telemetry feeds code changes, tests and pull requests for engineer review.
June 2025 announcement of interactive agent execution graphs and dataset-based LLM experiments.
Full transcript
Loading transcript…