AI Engineer Summit 2025
Building AI Agents with Real ROI in the Enterprise SDLC
Loading slides…
Read the talk
Building AI Agents with Real ROI in the Enterprise SDLC
Booking.com’s work with Sourcegraph connects AI adoption to delivery metrics, then tackles GraphQL generation, legacy migration, and code review through carefully selected context and explicit rules.
From a talk by Bruno Passos and Beyang Liu
Before you start: Familiarity with pull requests, CI pipelines, feature flags, and basic GraphQL will help with the examples.
What did adopting AI actually change?
The CEO says the company needs to adopt AI. Someone buys Copilot, perhaps more out of fear of falling behind than from a clear understanding of the problem. Six months later, the CFO asks what return the purchase produced. What changed for the engineering organization, and how can anyone measure it? Buying access is easy; demonstrating an improvement in software delivery is harder.
That question frames the collaboration between Sourcegraph and Booking.com. Beyang Liu presents Booking as a company actively following adoption through to measurable impact. Bruno Passos is more cautious: Booking still considers itself near the beginning. The useful material is the progression from that uncertain starting point—through rollout, measurement, and training—to agents aimed at specific maintenance tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Experimentation creates a cleanup problem
Booking’s developer-experience mission follows from its travel business: clear developers’ paths so they can build the features that help people experience the world. Progress varies across the company. Passos describes an operation with more than 3,000 developers and cites about 1.5 million room nights, without specifying a period for the latter figure.
Passos reports 2.5 million CI jobs annually. The deeper source of maintenance work, however, is Booking’s experimentation culture. A/B tests introduce experiment branches and feature flags. Teams move on to new features, but the flags and dead code often remain. Repeat that process for decades, and the accumulated experiments become a substantial part of the codebase developers must understand.
While Passos was preparing his slides, his children asked what feature flags were. His explanation—that they linger and pollute the codebase—prompted the response, “Like code farts?” He distinguishes that joke from code smells, then returns to the practical consequence: larger codebases mean longer cycle times and more debugging. Passos describes work in this bloated-codebase situation as over 90% toil, rather than presenting a company-wide time-allocation study.
Quarterly developer surveys reinforce the problem: developers report that working in the codebase keeps getting harder. Liu’s example is the talented engineer building an elaborate system to update legacy code at PwC. The engineering may be impressive, but it consumes expertise that could otherwise address new features and user problems. Dead-feature-flag migrations can become years of work in their own right.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search first, then an assistant people can use
Sourcegraph’s tools address different parts of that burden. Code Search helps developers find code and build a working understanding across a large codebase. Refactoring and migration tooling supports changes at scale. Cody, its coding assistant, generates code with codebase context. Liu separates the objectives into two loops: augment human creativity in the inner loop of development, and automate repetitive work in the outer loop of the software lifecycle.
Booking started with Search, which made scattered pieces of context easier to locate in its bloated codebase. It then began experimenting with Cody because the assistant could use that search context. The intended next step was to carry the same foundation into agents: an automation should be able to find relevant code before trying to act on it.
The rollout timeline described at the February 2025 AI Engineer Summit begins in January 2024. Booking made Cody available to its developer population, but availability did not ensure continued use. Some developers tried it, saw little value, and stopped. Initially, the company had one LLM choice and token limits that constrained experimentation.
Booking and Sourcegraph then relaxed those usage constraints and introduced model choice per developer. The reason was practical: the model that helped excavate a legacy codebase was not necessarily the best choice for creating a new service. Training followed in July. Developers who had previously abandoned the assistant began finding useful ways to apply it and became regular users. Model access, task fit, and education were separate parts of adoption.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure delivery, quality, and modernization
Booking’s initial metric was hours saved. Passos found its basis too weak: research involving only a few developers did not justify the sweeping totals circulating in the industry. Claims of tens of thousands of saved hours could create pressure to reproduce a headline without establishing what had actually improved. Booking began looking for measures more closely tied to its engineering workflow.
After defining new KPIs in October, Passos reports indications by November that daily Cody users were more than 30% faster. Booking defined daily use as activity on at least 12 days in a month, allowing for weekends and days without coding. This was an observed usage cohort, not a randomized treatment group; the talk does not establish that Cody alone caused the difference.
The partnership also produced an API layer in front of Cody, opening possibilities in Slack, Jira, and other tools outside the IDE. This was Booking’s historical integration work; the current Cody documentation does not establish the contract or availability of that particular API layer. Moving beyond the editor mattered because many useful software-development tasks begin in a ticket, a conversation, or an automated workflow.
Booking bounded its KPI program to a year. Short-, medium-, and long-term measures all had to fit that horizon, with long term explicitly meaning one year. Passos introduces four KPIs, discussing lead time for change, quality, and codebase insights before connecting them to modernization.
On delivery, the team examined merge-request review time and throughput. Passos reports that daily Cody users shipped 30% more merge requests than developers who did not use Cody. That throughput observation is distinct from the earlier speed claim. The users’ merge requests also contained less code; Passos had not yet determined what to make of that difference.
The remaining work links measurement to specific engineering interventions:
| Area | Proposed intervention | Purpose |
|---|---|---|
| Vulnerabilities | Combine past vulnerabilities with codebase context | Find lingering or likely weaknesses |
| Test coverage | Add tests around legacy behavior | Check replacement implementations |
| Codebase insights | Find unused code, stale flags, and slow code | Identify modernization targets |
| Replatforming | Use those findings to reduce migration work | Move from years toward months |
The test-coverage idea supplies a concrete bridge: capture legacy behavior in tests, then require the replatformed implementation to pass them. Reducing replatforming from years to months remained the objective, not a completed result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt customization becomes agent development
Engineers using the assistant also began experimenting with the underlying APIs. Customizing a prompt naturally led to another question: could several calls be composed into a longer automation? Those chains became the starting point for agents.
Early attempts ran into mismatched expectations about what LLMs could do reliably. The accompanying slide also identifies model choice, reproducibility, and reliability as pitfalls. Booking and Sourcegraph responded with a week-long joint hackathon in Amsterdam, bringing their teams together to work through concrete tasks and prompts. Three applications emerged: GraphQL generation, migration analysis, and organization-specific code review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve schema paths before generating GraphQL
The first agent addressed Booking’s large GraphQL API. Liu describes its schema as exceeding one million tokens. His explanation frames it as too large for the available context windows and difficult to use coherently even if it could be included. The capacity claim needs a historical qualification: Gemini 1.5 Pro already offered a two-million-token window before this talk. That does not establish whether Booking’s full schema fit its approved tooling, or whether a model could reliably generate queries from it. The engineering problem was selecting usable schema context, not merely increasing the token allowance.
The agent assembles that context in stages:
- Search the schema for candidate nodes related to the request.
- Select relevant nodes from the search results.
- Walk upward to retrieve the parent nodes needed to connect those selections.
- Generate the response after assembling the relevant schema structure.
Finding an isolated field is insufficient if the model does not know how to reach it from a query root. The upward traversal preserves that structural context. In the demonstration, the right-hand panel displays the agent’s node-selection process before the response appears.
For a small teaching example, suppose the requested value is a hotel’s name. A matching field such as Hotel.name needs the parent route through Query.hotel, including its required argument. The relevant schema slice could be:
graphql
type Query {
hotel(id: ID!): Hotel
}
type Hotel {
id: ID!
name: String!
}
With that route available, the generated query can include both the root field and the nested selection:
graphql
query HotelName($id: ID!) {
hotel(id: $id) {
name
}
}
This illustrates why schema retrieval must preserve relationships, rather than return only fragments containing a matching word. Liu reports that naive generation produced hallucinations despite a similar-looking interface; working through task-specific prompts and retrieval during the hackathon produced qualitatively better results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scope the migration before attempting the rewrite
The migration experiment began with legacy code containing functions longer than 10,000 lines. The approach combined code search, structured metaprompts, and divide-and-conquer decomposition. Before transforming an enormous function or service, the team needed to understand its dependencies and split the work into tractable pieces.
Expert pairing helped developers learn how to supply the right prompt and context. Passos connects this to the earlier adoption problem: some developers had dismissed the assistant because they did not yet know how to use it effectively on their own code. Bringing specialists into the office made that knowledge practical rather than abstract.
Passos reports that, after months of investigation, two hackathon days helped the team identify call sites and size the migration problem. That understanding gave the team a starting point and exposed easier pieces to tackle first. The months-to-days observation concerns understanding and scoping the codebase; the migration effort itself remained experimental. Finding the boundaries of a change can be valuable before an agent performs the change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make code review specific and selective
Code review initially looked like an unpromising area for another agent because so many AI-review products already existed. Conversations with Booking and other enterprises revealed an unmet requirement: review depends on a long tail of team-specific rules, guidelines, and conventions. A general reviewer could miss precisely the constraints an organization cared about.
Sourcegraph was productizing a process for building a reviewer tailored to those requirements:
- Define the team’s rules in a simple flat file.
- Identify the files modified by a pull request.
- Apply the rules relevant to those files.
- Post comments selectively, tied to the applicable rules.
The flat file makes the review expectations explicit, while relevance selection keeps unrelated rules out of a particular review.
The stated objective is precision over recall. The reviewer should prioritize comments that developers can act on, even if that means missing some potential issues. In this setting, producing more comments is not inherently useful: noise makes the agent another source of work in an already crowded review process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Declare service rules and enforce them earlier
After a year of collaboration, Booking began extending the review idea toward the service itself. If CI can report that a change violates a rule, could the developer receive that information earlier, inside the IDE, together with a proposed fix? The longer-term aspiration was a service capable of detecting and repairing some of its own problems.
The ingredients included codebase context and the prompt library developers had already built. Automating those questions against a server could expose knowledge about the codebase and identify work to perform. Passos described an ambition to reach parts of this vision by the end of 2025, with substantial work still ahead; self-healing services were a direction, not a delivered outcome.
Liu connects declarative rules to the loss of architectural cohesion familiar from The Mythical Man-Month. Successful software attracts users, revenue, feature requests, and bug reports. Competitive pressure encourages teams to take on technical debt. As more contributors join, a shared vision and consistent standards become harder to preserve.
Under the proposed approach, senior engineers and architects define constraints that should hold across the codebase. Those constraints can be applied during review and inside the editor, whether a human or an AI wrote the code. Enterprise compliance requirements are a concrete candidate: recurring obligations that developers must satisfy even though they do not directly create new user features. The accompanying slide extends the vision to background agents submitting pull requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Education turns access into regular practice
Declaring compliance requirements in service rules is one way to reduce recurring developer work. But Passos ends with the organizational practice he considers most important: education. Booking worked with entire business units, giving developers hands-on support and time to experiment in two-day workshops and hackathons. People who could see how the tools applied to their own work were more likely to become regular users.
That education was intended to sustain the delivery improvements Booking associated with daily use, not a separately measured training effect. The operational lesson is concrete: an assistant made available to everyone can still be abandoned; developers need to learn how to give it useful context, choose appropriate tasks, and recognize where it helps. The agents grew out of that same practice—working through real problems with the people who understood the code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current documentation for searching across repositories, filtering results, and exploring code changes.
Current guide to Sourcegraph's coding assistant, including codebase context, editor integrations, and reusable prompts.
Fred Brooks's essays on software project management and conceptual integrity, in the anniversary edition.
Further reading
Google's June 2024 announcement of broader access to a two-million-token context window and context caching.
Full transcript
Loading transcript…