# How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver

Noyan Tokgozoglu, Optiver

[Talk page](https://ai.engineer/talks/aGIUg8zKI8A-trading-firm-ships-new-ai-models-same) · [Video](https://www.youtube.com/watch?v=aGIUg8zKI8A)

## How Optiver Prepares to Ship New AI Models on Release Day

Noyan Tokgozoglu explains how Optiver combines internal benchmarks, model and harness portability, on-prem infrastructure, and token-cost controls to pursue same-day model adoption—and why replaying tool calls and scaling successful agents remain harder problems.

![Optiver title slide for “Scaling AI With P&L On The Line,” presented by Noyan Tokgozoglu.](https://ai.engineer/api/talk-essay-media/5a9a96ff730a501376acdeaad9f83a56e13711b714b224db18175e7046c941b3)

Noyan Tokgozoglu’s title slide frames the talk around scaling AI with profit and loss at stake.

### When a change can affect P&L within hours

[0:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=12s)

A change rolled out to Optiver’s traders and quantitative researchers can show up in profit and loss within hours. That short interval gives an AI platform an unusually direct incentive to make useful capabilities available quickly. Noyan Tokgozoglu, Optiver’s head of AI engineering at the time of the recording, starts with the scale behind that pressure: more than ten million trades a day, more than a million instruments priced, petabytes of data processed daily, and more than ten locations worldwide. The firm trades with its own capital, so the consequences of its decisions land on its own books.

![Two operating-model timelines compare traditional software rollout and revenue measurement with Optiver’s rollout to traders and quants followed by P&L impact within hours.](https://ai.engineer/api/talk-essay-media/20fb2beae51679e1fe4b0877175683058eb18c76aa59508c848bc0bda11f4192)

The operating-model slide contrasts longer engagement and revenue feedback loops with P&L impact within hours at Optiver.

The contrast is with a product business that first watches engagement over days or weeks, then revenue over months or quarters. At Optiver, the daily P&L provides a much earlier signal. Some instruments still have longer horizons, so the first day does not reveal all the value of a decision. But it can reveal enough to make waiting expensive: an opportunity tied to a particular market event may be gone by the time a slow deployment process finishes.

That creates two requirements at once. The platform must move quickly enough to capture an opportunity, and it must measure changes well enough to decide whether they deserve production use. Speed without measurement risks moving the wrong thing into a consequential workflow. Measurement without a fast delivery path leaves useful capabilities waiting.

### Move fast, but give self-service users sane defaults

[2:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=161s)

Modularity is Optiver’s way of keeping that delivery path open. Models, products, and agent architectures keep changing—“loops,” “swarms,” and “graphs” in Tokgozoglu’s quick succession. A useful new approach should be measurable and replaceable without requiring the entire platform to be rebuilt around it. External benchmark scores help identify candidates, but Optiver’s internal tasks have a different data distribution. The decisive question is how a candidate performs on the firm’s own work.

![AI platform principles include speed, modularity, measurement, self-service with good defaults, safety, and on-premises operation except for third-party model inference.](https://ai.engineer/api/talk-essay-media/f0c1edd1280f906327431fc09ccd83cfdaabcc108ea043ced943a7bf4bb5ebe7)

The platform principles emphasize speed, modularity, measurement, self-service with good defaults, and safety; third-party model inference is an explicit exception to on-premises operation.

Self-service introduces a separate tension. Traders are capable, motivated users who will build their own workflows, but unrestricted configuration can produce chaos. The platform therefore combines user agency with good defaults. AI engineers also work directly alongside trading teams: they learn how the teams trade, teach where AI can help, and bring those observations back into the shared platform. This gives platform decisions a concrete source of requirements rather than relying only on what engineers imagine users need.

The infrastructure reflects the firm’s regulatory and intellectual-property constraints. Optiver runs its own on-prem GPUs and infrastructure; third-party model inference is the stated exception to keeping work inside that environment, covered by agreements such as zero data retention. Agent safety matters because an agent acting unpredictably against the internal stack could have serious consequences. These constraints shape the platform even as AI use spreads across the company.

### Two gateways solve different problems

[5:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=311s)

The platform overview answers a useful architectural question: where do model access and agent coordination belong? At the bottom, Optiver uses third-party models, self-hosted models, and open-source models. An **inference gateway** routes requests among them, measures spending, and assigns budgets. That common access layer also provides an alternative when a provider becomes unavailable: the platform can switch providers instead of making every consuming workflow handle the outage itself.

An **agent gateway** serves a different purpose. It maintains a shared ledger of agents, supports communication and orchestration, and brings humans and agents into similar identity-management arrangements. The agent platform registers capabilities such as skills, plugins, and tools, while individual teams can bring their own knowledge and memory systems. Keeping the gateways distinct separates the choice of a model backend from the discovery and coordination of the agents that use models.

The stack ends with interfaces people can actually use. A web interface exposes sessions; **Agent Workbench** lets teams build and share profiles for how they use agents; and a personal assistant is available throughout the company. These interfaces are part of the platform’s effectiveness, not an optional finishing layer. A capable agent that people do not know how to reach contributes little to their work.

### Same-day adoption needs more than a new model endpoint

[7:10](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=430s)

Optiver’s goal is to make a newly public frontier model usable in production flows on the same day—if it adds value. The qualification matters. Trading teams contribute benchmarks to an internal evaluation framework, creating golden sets against which a new model can be run. The resulting metrics help teams answer whether a public release is useful for their particular cases, rather than treating a higher external benchmark score as sufficient reason to switch.

![Same-day flow from a frontier model’s public release through evaluation, benchmarks, gateway and harness updates, and support to production availability.](https://ai.engineer/api/talk-essay-media/a3b4f6e347f133dfd4cc94a66539e604d06637d7b0a44858fb499b6659dd1f11)

The same-day rollout diagram includes evaluation, customer benchmarks, gateway and harness updates, support, and documentation.

Several kinds of work support that goal:

- **Model integration:** Make the model available through the inference gateway, with the appropriate APIs and interfaces.
- **Cost accounting:** Set up cost management so the new capability enters the same spending controls as existing models.
- **Harness delivery:** Update and distribute the software around the model when its use requires harness changes.
- **Operational support:** Maintain regional support rotations for round-the-clock coverage across a global firm.
- **User guidance:** Share benchmark results, usage instructions, and a recommendation so users can make an informed choice.

This makes same-day adoption an operational capability rather than merely a fast API integration. A model has to be evaluated, reachable, accounted for, supported, and explained. The recording presents this as the platform’s operating goal; it does not provide a release-by-release record showing that every new model met it.

### Keep model choice separate from the agent experience

[8:30](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=510s)

Portability has practical reasons beyond avoiding a favorite vendor. Model availability varies by region and cloud provider. Some workloads must run in an isolated on-prem zone, for both isolation and cost reasons. Optiver therefore wants to remain agnostic to both the model and the harness—the software environment around it that shapes how an agent works.

![Model-and-harness-agnostic architecture showing team harness configuration, an Optiver meta-harness, knowledge bases, an inference gateway, third-party and on-premises providers, logging, and evaluations.](https://ai.engineer/api/talk-essay-media/2a6ce11afcb4dfa367544fda8771d50f3168ac9e67bcce407966173d45930e5d)

The architecture separates harness configuration from inference providers while sharing knowledge, logging, and evaluation services.

Third-party and on-prem model interactions pass through the inference gateway. Those interactions are logged and automatically sent to the evaluation system. Above the supported harnesses sits a **meta harness**: a configuration layer through which teams define their experience and retain some portability between harnesses. Team configurations connect to their knowledge bases, and traces provide material from which additional knowledge can be derived over time.

Teams can deploy managed agents on Optiver’s platform or access them through the portal/IDE interface, Tokgozoglu explains. In either case, model requests can reach third-party or on-prem backends through the same inference gateway. The portability he describes applies to team configurations across supported harnesses.

Separate agent access, harness configuration, and model routing
Agent delivery branches above the harnesses; model routing branches below them. Model-interaction logs separately feed evaluation.
- Meta harness — Team configurations provide some portability across supported harnesses.
- Team knowledge base — Connects team knowledge to the configured agent experience.
- Managed agents — Deploy agents on Optiver’s platform through the meta harness.
- Portal / IDE interface — Access agents interactively through the shared interface.
- Supported harnesses — Different agent harnesses sit beneath the configuration layer.
- Inference gateway — Provides the common route to model backends.
- Third-party models — External model inference is one backend option.
- On-prem models — Local model inference supports on-prem workload requirements.
- Evaluation system — Receives logged model interactions automatically.
- Team knowledge base → Meta harness (Supplies team knowledge)
- Meta harness → Managed agents (Supports managed deployment)
- Meta harness → Portal / IDE interface (Supports interactive access)
- Meta harness → Supported harnesses (Configures the experience)
- Supported harnesses → Inference gateway (Model interactions use the gateway)
- Inference gateway → Third-party models (Routes model requests)
- Inference gateway → On-prem models (Routes model requests)
- Inference gateway → Evaluation system (Sends interaction logs)

### A fresh session should not start with 200K tokens

[10:01](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=601s)

The spending chart asks a different question: what happened when powerful new models met growing internal use? Optiver’s illustrated spending curve peaks at the July frontier-model release marker, falls to the August spend-optimizations marker, then rises in September while staying below the July peak. Tokgozoglu credits the optimizations with helping control spending. The chart has no numerical spending scale from which to calculate savings or return on investment.

![Illustrative AI spending line chart from November 2025 to September 2026, marking a July frontier-model release and August spend optimizations; the vertical axis has no numeric scale.](https://ai.engineer/api/talk-essay-media/055d953785275bb2925514792f0461b7f975c6879a7bad760ca19f5691d3f5de)

The slide labels this spending curve illustrative. It marks new frontier-model releases and later spend optimizations without reporting numeric production costs.

One concrete failure was visible before an agent had done much work. Some fresh sessions began with 200K tokens of context by default, sometimes more. Users had accumulated MCP-connected tools, knowledge sources, and other material in their setups. Self-service made it easy to add useful capabilities individually; their combined cost became an unreasonable starting state. This was the earlier defaults problem arriving as a token bill.

The first corrections targeted individual sessions rather than requiring a sophisticated routing system:

- **Tool access:** Add MCP tool search and give agents the option of using a CLI that wraps several MCPs. Tokgozoglu reports the CLI approach as more efficient, without specifying its token accounting.
- **Cache behavior:** Investigate when caches become invalid, how long cache-related parameters should be set, and whether those settings should vary dynamically. Repeatedly losing reusable context can undermine the benefit of caching.
- **Prompt placement:** Work with teams to move suitable material out of massive system prompts and into knowledge bases, where it can be retrieved when relevant instead of accompanying every interaction.

These changes address how much context a session carries and how often that context must be paid for. Avoiding unnecessary tokens becomes more valuable as token prices rise. The useful surprise is how much room remained in ordinary session configuration, even before model routing or sub-agent optimization entered the discussion. Shipping the changes also took time across Optiver’s global regions; the point of deployment and the point of full effect were not necessarily the same.

A second set of controls changed what teams could see and choose:

- **Spending visibility:** Show teams their usage and let them set alerts and limits. Many teams set reasonable limits themselves once they had that information and control.
- **Open-source alternatives:** Offer models that are competitive on token cost for appropriate tasks. Optiver found useful replacements in some cases, not a universal substitute for frontier models.
- **Routing experiments:** Explore sending prompts to cheaper models, or using a frontier model to generate a plan whose implementation is divided among smaller models.
- **Selective retrieval:** Improve knowledge bases so agents retrieve the context they need rather than simply retrieving more context.

The aim is still substantial AI use. Tokgozoglu’s formulation is “more ROI out of the same tokens”: keep investing where the work is valuable, while removing context waste, exposing costs, and matching model expense to the task. Some interventions were already in use; dynamic routing and frontier-planner/smaller-implementer arrangements were described as approaches being explored.

### Replace bespoke access with something users want to use

[13:54](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=834s)

Before the shared portal, users had already solved access for themselves. Different harnesses came with bespoke arrangements for using `tmux` on development machines, and some people built improvised web interfaces. The platform team treated those workarounds as evidence of demand: build a better version of what people were already assembling, then make it available across the company.

![Slide titled Give Users Tools They Want with a conversation interface screenshot labeled Mock.](https://ai.engineer/api/talk-essay-media/025be1e0c5e274cc513515bbc96a7597ff28da6e3531f35e89e98ad9259e4a01)

The slide uses a conversation-interface mockup to illustrate tools designed around users’ needs.

The portal brought session management, chat windows, and artifacts into one place. An internal implementation fit Optiver’s infrastructure and let the team iterate at its own pace. Traders also began contributing code: a useful workflow improvement could arrive as a pull request and become a shared feature. Collaborative document editing is one example of what this contribution path produced.

The next direction was “agent multiplayer”: rooms bringing together one person’s agents, another person’s agents, and other deployed agents. A Slack client also provided another way to access the system. The product principle is deliberately practical. Users with enough agency will build around a missing interface anyway; a shared platform can turn that scattered effort into a better-supported experience.

### The knowledge base is useful—and still unsettled

[16:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=973s)

The questions begin by probing the knowledge layer that supports both agent quality and lower context costs. Optiver uses a hybrid approach that leaves room to explore different methods. Hierarchical Markdown and vector search are both discussed as options with places where they work, rather than as a settled winner. Tokgozoglu is blunt about the state of the work: he is not happy with the solution.

Graph databases are a direction he wants to investigate further, not a demonstrated improvement in this platform. That leaves an important distinction intact: better retrieval can reduce unnecessary context, but choosing and maintaining the knowledge system that delivers it remains an open engineering problem.

### Why build when commercial agent tools exist?

[17:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1031s)

A question about hand-built systems versus **Codex** brings the infrastructure decision into focus. Tokgozoglu describes a historical mismatch: tools did not necessarily support managed agents on Optiver’s own infrastructure in the way the firm needed when it began building. At the time of the recording, he says those options were improving and Optiver was working with the provider. The internal platform’s origin therefore reflects the capabilities and constraints encountered during its development, not a claim that commercial tools can never fit.

The more persistent concern is dependence on someone else’s release schedule. With a previous inference-gateway provider, a newly needed capability could turn into a conversation with a go-to-market team and a request to join the roadmap. That delay conflicts directly with same-day adoption. Deep investment in a provider’s model or surrounding system can make the platform move only as fast as that provider allows.

Building internally also rests on confidence in Optiver’s engineering team and its own managed-agent roadmap. That is a condition of the decision, not a free benefit of owning code. Tokgozoglu explicitly limits the recommendation: this approach will not work well for everyone. The firm’s combination of infrastructure requirements, urgency, and engineering capacity makes the tradeoff attractive in its circumstances.

### A logged tool call does not preserve the world it observed

[19:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1153s)

The next question exposes a limitation beneath the evaluation pipeline. Optiver logs tool-call inputs and outputs and sends them into its evaluation system. But consider the support bot it runs in Slack. To ask whether a different model would make that bot better, the team needs to rerun work involving tools whose answers depend on external state. By the time the evaluation runs again, that state may have changed.

The problem is causal, not merely archival. If a replacement model receives different tool results because the surrounding system changed, an outcome difference cannot be cleanly attributed to the model. Logging the original input and output preserves what happened in that invocation, but rerunning an agent against its tools also requires a way to recover the relevant historical environment.

The feasibility depends on the tool:

- **Trading backtest database:** Point-in-time access is already part of how the system works, so an evaluation can target the historical state.
- **Slack and other changing systems:** Reconstructing what was available at the original moment is harder. Tokgozoglu raises the possibility of recreating a point-in-time Slack environment, while noting that some systems offer no comparable answer.

Optiver’s partial solution is to record a confidence value for how well a particular tool invocation can be reconstructed at a point in time. During an evaluation rerun, a proxy around the MCPs lets the team request a particular snapshot. This makes historical-state fidelity an explicit part of the evaluation rather than silently assuming it. The mechanism is imperfect: where the original environment cannot be recovered faithfully, a model comparison remains confounded by changes in tool state.

### Scale a successful agent without giving it every job

[21:18](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1278s)

The final question follows a successful agent into a larger use case. One team had an agent for coding and analysis scoped to its repository. Extending it to work with another team sounded incremental, but it immediately introduced multiple repositories and different authentication requirements. The initial experiment was the obvious one: give a new agent the union of both sets of functionality. In practice, that approach often produced technical debt and poor behavior.

Optiver distinguishes small capability increments from a large expansion of responsibility. Agents are versioned, so a modest addition can ship as a new version. For broader growth, the preferred direction is to keep agents scoped to tasks they do well, can be measured on, and can have their lifecycle managed around. A second task can belong to a second agent rather than becoming another responsibility of the first.

The agent gateway makes that separation usable. Agents share a ledger through which they can find, communicate with, and spin up one another. A narrowly scoped agent can therefore call on another agent’s capability instead of absorbing its tools, repositories, and responsibilities into one growing implementation. This turns expansion into a multi-agent orchestration problem, which still needs engineering, but preserves more manageable units of behavior.

Measurability is the reason to accept that orchestration work. When one large agent does many things, a change can improve some functions while worsening others, making diagnosis and tuning difficult. Smaller task-focused agents keep the affected behavior easier to isolate. The ending returns to the condition behind same-day shipping: changing the system quickly is useful only while the team can still tell what the change did.

## Key takeaways

- Same-day model adoption depends on internal golden-set evaluations, gateway integration, cost management, harness updates, support, and clear user recommendations—not just access to a new endpoint.
- The inference gateway handles model routing and spending controls; the agent gateway handles agent registration and coordination. The meta harness gives teams configuration portability across supported harnesses.
- Some fresh sessions started with 200K tokens of context or more. Tool search, CLI wrappers, cache tuning, and moving material out of oversized system prompts addressed waste before more complex routing optimizations.
- Tool-call logs alone cannot make an agent evaluation reproducible. Historical tool state matters; Optiver requests snapshots through an MCP proxy and records confidence in how faithfully each invocation can be reconstructed.
- Expanding one agent across teams and repositories can make behavior harder to manage and measure. Optiver’s preferred direction is versioned, task-scoped agents that coordinate through the agent gateway.

## Related talks

- [Building agents is trivial now, context is the next frontier](https://ai.engineer/talks/HvMyYLTfvhg-building-agents-is-trivial-now-context-is) — Optiver’s oversized sessions illustrate why context matters when building agents.
- [Building the platform for agent coordination](https://ai.engineer/talks/UG9IAdmi2Dg-building-platform-agent-coordination) — Optiver’s shared agent ledger is one example of a platform for agent coordination.
- [How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase](https://ai.engineer/talks/_4Ph2dhmUFI-know-when-ai-agent-work-unsupervised-evan) — Optiver’s Slack bot example raises an evaluation question relevant to deciding when an agent can work unsupervised.

## Slides

### Scaling AI With P&L On The Line

Noyan Tokgozoglu, Optiver’s Global Head of AI Engineering, introduces scaling AI with the firm’s own money at stake in its trading decisions.

[0:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=12s) [Optiver title slide reading Scaling AI With P&L On The Line, with the presenter name and Global Head of AI Engineering role on a blue background.](https://assets.ai.engineer/sites/aiecode2025/sha256/012ba01289b213e41a6f1623f604bb4c8aa75ca3bfd7f0bf5e07cb9fa4276e61.webp)

Scaling AI With P&L On The Line

Optiver

Noyan Tokgozoglu
Global Head of AI Engineering

Document title | June-23

AI Engineer

### What Do We Do?

Optiver turns data into models and trading decisions using its own money, with no clients. Its systems handle 10M+ trades daily, price 1M+ instruments, and process petabytes of data daily.

[0:20](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=20s) [Optiver overview describing a technology-powered global market maker that makes trading easier, turns data into decisions, and builds high-performance systems; figures include over ten million daily trades and over one million instruments priced.](https://assets.ai.engineer/sites/aiecode2025/sha256/34925f09f9e7146bb7fc892d4081d8e901ca9e40b3a7fd40b1827199ae1372d8.webp)

What do we do?

Optiver

Global market maker, powered by technology

Make trading easier
Turn data into decisions
Build high performance systems

10M+ trades daily | 1M+ instruments priced | Petabytes of data processed daily

Engineering the future of AI

### Global Footprint

Optiver marks 12 locations across North America, Europe, Asia and Australia. Six are in Asia, the largest regional cluster on the slide.

[0:50](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=50s) [Global Footprint slide with geographically arranged markers for Chicago, New York, Austin, London, Amsterdam, Mumbai, Hong Kong, Shanghai, Taipei, Tokyo, Singapore, and Sydney.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a7bd13ec36472f4a38b972b6a7c48e2818802cf6d09f22e416b366676d3aed8.webp)

Global Footprint

Chicago
New York
Austin

London
Amsterdam

Shanghai
Hong Kong
Taipei
Tokyo
Mumbai
Singapore

Sydney

3

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Operating model

Optiver rolls changes out to traders and quants and sees P&L impact within hours. The traditional tech-company path takes days or weeks to collect engagement metrics, then months or quarters to measure revenue and LTV.

[0:59](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=59s) [Two process diagrams compare a traditional technology company's rollout, engagement measurement, and revenue assessment over days to quarters with Optiver's rollout to traders and quants and P&L impact within hours.](https://assets.ai.engineer/sites/aiecode2025/sha256/56c64ebe4d5129f4e068d64ee5b772f1a91e76cbc73624eb36f78d37a406ec7f.webp)

Operating model

Traditional tech company
Implement
→
Roll out to users
→ Days/weeks
Collect engagement metrics
→ Months/quarters
Measure revenue & LTV

Optiver
Implement
→
Roll out to traders/quants
→ Hours
P&L impact

4

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### What does this mean?

At Optiver, same-day feedback and costly missed opportunities favor quick, measured decisions. The speaker contrasts finite user attention with trading’s higher ceiling: more trading, he argues, means more earnings.

[1:47](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=107s) [Bullets describe an immediate feedback loop, costly missed opportunities, quick decision-making and measurement, a bias for action, and a higher ceiling, contrasting finite user attention with the stated benefits of increased trading.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b57048f315167f2cd9525e972682322d9d1bab6792ab5f59df9b8dfca97bca6.webp)

What does this mean?

• Immediate feedback loop
• Missed opportunities costly
• Quick decision making, measurement & bias for action
• Much higher ceiling
  - Attention of end users of a service is finite
  - The more you trade, the more you make

5

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Need an AI Platform — Core principles

Optiver ties speed to modularity and measurement, pairing self-service defaults with engineers embedded in trading teams. Everyone uses AI; infrastructure stays on-premises except third-party model inference.

[2:51](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=171s) [Eight principles call for speed, modularity, measurement at every layer, self-service with good defaults, forward deployment into trading teams, safety, company-wide AI use, and on-premises operation except third-party model inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac18655cd1a3bcef44ef290fd74e5bdf4f4f4e667a2f2d5115ccdbeaed6b9256.webp)

Need an AI Platform

Core principles

• Move fast
• Be modular (upstream can change)
• Measure at every layer
• Self-service, but provide good defaults
• Forward-deploy into trading teams
• Be safe
• Everyone in the company uses AI
• Everything is on-prem (except 3P model inference)

6

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

[4:04](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=244s) [Eight principles covering speed, modularity, measurement, self-service defaults, embedding engineers with trading teams, safety, company-wide AI use, and on-premises infrastructure except third-party model inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/0121dfc39259654ca19b249c67bb7b75b2733bbcf144d0b6e115601345c43622.webp)

Need an AI Platform

Core principles

• Move fast
• Be modular (upstream can change)
• Measure at every layer
• Self-service, but provide good defaults
• Forward-deploy into trading teams
• Be safe
• Everyone in the company uses AI
• Everything is on-prem (except 3P model inference)

6

Interface text:
Engineering the future of AI

### The Optiver AI Platform — Architecture

Optiver connects chat, coding tools and assistants to replaceable models through shared agent and inference layers. Trust controls span the stack; agent traces feed evaluations and human-approved improvements.

[5:09](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=309s) [Platform architecture showing user-facing surfaces, an agent gateway, agents and context, an inference gateway, and models, with trust controls on the left and a learning loop on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a119b62db8071f830abf2188ad3e2a2598369ec91b386276279958a50af521d.webp)

The Optiver AI Platform

Trust

Single sign-on
People and agents sign in once. Agents act as the user or as themselves.

Permissions
Team-scoped access to agents, tools and data, enforced at the gateway.

Isolation
Each conversation runs in its own sandbox.

Data handling
Tiered access to prompts and traces. Aggregates by default.

Regions
Hosted where the data and the users are.

Surfaces
people meet the platform where they already work

Chat & web console
Our clients inside team messaging, the web, and the developer session hub.

Coding harnesses
Vendor CLIs and IDEs, run as managed sessions.

Agent Workbench
Define an agent in a text file. Run it locally or deployed.

Personal assistant
One assistant per person, on every surface, with one memory.

Your own client
Teams build their own interfaces on the same libraries.

Agent gateway
one conversation layer for everything

Durable conversations
A room anyone or any agent can join, from any surface, at any time.

Identity & policy
Every participant has an identity. Access and budgets are enforced here.

Schedules & triggers
Agents run on timers and events, not only on request.

Session broker
One isolated, budgeted session per conversation, started on demand.

Agents, capabilities and context
one runtime, any SDK or harness

Agents
Personas and custom agents on one runtime, local or deployed. Tracing, workspaces and agent teams built once.

Skills, plugins & tools
Written once, translated to every harness. Team-owned tools, one catalogue.

Knowledge & memory
Team knowledge bases, personal and team memory, enterprise sources reached as the user.

Inference gateway
every model call, one door

Routing
Workloads ask for a capability and a budget, not a model. The router chooses from evidence.

Resilience
Several channels per vendor, automatic failover, regional placement.

Economics
Caching policy and per-request attribution to person, agent and task.

Keys & budgets
Short-lived credentials. Budgets checked before the call.

Models
replaceable by design

Third-party models
Frontier and open-weight models from vendors and clouds, each reachable through more than one channel.

Self-hosted models
Open weights on our own GPUs, for independence and sensitive work.

Learning loop
evidence → improvement

Dreaming
Mines real agent traces and proposes fixes to knowledge, skills and workflows. Humans approve.

Evaluation & certification
Every agent has evals built from real traces. Evals gate releases, model upgrades and routing.

Spend controls
Budgets follow the org chart. Cost per resolved task is the metric.

Logging & observability
Every conversation and model call recorded: steps, cost, outcome.

Legend:
What we build and own
Shared platform layer
Engines we expect to swap as the market moves

7

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### The Optiver AI Platform — How we move on the principles

Optiver targets same-day production access for new models that prove useful. Customer golden-set benchmarks test value; gateway and harness updates, round-the-clock support, and documentation prepare the rollout.

[7:22](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=442s) [Same-day model-availability diagram showing public release, a mature evaluation framework, partnering with key customers to onboard, benchmarking on golden sets, and production availability.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1110512243da7150130ef7c494f4a4ad341a77e269f3b2e0d743da092fda71a.webp)

The Optiver AI Platform
How we move on the principles

New frontier model available publicly
Mature eval framework
Partner with key customers to onboard
Benchmark on golden sets
Model available to production flows

Same day

8

Engineering the future of AI

[7:57](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=477s) [Same-day model rollout diagram covering evaluation, customer onboarding, gateway updates, harness validation, golden-set benchmarks, support and documentation before production availability.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac4cffe352d492f150e04490b742386bd0fdbf43b8ea9b746be1896bdc20b310.webp)

The Optiver AI Platform
How we move on the principles

New frontier model available publicly
Mature eval framework
Update model gateway
Partner with key customers to onboard
Swift update & validation of harnesses
Benchmark on golden sets
Round-the-clock support
Comms & documentation to customers
Model available to production flows

Same day

8

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Model & Harness Agnostic

Optiver routes third-party and on-prem models through one inference gateway with logging and evals. A meta-harness lets teams configure portable harness experiences, connect knowledge bases, and use agents or the AI portal.

[9:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=552s) [Architecture with provider and on-premises inputs feeding an inference gateway beneath stacked harnesses and an Optiver meta-harness; logging and evaluation boxes sit to the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/11b1460606871a9a5a1302dc3dd813c76d8b9e4387e2292c1d1350c813f06373.webp)

Model & Harness Agnostic

• We integrate any model if it’s adding value
• Insulate against model availability
• On-prem serving for DMZ & cost efficiency

Optiver meta-harness
Harnesses
Inference gateway
3PC Provider
On-prem serving
Logging
Evals

9

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

[9:38](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=578s) [Model-agnostic architecture with third-party and on-prem serving, an inference gateway, harnesses, team configuration, knowledge bases, agents, an AI portal, logging and evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/a4130a49f4d1173837bb085c59a8168595b210de26476aaad951d1f1fccc6f52.webp)

Model & Harness Agnostic

• We integrate any model if it’s adding value
• Insulate against model availability
• On-prem serving for DMZ & cost efficiency

Agents
Optiver AI Portal
Optiver meta-harness
Knowledge Bases
Team-specific harness config
Harnesses
Inference gateway
3PC Provider
On-prem serving
Logging
Evals

9

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### We Are Doing More With AI

Optiver’s illustrative AI-spending curve spikes with July’s frontier-model releases, then falls as optimizations bring spending under control. Spending edges up again in September; no spending amounts are shown.

[9:53](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=593s) [Illustrative AI spending chart from November 2025 to September 2026, rising sharply around July, falling after spend optimizations in August, then rising modestly.](https://assets.ai.engineer/sites/aiecode2025/sha256/d39fc06ba95582e596512fc675577e11b8ba92c79fb4a279ea95b4b4dc59dbc6.webp)

We Are Doing More With AI
Also spending more

New frontier models release
Spend optimizations

Timeline: Nov-25, Dec-25, Jan-26, Feb-26, Mar-26, Apr-26, May-26, Jun-26, Jul-26, Aug-26, Sep-26

(illustrative graph)

10

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Spend Wisely, Not Less

Optiver cut average fresh-session context from ~200k to 15k tokens using MCP tool search, agent CLIs, caching, and team-specific harness tuning. The chart drops sharply through August, then stays low.

[10:30](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=630s) [Slide stating fresh-session context usage decreased from approximately 200k to 15k, with a declining chart and tool-search, CLI, caching and team-harness optimizations.](https://assets.ai.engineer/sites/aiecode2025/sha256/22a9f9abd89ab110c71bb2177743c0a38d6f94c0cbaf0797734256541db82b2b.webp)

Spend Wisely, Not Less

Average context usage on fresh session ~200k -> 15k

MCP tool search
Giving agents CLI
Caching optimizations
Per-team-harness optimizations
These scale with token cost

Timeline: 7/6, 8/3, 8/31, 9/28

11

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Spend Wisely, Not Less

Optiver cut average fresh-session context from ~200k to 15k tokens. The chart drops sharply through August, then levels off, alongside MCP tool search, agent CLIs, caching and team-specific harness optimizations.

[10:35](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=635s) [Slide stating fresh-session context usage decreased from approximately 200k to 15k, alongside a declining chart and a list of tool-search, CLI, caching and team-harness optimizations.](https://assets.ai.engineer/sites/aiecode2025/sha256/f500d1cc29a524c81e06578595e81a8c3143efc8a194030390d23a46133c6493.webp)

Spend Wisely, Not Less

Average context usage on fresh session ~200k -> 15k

MCP tool search
Giving agents CLI
Caching optimizations
Per-team-harness optimizations
These scale with token cost

Timeline: 7/6, 8/3, 8/31, 9/28

11

Interface text:
AI Engineer

[10:50](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=650s) [Line chart showing declining context usage over dates from July through September. The subtitle states approximately 200k to 15k; annotations list tool search, agent CLI access, caching and team-specific harness optimizations.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1df3e3dd913fddd0ef9cfa24539b293b45c0a7eba8015ee404e32262557c6e8.webp)

Spend Wisely, Not Less

Average context usage on fresh session ~200k -> 15k

MCP tool search
Giving agents CLI
Caching optimizations
Per-team-harness optimizations
These scale with token cost

Timeline: 7/6, 8/3, 8/31, 9/28

11

Interface text:
Engineering the future of AI

### Spend Wisely, Not Less — Improving Token ROI

Teams set their own spending limits; Optiver also serves open models on-prem and explores cheaper-model routing. Better knowledge bases reduce excess context, aiming for more ROI from continued token investment.

[12:23](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=743s) [Bullets describe team visibility and alerts, on-premises open-model serving, model and subagent routing, better knowledge bases, and continued token investment for higher ROI.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ee41f96e05575411507c98c9b02a11e05ee9ac4a7bfadeb573d53ee170751f5.webp)

Spend Wisely, Not Less

• Giving teams more visibility, alerting, agency
• On-prem serving of open models
• Model & subagent routing
• Better knowledge bases -> model flounders less

• Still investing in tokens, but getting more ROI

12

Interface text:
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### Give Users Tools They Want

Optiver built a web portal for sessions, chats and artifacts. The mockup pairs an onboarding draft with a topic chart: selecting Onboarding sends that choice back to the agent to expand the guide.

[13:52](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=832s) [Slide labeled Mock shows an agent workspace with session navigation, an onboarding-draft conversation, and a side-by-side topic bar chart with a control for sending a selection to an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/66ae2923d3fec15bc36b90ffa8ac63989bbeb02ce7e5c39b8a517dbe1f889258.webp)

Give Users Tools They Want

Mock

Draft guide
CLAUDE CODE

Turn the research notes in this folder into a short onboarding guide.
Keep the open questions separate.

Read notes/*.md (14 files)

A first draft

I read the 14 notes and grouped 43 questions into four topics. The guide below covers the two that come up most often; the rest go in an appendix.

• Opening a session and giving it a task with a clear result.
• Keeping conversations, files and an interactive chart side by side.
• Starring a recurring assistant so it stays in every View.

I opened the topic breakdown as an AUI page beside this conversation. Select a topic there to send it back to me and I'll expand that section.

aui open questions-by-topic → pane 2

Onboarding first. Also check what the payments team wrote about access setup last month.

aui event select=Onboarding [illegible] Questions by topic
Grep "access" ~/work/payments/docs (3 matches)

Expanding the onboarding section now. The payments notes describe the same access flow with one extra step, so I'll fold that in rather than repeat it.

Illustrative conversation and data.

Questions by topic
AUI
AUI · AGENT-BUILT ANALYSIS
Illustrative data

What are colleagues asking about?
Select a topic to send it back to the agent.

Questions by topic
Onboarding: 18
Access & permissions: 11
Agent workflows: 8
Finding documentation: 6
43 example questions · Select any bar

YOUR SELECTION
Onboarding
18 example questions

Find the recurring questions in this topic and suggest what the quick start should explain.

Investigate with agent →
Sends your selection back to the agent

1. Select a topic → 2. Send to your agent → 3. Continue the analysis

What you're looking at
✓ Agents run on your own machine and keep working when the browser closes.
✓ Sessions, files and agent-built interactive pages sit side by side in one workspace.
✓ Controls on an AUI page send choices straight back to the agent that built it.

13

Interface text:
dev-machine
V1.2.0
LIVE
Yard
Rooms
Views
+ New session
Starred
Research assistant · .../research
Onboarding guide · 3
Draft guide · .../onboarding
Explore questions · .../onboarding
Build fixtures · .../onboarding
Payments service · 2
fix/retry-backoff · .../payments
Review PR 418 · .../payments
AUI surfaces
Questions by topic
Aura · Explore questions
Web panels
Staging dashboard
localhost:3000
Draft guide
~/work/onboarding
Questions by topic
Aura · Explore questions
DJ
Message Draft guide... (Enter to send, Shift+Enter for a new line)

OCTOBER 13, 2026
EXPO SESSION
Sutton Room

[14:34](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=874s) [Workspace mockup with a session sidebar, onboarding conversation, and interactive chart of questions by topic.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf6978a596de40224d0135e15136231289bbc3cb5d48f7645d2bf58a5cbf7bdb.webp)

Give Users Tools They Want

Mock

Draft guide
CLAUDE CODE

Turn the research notes in this folder into a short onboarding guide.
Keep the open questions separate.

Read notes/*.md (14 files)

A first draft

I read the 14 notes and grouped 43 questions into four topics. The guide below covers the two that come up most often; the rest go in an appendix.

• Opening a session and giving it a task with a clear result.
• Keeping conversations, files and an interactive chart side by side.
• Starring a recurring assistant so it stays in every View.

I opened the topic breakdown as an AUI page beside this conversation. Select a topic there to send it back to me and I'll expand that section.

aui open questions-by-topic → pane 2

Onboarding first. Also check what the payments team wrote about access setup last month.

aui event select=Onboarding [illegible] Questions by topic
Grep "access" ~/work/payments/docs (3 matches)

Expanding the onboarding section now. The payments notes describe the same access flow with one extra step, so I'll fold that in rather than repeat it.

Illustrative conversation and data.

Questions by topic
AUI
AUI · AGENT-BUILT ANALYSIS
Illustrative data

What are colleagues asking about?
Select a topic to send it back to the agent.

Questions by topic
Onboarding: 18
Access & permissions: 11
Agent workflows: 8
Finding documentation: 6
43 example questions · Select any bar

YOUR SELECTION
Onboarding
18 example questions

Find the recurring questions in this topic and suggest what the quick start should explain.

Investigate with agent →
Sends your selection back to the agent

1. Select a topic → 2. Send to your agent → 3. Continue the analysis

What you're looking at
✓ Agents run on your own machine and keep working when the browser closes.
✓ Sessions, files and agent-built interactive pages sit side by side in one workspace.
✓ Controls on an AUI page send choices straight back to the agent that built it.

13

Interface text:
dev-machine
V1.2.0
LIVE
Yard
Rooms
Views
+ New session
Starred
Research assistant · .../research
Onboarding guide · 3
Draft guide · .../onboarding
Explore questions · .../onboarding
Build fixtures · .../onboarding
Payments service · 2
fix/retry-backoff · .../payments
Review PR 418 · .../payments
AUI surfaces
Questions by topic
Aura · Explore questions
Web panels
Staging dashboard
localhost:3000
Draft guide
~/work/onboarding
Questions by topic
Aura · Explore questions
DJ
Message Draft guide... (Enter to send, Shift+Enter for a new line)

Engineering the future of AI

### Give Users Tools They Want — Collaborative Room Mockup

This room mockup puts people and agents in one release workflow: agents split readiness checks into a sub-room, return a summary, and ask for human approval before deployment.

[15:00](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=900s) [Illustrative shared room where people and agents coordinate release checks and request human approval before deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/2697add027ec79d985da10312863efb4e458244f46a7e83d435e2fd3b47c83b5.webp)

Give Users Tools They Want

Mock

Release readiness · payments 2.14
room
label: release
label: needs-approval

Mia Alvarez — human — 09:02
@Release captain we want payments 2.14 out today. Run the readiness checks, bring in whoever you need, and stop for my approval before anything touches production.

Release captain — agent — WORKING — 09:02
Plan: three checks in parallel, then a go/no-go. I've asked Test runner for the CI matrix, Infra reviewer for the config diff, and Docs writer for release notes. Deliberation in a sub-room; you'll see the summary here.
open sub-room → readiness-council (3 agents)

Test runner — agent — COMPLETED — 09:06
CI matrix complete on rc2. One flaky contract test re-ran green; I've filed it for follow-up.

CI matrix · payments@2.14-rc2
unit: 412 passed
integration: 88 passed
contract: 1 flaky · retried OK
load: p95 184 ms

Infra reviewer — agent — INPUT_REQUIRED — 09:07
Config diff is small: two new feature flags and a connection-pool bump 32 → 48. The pool change is above the self-serve threshold, so this one needs a human. @Jordan Reyes you own that service.

Jordan Reyes — human — 09:09
Pool bump is fine, we sized for it last quarter. Approved from my side.

Release captain — agent — INPUT_REQUIRED — 09:12
Council summary: go. Test runner and Infra reviewer both green, Jordan approved the pool change. Docs writer's notes are in the checklist pane.

Approval requested · deploy payments 2.14 to production
All checks green. Release notes drafted. Scheduled re-check of CI in 10 min if you're not ready.
Approve
Ask for changes
timer · fires 09:22

PARTIES
Mia Alvarez — human · owner
Jordan Reyes — human
Release captain — agent · orchestrator
Test runner — agent · ci
Infra reviewer — agent · config
Docs writer — agent · notes

SUB-ROOM
readiness-council — 3 agents
Captain, Test runner, Infra reviewer deliberate here; only the summary is posted to the parent room. Open to watch the full exchange.
Open sub-room →

ROOM
owner: Mia Alvarez
created: today 09:02
events: 27
surfaces: Aura · Slack #payments-release
schedule: CI re-check in 10 min

How this works
A room is a shared conversation owned by the platform. People and agents are equal parties. Every surface (web, Slack) is a view onto it.

Illustrative people, agents and data

14

Interface text:
dev-machine
V1.2.0
LIVE
Yard
Rooms
Views
+ New session
Starred
Release captain
.../payments
Rooms 3
Release readiness · payments 2.14
6 parties · 2 humans · 4 agents · working
Incident 2291 · checkout latency
closed 2h ago · summary posted
Weekly data-quality sweep
schedule: next run 06:00
AUI surfaces
Release checklist
Aura · Release captain
6
@Release captain
Message the room... anyone can speak; @ asks a party to respond
DJ
OCTOBER 13, 2026
EXPO SESSION
Sutton Room

### The Optiver AI Platform

Optiver’s design connects user tools through a shared agent gateway, then routes model calls through an inference gateway to replaceable models. A learning loop turns agent traces into proposed fixes, with human approval.

[17:35](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1055s) [Platform architecture linking user surfaces, an agent gateway, shared agent capabilities, an inference gateway and models, with trust controls and a learning loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/14aa4acd6786ba0be8358e10eb72fb15d7267000a701c6c096014598b0cca9a9.webp)

The Optiver AI Platform

Trust

Single sign-on
People and agents sign in once. Agents act as the user or as themselves.

Permissions
Team-scoped access to agents, tools and data, enforced at the gateway.

Isolation
Each conversation runs in its own sandbox.

Data handling
Tiered access to prompts and traces. Aggregates by default.

Regions
Hosted where the data and the users are.

Surfaces
people meet the platform where they already work

Chat & web console
Our clients inside team messaging, the web, and the developer session hub.

Coding harnesses
Vendor CLIs and IDEs, run as managed sessions.

Agent Workbench
Define an agent in a text file. Run it locally or deployed.

Personal assistant
One assistant per person, on every surface, with one memory.

Your own client
Teams build their own interfaces on the same libraries.

Agent gateway
one conversation layer for everything

Durable conversations
A room anyone or any agent can join, from any surface, at any time.

Identity & policy
Every participant has an identity. Access and budgets are enforced here.

Schedules & triggers
Agents run on timers and events, not only on request.

Session broker
One isolated, budgeted session per conversation, started on demand.

Agents, capabilities and context
one runtime, any SDK or harness

Agents
Personas and custom agents on one runtime, local or deployed. Tracing, workspaces and agent teams built once.

Skills, plugins & tools
Written once, translated to every harness. Team-owned tools, one catalogue.

Knowledge & memory
Team knowledge bases, personal and team memory, enterprise sources reached as the user.

Inference gateway
every model call, one door

Routing
Workloads ask for a capability and a budget, not a model. The router chooses from evidence.

Resilience
Several channels per vendor, automatic failover, regional placement.

Economics
Caching policy and per-request attribution to person, agent and task.

Keys & budgets
Short-lived credentials. Budgets checked before the call.

Models
replaceable by design

Third-party models
Frontier and open-weight models from vendors and clouds, each reachable through more than one channel.

Self-hosted models
Open weights on our own GPUs, for independence and sensitive work.

evidence → improvement

Learning loop

Dreaming
Mines real agent traces and proposes fixes to knowledge, skills and workflows. Humans approve.

Evaluation & certification
Every agent has evals built from real traces. Evals gate releases, model upgrades and routing.

Spend controls
Budgets follow the org chart. Cost per resolved task is the metric.

Logging & observability
Every conversation and model call recorded: steps, cost, outcome.

What we build and own
Shared platform layer
Engines we expect to swap as the market moves

15

Interface text:
AI Engineer

## Full transcript

[0:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=12s) My name is Noyan. I'm the head of AI engineering at Optiver, and today I'm gonna talk to you a little bit about how we scale AI with P&L on the line. Like, what does that mean though, right? So first, I think to contextualize, uh, let me tell you a little bit about what we do at Optiver. So we're a market maker. Uh, what that means is that we facilitate trading. Uh, we're a trading firm. We trade with our own money, um, and we don't have clients. Uh, basically, you know, we... i-in the tech sense, we take a lot of data, you know, we build models, and then we make decisions with them,

[0:42](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=42s) and these decisions need to work in a very high-performance manner. You know, we make ten million plus trades a day, pricing a million plus instruments, and petabytes of data processed daily. And also, we're a pretty global firm. We have ten plus locations across the world, um, and that comes with, you know, certain implications that we'll get into for a second.  So typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make. Uh, it's something along the lines of you implement some change, you roll it out to users, and then users use it

[1:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=72s) for days and weeks, et cetera, and you collect some engagement metrics. Uh, and then over, you know, months and quarters, maybe years, you'll look at like revenue, longtime value, et cetera. I used to be at Apple, um, so I know how this works.  Um, but, um, at Optiver, we implement something, uh, we roll out that change to our traders and our, our quants, and literally within hours you can see the P&L impact. In fact, you know, at the top of every day, we basically zero out the balance sheet and then look by the end of the day how much money we made. So this has certain

[1:42](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=102s) implications around how, you know, we make decisions around how we want to scale AI and how do we wanna build things.  First of all, the feedback loop is much more immediate, right? Like, when you make a decision, by the end of the day, you'll start seeing the impact of that decision. Of course, like over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long term, but immediately you'll start seeing the impact of something. But the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly, especially

[2:12](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=132s) if, you know, there's like a special thing going on in the market. Let's say that there's a SpaceX AI IPO or something like that, and you miss out on that, then that's probably, you know, pretty costly. Which means that we need to make decisions pretty quickly. Uh, we need to be able to measure them really effectively and really efficiently, and we gotta have a bias for action. Um, that means like the ceiling of the value and, and importance of a decision, decision you make is really, really high. Um, so in the, in the case of like a service, like a user, uh, facing service business like Spotify or whatever, like

[2:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=161s) fundamentally they're competing for a finite amount of attention of the customers versus with your trading, the more you trade, the more money you make, right? So, you know, the economics are a little bit more different. So to accomplish this, uh, and to be able to scale and operate at this way, we basically needed to build our own AI platform, um, around a few core principles. Move fast. I mean, we talked about that already, but of course, part of moving fast is that being able to be modular because, you know, in the AI world every day

[3:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=191s) there's a new model, there's a new product, there's a new paradigm. We're doing loops. No, we're doing swarms. No, we're doing graphs, et cetera. And we need to be able to account for that. And if that paradigm is valuable, like measure it. And if it is, bring it in because the moment you bring it in, you're gonna start, you know, getting more value. And of course, to be able to understand that we need to be able to measure at every layer. You know, of course like trading means lots of backtesting, et cetera, but also, you know, we need to... When a new model comes out, yes, there's of course external benchmarks as to how this model is performing, but we also need to be

[3:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=221s) able to understand how in our internal benchmarks these things perform because the data distribution of the benchmarks that we have is different from what they have a-available externally. Now, the thing is traders are pretty smart people. They're pretty high agency people, so we want our platform to be very self-service. But of course, anytime you provide something purely self-service, it can be really chaotic, so we also need to be able to provide good defaults. Uh, and one way we kind of have this, um, way of working with them is also being able to forward deploy. So we have a fairly

[4:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=251s) decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade, teach them how AI can add more value to them, learn and bring it back to the platform. Now, of course, you've all heard the stories of like OpenAI Hugging Face, et cetera. So of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner. Uh, as you can imagine, uh, there's potentially, uh, pretty negative impacts if the agents kind of go haywire and do stuff with our stack. Um, but I do wanna underscore that everybody in the

[4:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=281s) company is using AI. Like everybody's using it a little bit differently, but fundamentally we're very, very, um, AI native I guess is the term that we're using now. Um, and of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on-prem. So we have our own on-prem GPUs, our on-prem infrastructure, et cetera. The only thing that we phone home outside of our service really is for like third-party model inference. And of course, in those cases we have like things like zero data retention, et cetera, agreements with the companies.

[5:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=311s) So here's kind of like a very, you know, shock and awe, like a lot of information on the sl-slide type of view of our AI platform. I don't expect you to, you know, digest all of this, but you know, if you wanna take a photo and talk about it to me afterwards, uh, be my guest. I'll flash the slide again at the end. But broadly, the things that I wanna highlight is that, um, we, uh, have to be accountable for every layer of the stack. At the bottom, you know, we have models from third-party providers. We also self-host, uh, our own models and the open source models. We have an inference gateway that provides

[5:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=341s) things like routing, uh, between these different models. Um, we can, uh, measure how much we're spending on each of these models. We can assign budgets, et cetera. I'll go to that in, in a second. And of course, this gives us resilience. Like if one provider goes down, we can always swap out to a different provider, et cetera, because again, any time that we're down, you know, that's basically we're not making money. Um, and then we have our own agent platform where Uh, agents have, uh, registered, uh, capabilities like skills, plugins, tools. Each agent in each team with

[6:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=371s) using an agent can bring in their knowledge and memory systems. We have an agent gateway, which is basically like a s- single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management, uh, et cetera. So this kind of goes into things like the agent swarm, et cetera, concepts that you also may have heard about. And finally, what's really important to me, I'm a product guy as well, is that, um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to

[6:40](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=400s) use it, then it doesn't really matter. So we've built our own kind of, um, web interface for people to access like their sessions, et cetera. Uh, we have this thing called our Agent Workbench, which is basically persona management. Teams can kind of build their own profile of how they use agents and share. Uh, we have a personal assistant that everybody in the company, uh, has access to. Um, yeah. So again, I'll get back to this later, uh, but if y'all have questions, we can talk about it afterwards. Right. So how do we move on these principles? Uh, one principle that we have is that,

[7:10](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=430s) you know, if a new frontier model is made available publicly, uh, we wanna be able to leverage it in the production flows the same day because there's immense value in that if it is valuable. But is it valuable, right? We wanna be able to benchmark it on golden sets. So that means that we need to have a mature evaluation framework, um, and then we partner with key customer trading teams where we collect their benchmarks, we onboard it onto them, and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the f- folks in the company to see, hey, is this really valuable for our use

[7:41](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=461s) cases? But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway. We have the right APIs, interfaces, we have the cost management, et cetera, set up correctly, and oftentimes new models mean, like, the harnesses need to be updated, et cetera, and we also manage the update and delivery of that as well. And then of course, yes, we launch it, but then that also comes with the expectations. We're a global company, so we need to have round-the-clock support. So our team, you know, has, in different regions, uh, support rotations to make sure that the uptime is

[8:10](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=490s) high. I think some people here in the audience are actually on support today, but anyway. Um, and then of course, like, we run these benchmarks. We also need to be able to have, um, the people in the company understand like, okay, we looked into this model, we benchmarked it, here's how well it works, here's how you use it, here's our recommendation, and kind of keep that communication line open.

[8:30](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=510s) Um, another thing is that because we have optionality between different models, uh, if any model is adding value, we wanna integrate it, uh, and we need to be able to insulate against model availability in, like, different regions, uh, or different, you know, cloud providers, et cetera. Uh, and also we have a requirement that some of our workloads run on a completely DMZ, uh, on-prem zone. Uh, so we also-- There's cost reasons for that. There is, um, reasons of isolation for that. So for those reasons, we wanna be able to be agnostic to the model and harness. So which means that

[9:01](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=541s) we have the third-party providers, uh, we have on-prem models, and all of it kind of goes through our inference gateway. We log everything. Uh, every interaction going with the models is logged, and we also automatically m- move them to our evaluation system. Uh, we have a bunch of different harnesses that we support, and on top of that, we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses. Um,  so teams can define their own harness configs in this meta harness layer, and that plugs

[9:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=571s) into the team's knowledge base. Automatically, knowledge is harvested over time. We look at the traces and then derive ways to, uh, get additional knowledge as well. Uh, and then through this meta harness, you can either deploy managed agents on our platform, or you get access to them through the web portal IDE UI that I described earlier. I see some people taking photos, so I'll give it a second. All right.  So, um, this type of chart may be familiar to a bunch of you. Uh, this is kind of a

[10:01](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=601s) roughly illustrated version of how much we're spending on AI. No numbers on it. Uh, but we are doing more with it. And this is, you know, in December, you see like the Claude Code moment and things go up. By the way, this inflection point here is, like, when I joined the company, so I don't know if I have anything to do with that. Um, and then, um, then we have here, you know, like, the big frontier model releases of the summer. As you can see, like, spend really shot up there. But then we kinda got it back under control because we built some optimizations into the process that helped us really get this thing, uh, in check.

[10:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=631s) So, um, the thing is, the more we spend, the more value we can get. We are pretty confident o- that we can, you know, keep getting returns, but we wanna make sure that we're spending wisely, right? So one thing that we discovered was that, um, y- we have a bunch of MCPs, we have a bunch of knowledge bases and information, et cetera, um, and on a f- fresh session, we were seeing that, you know, sometimes people have like 200K tokens worth of context, like, by default minimum, sometimes even more. That's, that's ridiculous, right? Because everybody was kinda dumping a bunch of stuff in there. This is the kind

[11:01](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=661s) of problem with, like, doing s- things completely self-service, right? We need to have sane defaults. So we implemented a bunch of different changes. Uh, we brought in MCP tool search. Uh, we gave the agents the option to, um, rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is, like, much more efficient. Um, we looked at the caching characteristics of, like, when caches are getting invalidated. How long do we need to set these parameters? Do we need to set them differently? Do we set them dynamically? Um, and then we worked with the individual teams to kind of come up with

[11:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=691s) further optimizations to their harnesses. Um, you know, we- do we put some things into the knowledge base rather than this, like, massive, um, system prompt, uh, et cetera? Uh, and the good thing about these improvements is that, um, these things all scale with token cost, right? So basically, um, if the tokens are more expensive, you're saving more on this. But the point here is that, um, yes, you can dump models out there and make them available to people, and I'm not even g- going into the work that we're doing around things like, you know,

[12:01](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=721s) model routing, um, or sub-agent optimizations, et cetera. Sheer single session stuff, um, there's almost always value left on the table and we're, you know, ruthless- ruthlessly optimizing towards that.

[12:15](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=735s) Yeah, there we go. And we shipped optimizations at this point in time, but of course, there's always, like, rollout, et cetera. Uh, our tech stack is a little bit interesting when it comes to rolling out across the different global regions. Uh, but then also other things that really helped us spend less was that, um, turns out that if you tell people, like, how much they're spending and, like, give them a little bit more visibility, a little bit more alerting, give them the agency to self-manage, they actually do a pretty good job, you know? We built a system to basically have teams kind of set their own alerting and limits, and we've seen, like, many teams,

[12:45](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=765s) you know, basically set reasonable limits for themselves. Another option is, you know, exploring open source models. Uh, you know, there are a bunch of open source models that are, on token cost, pretty competitive with the frontier models out there. It's not necessarily right for every use case, uh, but we've found opportunities where they're a good value replacement, and we've been basically making that available internally, uh, and we've seen more and more people, uh, leverage those. Then of course, like, there's things like model routing, et cetera. There's, um... We've been

[13:15](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=795s) exploring different approaches to dynamically send the prompt to, like, a cheaper model, et cetera, or sometimes have a way where, um, we have the, like, plan generated by, like, a more frontier model and the implementation be broken down by smaller models, et cetera. Um, and finally, you know, I think knowledge bases are underrated. Uh, which, I mean, they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context. Um, so

[13:45](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=825s) long story short is that we're still investing and using a lot of tokens, but this is just about getting more ROI out of the same tokens.

[13:54](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=834s) And finally, I kinda wanna talk about, uh, meeting the users where they're at. Um, so a few months ago, we were in this place where, like, a bunch of people are using different harnesses. They have all these weird bespoke setups to kind of tmux into their dev boxes, and some people were building, you know, hacky web UIs, et cet- et cetera, whatever. So basically what we said was like, "Hey, you know, there's a bunch of people doing these types of things. Why don't we just build a nice version of this and then make it available to everybody in the company?" So that's exactly what we did. In our web portal, we built this basically system where you can manage your

[14:24](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=864s) sessions, you can have chat windows, you can have, like, uh, artifacts in there. Um, there are similar products out there offered by, you know, you know, some different, uh, harness companies, but, uh, we feel that this worked very well with our infrastructure, um, and we can kind of iterate it at the rate that we would like to. Um, so basically, this was actually, uh, also at a, at a place where different traders in the company are now kinda contributing code. They find something that's valuable to their workflow, and then they, they just send up a

[14:54](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=894s) PR and we just update it, and then now we have things like, you know, collaborative document editing and things like that. And in fact, we're kind of getting this to a place where, uh, we're doing this, you know, agent multiplayer thing where now I have my sessions in there. We can create rooms with, like, my agents, your agents, like, other deployed agents, and we can kind of have all of that in one place, and this really helps, uh, people be more productive and, uh, you know, we have now a Slack client, et cetera, so you can also do it on the go and whatnot. So

[15:24](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=924s) basically, um, our users are very high agency. Uh, they will go and build something if we don't build it for them, so we just build a better version of it.

[15:34](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=934s) Yeah. So, uh, basically, this is, um, the overall overarching philosophy of how we did this. Um, we have a platform, very modular, uh, and, um, we continue to iterate on it, but so far we're having a pretty good time. Um, that's the end of the talk, uh, but if you have questions about anything on this or any other area, we can talk. And by the way, we have a networking event, uh, this evening. Uh, please come to our booth and register. It's gonna be fun. We have cool prizes.

[16:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=971s) Questions?

[16:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=973s) Can you talk a little bit more about what your knowledge base looks like? Are you using stock and storage vector search?

[16:19](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=979s) Yeah. So we've, um, definitely been exploring a f- bunch of different options. Um, personally, I'm not happy with any of the options out there. There's trade-offs, but I mean, if you ask, like, some of the frontier labs what you do for knowledge base, they have, like, just huge markdown files, hierarchical markdowns, et cetera. Um, there's places where that works, but, um, there is, um... Vector search has its own benefits and disadvantages. We have a hybrid approach that kind of, uh, allows us to explore these different options.

[16:49](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1009s) Personally, I want us to spend more time around, like, doing things like perhaps some graph databases, et cetera, that are going to be, uh, more valuable. I mean, every day on Twitter I see something, like, new way of, like, m- handling memory and knowledge. Um, that's a problem that we want to spend more time on. I'm not happy with our solution.

[17:10](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1030s) Go ahead.

[17:11](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1031s) Um, what do you think is-- what are your biggest value gains from, like, either hand-built versus Codex solutions? Like, what kind of

[17:21](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1041s) Yeah. Good question. Um, value gain from using hand-built solutions versus Codex. So there's a few things. Um, I mean, one kinda boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to. Um, un- until recently, Codex didn't really have the option to kind of run managed agents, et cetera, or some of these things on your own infrastructure. Now they've started offering them, and we've been working with them on it actually. But, um, that was one reason why we started this

[17:51](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1071s) way. Secondly, I think the, um, most-- many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution. But the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move. Uh, we've actually encountered with, um, that with one of the, uh, inference gateway providers that we used to use. Um, and

[18:21](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1101s) basically, we would encounter scenarios where something comes out, we need it, and they're like, "Okay, uh, let's talk to our GTM team. Let's put you on the roadmap. We'll..." And it's like, and then by that point it's like, okay, no, we need to actually move faster. Um, so that's also one of the reasons. Uh, and finally, I think the In some of these areas, we've been, um, consistently ahead of the curve. Like talking to some of the frontier labs, um, the, the way that we've been doing managed agent

[18:50](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1130s) capabilities, et cetera, like they're asking us to tell them how we did it.  Not to be bragging too much. But I think the, um, we feel pretty confident in our engineering and kind of our roadmap, so that's also, uh, been well. I don't think this approach works well for everybody, but I think we have a unique, um, coalition of circumstances that allow for us to do that. Go ahead.

[19:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1153s) Uh, can you talk a bit about your tool chain in terms of, uh, best practices, like integration to, um, the other part of this platform such as eval or memory?

[19:24](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1164s) Um, yeah. So ju- just to make sure I understand your question correctly, um, what are some of the best practices of how we integrate the rest of this into eval and memory?

[19:33](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1173s) Uh, culture tool shift. So the scale of function.

[19:36](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1176s) Right. Um, so I think the-- we started at a place where, you know, we log everything. Like I said, every tool call input, output, et cetera, uh, just hotel logging, and, um, that goes into our, uh, evaluation system. Um, the challenge that we found with that, and we-- it's, uh, kind of an open problem, I would say, is that, um, when you're kind of r- trying to evaluate like a tool call, I mean, we have like, for example, a support bot agent that we run in Slack,

[20:06](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1206s) um, and we want to evaluate that agent. Let's say, what if we ran this agent with a different model? Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it, and that's the pretty, like, typical problem. Um, we've been working on basically things like can we snapshot world states with these tools? And depending on the system that it integrates with, it's, you know, there's different answers to that question. If you're looking at some, like, trading backtest database, easy like that. You can always point in time. That's what we do. Um, but if you're looking at some other system,

[20:36](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1236s) like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated? Maybe there are some ways to kind of replace Slack for a particular point in time, but other systems, not really. So we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation, uh, and we still log it as such. And when we're rerunning the eval, we will run the tool with kind of saying, "Okay, I want kind of this snapshot." We have like a proxy around the MCPs. Um, but you know, it's imperfect and,

[21:06](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1266s) uh, I don't know if there's a perfect solution other than just record, you know, everything.

[21:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1273s) Yeah. Do we have-- Maybe we have ti- time for one more? Go ahead.

[21:18](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1278s) So let's say you built some agent for some team, um, and they really like it. Now they want to scale, uh, uh, drastically the use case.

[21:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1291s) Yeah.

[21:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1291s) How do you system design that or how do you think about that?

[21:36](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1296s) That's a very classic problem that we've actually run into several times. Uh, you know, we had this one team that ha- built an agent that did like coding and, uh, like analysis with them. Uh, it was kind of scoped to their repo, and it was like, well, what if we also ha- worked with this team and have their thing also come with us and immediately kind of explodes in complexity. Now it's checking out multiple repos at a different auth, et cetera. Like typical problem. Um, so initially our solution was explore, okay, what if we like literally just did that? What if we just made the new agent that just had

[22:06](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1326s) a union of the functionality? But oftentimes that ends up, you know, a lot of tech debt, works very poorly, et cetera. I think our current answer to that type of paradigm is, obviously every agent is versioned, et cetera. So you can, if it's small increments, you can push a new version to the agent and make, make new capabilities. But going forward, we would more encourage people to basically leverage our agent gateway, where rather than one agent having like a ten different things that it does, scope your agents to like a task that they're good at, you can easily measure them at, you can easily

[22:36](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1356s) manage the life cycle of it. And then there'll be another agent that it can talk to easily so as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, et cetera. And that will help us, um, scale these types of functionality more effectively rather than just bloating one agent to do everything. Uh, it becomes kind of like a multi-agent orchestration problem, but, you know, that's-- I think keeping things more measurable, uh, is really

[23:05](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1385s) important because if you really bloat the agent, then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tw- tweak it easily.
