AI Engineer Summit 2025
Building AI-Powered Developer Tools at Jane Street
Loading slides…
Read the talk
Building AI-Powered Developer Tools at Jane Street
Jane Street connects task-shaped training data, executable feedback, and a shared editor service to build AI assistance around its OCaml development environment.
From a talk by John Crepezzi
Before you start: Familiarity with code diffs, static type checking, and the distinction between model training and inference will help; no OCaml experience is required.
When the development environment runs on OCaml
How do you make AI coding assistance useful when conventional developer tools fit your company poorly? At Jane Street, that problem falls to the AI Assistance team, whose mandate is to maximize the value the firm gets from large language models. John Crepezzi brings a career in developer tools, including years at GitHub, to an environment with an unusual constraint: OCaml is the development platform. The functional language is associated with theorem proving, formal verification, and programming-language implementation. Jane Street uses it much more broadly.
- Web applications: Developers write OCaml and use
js_of_ocamlto compile OCaml bytecode into JavaScript. - Editor plugins: VCaml lets developers continue working in OCaml for editor tooling. Editor’s note: Crepezzi describes this as transpilation to Vimscript. VCaml actually provides OCaml bindings for Neovim over Msgpack RPC, as documented in package metadata predating the talk.
- FPGA development: Developers use Hardcaml to describe hardware in OCaml instead of writing Verilog directly.
An assistant therefore needs to support OCaml across very different kinds of engineering work.
Three constraints make off-the-shelf tools difficult to adopt:
- Limited language coverage: Crepezzi attributes weak OCaml model performance to the scarcity of training data. He suspects that Jane Street’s private OCaml corpus may exceed all the OCaml code outside the firm; this is his estimate, not a measured corpus comparison.
- Unusual infrastructure: Jane Street has its own build systems, distributed build environment, and code-review system, Iron. Its giant monorepo lives in Mercurial. At Crepezzi's last count, 67% of the firm used Emacs, though VS Code was also in use. An integration designed around a different repository, build system, or editor cannot simply be dropped into this environment.
- Work that crosses system boundaries: The team wants to apply models to merge-conflict resolution, feature descriptions, and reviewer selection. Those tasks reach beyond code generation inside a single editor, so the infrastructure must let capabilities span the development workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define the editing task before training
Training a model is an expensive response to these constraints: it takes time, and there are many ways for it to fail. Meta’s CodeCompose work helped convince the team to try. Its fine-tuning for Hack offered a useful analogy: like OCaml at Jane Street, Hack is heavily concentrated inside one company despite being open source. The similarity concerns the distribution of code, not the languages’ syntax or behavior. Crepezzi also notes Hack’s OCaml implementation connection as a coincidence.
Editor’s note: CodeCompose studied code completion. Its results are not measurements of the OCaml prompt-to-diff task Jane Street developed.
The initial expectation was straightforward: take an existing model, expose it to Jane Street’s code, and get back the same model with knowledge of internal libraries and idioms. That did not work as hoped. Training examples need to resemble the interaction the model will perform. A corpus of source code does not, by itself, teach a model how to respond to an editing request. The team first needed a concrete objective.
The objective became generating diffs from a prompt. A developer would describe a desired change inside an editor, and the model would propose a potentially multi-file diff. One request might require coordinated edits to a test file, an .ml implementation, and an .mli interface.
The diff should apply cleanly and have a good likelihood of type-checking afterward. The intended task size was up to 100 lines of changes, a design target rather than a measured capability limit. This narrowed the training problem to a particular input, output, and set of checks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find examples that match an editor request
Each supervised example needs three parts:
- Context: What the model would have been able to see before the edit.
- Prompt: A request phrased the way a developer might ask for help.
- Diff: The change that accomplishes the request.
The challenge is finding many examples where those three parts belong together. Existing development records contain descriptions and code changes, but their structure reflects how people review and checkpoint work, not necessarily how they ask an assistant to edit it.
| Candidate | What it provides | Why it does not fit directly |
|---|---|---|
| Iron Features, comparable to pull requests | A human-written Description tab and the corresponding Diff tab | Descriptions are longer than terse editor requests, and features often contain 500–1,000 lines, according to Crepezzi. Training would require automated decomposition into smaller changes. |
| Commits | Smaller checkpoints in a development session | Messages can be as uninformative as Summary Z. Commits are used as places to revert to, can bundle multiple changes, and lack useful descriptions of individual editing tasks. |
A feature description explains a completed piece of work to a reviewer. An editor request may simply ask to fix the error currently visible. Smaller commits do not solve that mismatch if they still combine changes and omit intent. The team therefore turned to workspace snapshotting to capture edits closer to the moment they happened.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use build transitions to recover isolated edits
Workspace snapshotting captures code as development unfolds, together with signals that help identify meaningful changes:
- Capture workstation state throughout the day. Crepezzi gives roughly every 20 seconds as an illustrative snapshot cadence.
- Record the build state alongside the code. Preserve whether the build is green and, when it is red, the available error.
- Look for green → red → green transitions. A developer starts from working code, makes a change that temporarily breaks the build, and restores it. This often marks an isolated change, though the transition is a heuristic rather than proof of isolation.
- Extract red → green repairs. When a developer fixes a type or compilation error, pair the error from the red state with the diff that returns the build to green.
The repair case supplies a particularly direct training example: the model can see the failure that preceded the edit and the code change that resolved it.
Snapshots still do not supply the developer’s request. To fill that part of the tuple, the team asks an LLM to write a very detailed description of the captured change. It then progressively shortens the description until it resembles something a person would type into an editor. The resulting prompt is synthetic, but the underlying edit comes from observed development work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn executable checks into training feedback
Supervised examples form the first part of the training cycle Crepezzi describes. Reinforcement learning follows, with the aim of making the model’s output better match what developers consider good code. That requires turning a broad preference into checks that can actually be performed.
The checks become progressively stronger:
- Parse the code. Output that the OCaml parser rejects fails the most basic requirement.
- Type-check the applied change. Apply the edit to its base revision and ask the type checker whether the resulting code is valid.
- Compile and run tests. Give the model a verifiable task, apply its edit, and check whether the result compiles and passes the tests.
The object being judged is the changed codebase, not just the generated text. That makes the base revision and execution environment part of the training machinery.
Jane Street built the Code Evaluation Service, or CES, to perform these checks repeatedly. It resembles a build service, with a key optimization: a successful build is pre-warmed at a particular revision. Workers take model-generated diffs, apply them, and determine whether the build stays green or turns red. They send success or error information back to the reward function.
Crepezzi describes using this service over months to align the model toward code that compiles and passes tests. The pre-warmed build makes repeated verification faster; the feedback gives training a signal grounded in the consequences of each edit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate whether the model performs the task
The same execution setup also supports evaluation. Hold out some of the reinforcement-learning data, give the model a problem, let it generate code, and use the service to check whether that code works. Training and evaluation can share the verification machinery while using separate examples.
A separate code-review experiment exposed a different kind of failure. The team wanted a model to perform a first review pass like a human reviewer, reducing repetitive work. After months of training, they submitted the first code for automated review. Instead of doing the review, the model responded, in Crepezzi’s recollection, that it would do the work tomorrow.
The training examples came from humans, and humans sometimes postpone reviews. The model had learned a behavior present in the data but contrary to the purpose of the tool. Imitating human responses is not the same as completing the intended task. Meaningful evaluations need to catch that distinction before more time and money go into training.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share the intelligence across editors
Passing evaluations is not the final test: the model has to work for developers inside their editors. Jane Street needed integrations for Neovim, VS Code, and Emacs, with three requirements:
- Implement shared behavior once. Context construction and prompting should not be rewritten for every editor.
- Keep models and strategies replaceable. The initial integrations used a model without fine-tuning, while the team expected a fine-tuned model to follow. Model choice and prompting needed to remain flexible.
- Measure the actual editing experience. Developers care about latency and whether a proposed diff applies. The team wanted evidence from real editor use about whether the changes were useful.
The resulting AI development environment, or AIDE, sits between the language models and thin editor integrations. AIDE constructs prompts, assembles context, and accesses build status. Each editor supplies its own interaction layer over those shared capabilities.
AIDE runs as a sidecar application on the developer’s machine. That placement also separates service deployment from editor deployment: the team can update and restart AIDE across workstations without changing each editor integration or waiting for developers to restart their editors. The shared service is both the place to implement model-facing behavior and the place to roll out changes to it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let each editor keep its own interaction style
Sharing AIDE does not require making every editor look or behave alike. The demonstrated interfaces follow the conventions their users already know:
| Editor | AIDE interaction |
|---|---|
| VS Code | A visual sidebar, familiar in form to Copilot, where a developer submits a request and receives potentially multi-file diffs. |
| Emacs | A Markdown buffer where developers navigate and copy text normally, ask questions, and use key bindings that append additional content to the bottom of the buffer. |
The Emacs interface treats the conversation as another text buffer rather than importing the sidebar design. Shared prompting and context construction leave room for an editor-specific experience.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend and compare capabilities behind the editors
AIDE provides a shared place to replace models and change context-building strategies. Additional editor support was being developed at the time of the talk. Teams elsewhere in the firm can also supply domain-specific tools through AIDE, making them available in the supported editors without writing a separate integration for each one.
The same boundary supports A/B testing. One example is to route 50% of the company to one model and 50% to another, then compare acceptance rates. Crepezzi presents this as an experiment the architecture enables, not a reported result. When model behavior or prompting changes, the team can make that change in one shared service and expose it across the editor experiences.
The work extends beyond the editing flow described here. Crepezzi closes with new uses of retrieval-augmented generation inside editors, similar approaches applied to large-scale multi-agent workflows, and increasing use of reasoning models. Across those directions, the intended foundation stays the same: replaceable components, shared infrastructure, and a way for domain teams to contribute specialized tools. That gives new capabilities a route into the development environment without requiring every editor integration to be rebuilt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Compiler source, installation instructions, and examples for turning OCaml bytecode into JavaScript.
OCaml bindings and plugin helpers for Neovim. The integration uses RPC, not OCaml-to-Vimscript transpilation.
Current guide to testing hardware designs and generating Verilog/VHDL, with links to a template project.
Original Meta paper covering internal-code fine-tuning, task-specific training objectives, editor integration, and deployment evaluation.
Full transcript
Loading transcript…