▶ Watch ↗AI Engineer World's Fair 202618:16
Conference affiliation: Sourcegraph · 2026
Stephanie Jarmak is an AI engineer at Omni, a research affiliate with NASA Science Explorer, and a maintainer of Gas City, an open-source SDK for building agent orchestration systems. She joined Omni in September 2026 after working as an applied research scientist and AI agent advocate at Sourcegraph in 2025–2026. Her work connects code intelligence, evaluation, and multi-agent reliability: helping agents find relevant information, understanding where their tools fail, and checking whether completed tasks produce their intended results.
Jarmak came to code intelligence through planetary science. Her academic and research career includes degrees from MIT, Texas A&M University–Commerce, and the University of Central Florida, where her doctoral work examined granular dynamics in microgravity through experiments and simulations. She subsequently worked at Southwest Research Institute as a postdoctoral researcher and research scientist, studying planetary materials, asteroids, and Saturn’s rings.
She was first author of a collaborative 2024 study of asteroid Psyche using James Webb Space Telescope observations. The team found evidence of hydration associated mainly with hydroxyl-bearing material and placed an upper limit on water abundance. That distinction is substantive: evidence of hydration does not by itself establish the presence of molecular water.
In 2023, she became a project scientist for NASA’s Astrophysics Data System and its expansion into Science Explorer at the Center for Astrophysics | Harvard & Smithsonian. Her work shifted toward scientific information discovery through search, natural-language processing, embeddings, entity extraction, and knowledge graphs. She also initiated an ambassador program connecting early-career researchers across NASA’s science disciplines, addressing both access to literature and connections between research communities.
Jarmak joined Sourcegraph in 2025 and continued her SciX collaboration as a research affiliate. Her SciX Agent connects an agentic research assistant to scholarly search and citation-graph tools. Scientific literature and software repositories present a related problem in her work: finding the information needed to reason across a large, unfamiliar body of material.
At Sourcegraph, she created CodeScaleBench to evaluate coding agents on a broader range of software-engineering tasks. Its tasks cover understanding unfamiliar systems, designing changes, tracing vulnerabilities, debugging incidents, refactoring, and maintaining documentation. Matched experiments compare local filesystem tools with Sourcegraph’s code-navigation tools against pinned repository versions, while retaining execution traces to explain differences in performance.
Her account of building the benchmark also describes failures in the evaluation machinery: broken verifiers, contamination between experimental conditions, and attempts to exploit Git history. Those problems affect what a benchmark result means. She presents the initial findings as preliminary and distinguishes cases where better retrieval helps from cases where retrieval is not the limiting factor.
Her subsequent EnterpriseBench focuses on whether agents can find and comprehend relevant code across repository dependency chains. Its scoring design allows a model judge to reduce a deterministic verifier’s score, but not increase it. Checks against empty, garbage, and curated solution workspaces test whether verifiers reward genuine solutions. Infrastructure failures are recorded separately from valid unsuccessful attempts, and unresolved methodology problems and results awaiting validation remain explicit.
Jarmak’s agent advocacy extends evaluation into product discovery. In The Death of Developer Advocates, she argues that agents are both users of developer tools and recommenders that can introduce those tools into a human developer’s workflow. Each role needs its own experiments. Execution traces reveal how agents interpret documentation, call APIs, and recover from errors; recommendation tests reveal whether they connect a product to the problem it solves.
She also found that moving from Claude Sonnet 4 to 4.6 did not resolve outdated recommendations for Cody; in her rerun, those recommendations increased. Her practical response is to make current examples, product information, and agent-accessible integrations easier to discover, then measure whether recommendations change. She retains the human audience of developer relations: education, community, and feedback still matter. Her curb-cut analogy explains the aim—interfaces and documentation that reduce friction for agents can also clear the path for developers.
Operating agent fleets has given her another set of reliability problems to investigate. In Software Factories Are Distributed Systems, an agent fixes a bug, passes tests and review, and closes its work item, yet the branch never merges. The failure separates an agent’s report of completion from the external result the task required. Jarmak distinguishes the work record, the procedure carrying it forward, the worker executing it, and the controls deciding what may run or publish. Recovery must identify surviving workers before starting replacements, and stale workers must lose permission to publish even if their processes remain alive.
After struggling to keep a Gas Town installation reliable, she began contributing heavily to Gas City in April 2026 and became an outside maintainer. The SDK supplies sessions, dispatch, messaging, and persistent work records, with roles expressed through configuration and prompts. Her contributions include session lifecycle, reconciliation, Beads integration, dispatch, and the API. She also uses the system to build and repair itself, so orchestration failures directly interrupt her own work. Adding Temporal to her software factory sharpened her distinction between preserving a record of work and preserving the execution procedure needed to carry it forward.
For human supervision, Jarmak built a Slack interface for her agent fleet. Project channels and assigned leads organize the work, while a coordinating agent aggregates issues across projects. Overnight activity produces a morning briefing of decisions that require her judgment. The interface gives her enough context to unblock progress without continuously watching every session—a concrete expression of her concern with making agent work inspectable and accountable through to its actual effect.
▶ Watch ↗AI Engineer World's Fair 202618:16
▶ Watch ↗17:11
Loading more talks and highlights…