Stephanie Jarmak

Stephanie Jarmak is an applied research scientist and AI agent advocate at Sourcegraph, a research affiliate with NASA Science Explorer, and a maintainer of Gas City, an open-source SDK for building agent orchestration systems. Her work connects code intelligence, evaluation, and multi-agent reliability: helping agents find relevant information, understanding where their tools fail, and checking whether completed tasks produce their intended results.

From planetary science to information discovery

Jarmak came to code intelligence through planetary science. Her academic and research career includes degrees from MIT, Texas A&M University–Commerce, and the University of Central Florida, where her doctoral work examined granular dynamics in microgravity through experiments and simulations. She subsequently worked at Southwest Research Institute as a postdoctoral researcher and research scientist, studying planetary materials, asteroids, and Saturn’s rings.

She was first author of a collaborative 2024 study of asteroid Psyche using James Webb Space Telescope observations. The team found evidence of hydration associated mainly with hydroxyl-bearing material and placed an upper limit on water abundance. That distinction is substantive: evidence of hydration does not by itself establish the presence of molecular water.

In 2023, she became a project scientist for NASA’s Astrophysics Data System and its expansion into Science Explorer at the Center for Astrophysics | Harvard & Smithsonian. Her work shifted toward scientific information discovery through search, natural-language processing, embeddings, entity extraction, and knowledge graphs. She also initiated an ambassador program connecting early-career researchers across NASA’s science disciplines, addressing both access to literature and connections between research communities.

Jarmak joined Sourcegraph in 2025 and continued her SciX collaboration as a research affiliate. Her SciX Agent connects an agentic research assistant to scholarly search and citation-graph tools. Scientific literature and software repositories present a related problem in her work: finding the information needed to reason across a large, unfamiliar body of material.

Evaluating agents on unfamiliar codebases

At Sourcegraph, she created CodeScaleBench to evaluate coding agents on a broader range of software-engineering tasks. Its tasks cover understanding unfamiliar systems, designing changes, tracing vulnerabilities, debugging incidents, refactoring, and maintaining documentation. Matched experiments compare local filesystem tools with Sourcegraph’s code-navigation tools against pinned repository versions, while retaining execution traces to explain differences in performance.

Her account of building the benchmark also describes failures in the evaluation machinery: broken verifiers, contamination between experimental conditions, and attempts to exploit Git history. Those problems affect what a benchmark result means. She presents the initial findings as preliminary and distinguishes cases where better retrieval helps from cases where retrieval is not the limiting factor.

Her subsequent EnterpriseBench focuses on whether agents can find and comprehend relevant code across repository dependency chains. Its scoring design allows a model judge to reduce a deterministic verifier’s score, but not increase it. Checks against empty, garbage, and curated solution workspaces test whether verifiers reward genuine solutions. Infrastructure failures are recorded separately from valid unsuccessful attempts, and unresolved methodology problems and results awaiting validation remain explicit.

Agents as tool users and recommenders

Jarmak’s agent advocacy extends evaluation into product discovery. In The Death of Developer Advocates, she argues that agents are both users of developer tools and recommenders that can introduce those tools into a human developer’s workflow. Each role needs its own experiments. Execution traces reveal how agents interpret documentation, call APIs, and recover from errors; recommendation tests reveal whether they connect a product to the problem it solves.

• Tool-use friction: One trace exposed an agent using an expected read-tool parameter that the interface did not support. The error message let it recover, but the mistake still consumed a turn. Clearer tool descriptions could prevent that cost. Jarmak treats latency, token use, and avoidable retries as part of the user experience, rather than judging a tool only by whether the task eventually succeeds.
• Recommendation gaps: Her recommendation pilot found a different gap. Sourcegraph appeared in roughly 65% of responses to prompts explicitly shopping for code-intelligence tools, but received no mentions in the problem-focused prompts she tested. One described shared-library changes breaking downstream services because the team could not see all their consumers. The response suggested maintaining a wiki rather than a tool for navigating dependencies across repositories. She used that result to form a testable hypothesis: product messaging might describe the category more clearly than the situations in which someone needs it. Website and content changes were proposed interventions, not established improvements.

She also found that moving from Claude Sonnet 4 to 4.6 did not resolve outdated recommendations for Cody; in her rerun, those recommendations increased. Her practical response is to make current examples, product information, and agent-accessible integrations easier to discover, then measure whether recommendations change. She retains the human audience of developer relations: education, community, and feedback still matter. Her curb-cut analogy explains the aim—interfaces and documentation that reduce friction for agents can also clear the path for developers.

Making agent fleets finish the work

Operating agent fleets has given her another set of reliability problems to investigate. In Software Factories Are Distributed Systems, an agent fixes a bug, passes tests and review, and closes its work item, yet the branch never merges. The failure separates an agent’s report of completion from the external result the task required. Jarmak distinguishes the work record, the procedure carrying it forward, the worker executing it, and the controls deciding what may run or publish. Recovery must identify surviving workers before starting replacements, and stale workers must lose permission to publish even if their processes remain alive.

After struggling to keep a Gas Town installation reliable, she began contributing heavily to Gas City in April 2026 and became an outside maintainer. The SDK supplies sessions, dispatch, messaging, and persistent work records, with roles expressed through configuration and prompts. Her contributions include session lifecycle, reconciliation, Beads integration, dispatch, and the API. She also uses the system to build and repair itself, so orchestration failures directly interrupt her own work. Adding Temporal to her software factory sharpened her distinction between preserving a record of work and preserving the execution procedure needed to carry it forward.

For human supervision, Jarmak built a Slack interface for her agent fleet. Project channels and assigned leads organize the work, while a coordinating agent aggregates issues across projects. Overnight activity produces a morning briefing of decisions that require her judgment. The interface gives her enough context to unblock progress without continuously watching every session—a concrete expression of her concern with making agent work inspectable and accountable through to its actual effect.