# Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

Stephanie Jarmak, Omni Analytics

[Talk page](https://ai.engineer/talks/A3FL_zTQKU0-dont-trust-your-ai-agents-test-them) · [Video](https://www.youtube.com/watch?v=A3FL_zTQKU0)

## Don't Trust Your AI Agents. Test Them Like a Scientist

Stephanie Jarmak follows agent evaluation from a misleading cost improvement to production measurement, showing why realistic workloads, full traces, enforceable gates, and trustworthy records answer different questions.

![Opening slide titled “Lies, Damned Lies, and Evals” identifies Stephanie Jarmak as AI Engineer at Omni, with the subtitle “How to trust your agents when the stakes are high.”](https://ai.engineer/api/talk-essay-media/cad47c370eb5cddfb23fd41180d41fd89eda38b8ea9e8d9f9abb2a3638b830c2)

Jarmak’s opening slide frames agent evaluation as a question of trust when the stakes are high. The recording identifies her as an AI Engineer at Omni at AI Engineer NYC on October 13, 2026.

### Twenty-eight percent cheaper—and still ignoring the tool

[0:12](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=12s)

After a prompt change, a coding-agent run cost 28% less. The agent still ignored the code-search tool the experiment was meant to test. That discrepancy opens Stephanie Jarmak's practical case for treating agent evaluations as experiments. An AI engineer at Omni Analytics at recording time, she previously worked on Amp and evaluation infrastructure at Sourcegraph, including [CodeScaleBench](https://www.sjarmak.ai/projects/codescalebench). Before that, she was an astronomer. The scientific habit she brings to agents is to question what a result actually establishes.

The Sourcegraph experiment concerned a code-search MCP tool. The intervention was a change to the agent's initial prompt, intended to encourage tool use; the run retained traces and produced several metrics. The question to inspect in the comparison is simple: did the intended behavior change alongside the cost? It did not. The 28% reduction was an observation about spending, while the agent continued to skip the tool.

The cheaper run could justify a separate investigation into cost variation. It could not establish that the code-search integration was working. Keeping those questions separate prevents an attractive secondary metric from deciding an experiment whose primary objective remains unmet.

### The harness can reverse the model ranking

[2:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=161s)

A second comparison asks what happens when the same two models run under different harnesses. In a third-party Terminal-Bench test on 63 tasks requiring no GPU, Claude Opus 5 solved 36 tasks under OpenHands and GPT-6 Astra solved 31. Under PI, Opus solved 19 and Astra solved 38, giving Astra a lead of about 30 percentage points. Each configuration had one scored run per task after infrastructure retries.

![Bar chart for the same 63 tasks compares Claude Opus 5 and GPT-6 Astra under OpenHands and PI; the ordering of the two models reverses between harnesses.](https://ai.engineer/api/talk-essay-media/71f61437e8b89f69102d1e93e5640e47e84cca07938b5d1797905da6419c0122)

On the same 63 tasks without a GPU, Claude Opus 5 leads under OpenHands and GPT-6 Astra leads under PI. The slide compares the configurations actually run, rather than assigning a universal ranking to the models.

The harness is the surrounding setup through which the model receives instructions, accesses tools, and carries out work. A benchmark result therefore describes a configuration, not a model in isolation. If a harness change can reverse the ordering, choosing a model from a leaderboard without testing the intended setup can select the wrong combination for the actual job.

This gives public benchmarks and product evaluations different roles:

- **Public benchmarks narrow the search.** They provide a standardized reference set and a view of the wider landscape, helping a team shortlist configurations worth investigating.
- **Internal evaluations support a decision.** They measure changes to a product or process using tasks and environments that reflect its own work. A standardized internal benchmark can then make successive configurations comparable.

The law-firm example makes the relevance test concrete. Terminal-Bench results might help the firm's developers choose a coding agent; they do not automatically tell the firm how an agent will perform its other work. Match the benchmark to the task, then test the configuration as close as possible to the environment where it will run.

### Choose workloads, repetitions, and interacting variables

[5:33](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=333s)

Once a shortlist exists, experimental design starts with the work the system must cover. Grouping tasks into families makes it harder for one convenient task type to stand in for the whole product. Jarmak built a broader coding benchmark because, at the time, the bug-fixing emphasis of SWE-bench did not cover the range of developer work she wanted to study. Understanding a codebase and fixing a bug are different tasks; a model can be effective at one without being equally effective at the other.

Three design choices deserve separate attention:

- **Workload coverage:** Select representative tasks and organize them into meaningful families. Tooling may need to work differently across those families.
- **Repeated runs:** Decide how many runs to perform because the same setup can produce different results from one attempt to the next. A single run cannot reveal that variation.
- **Interaction effects:** Test relevant combinations of changes. The model-and-harness comparison shows why the effect of one component can depend on which other component accompanies it.

These choices determine what a result can mean. A configuration that improves bug fixing may leave code understanding unchanged; a strong run may not repeat; a model that wins in one harness may lose in another. The experiment needs enough coverage to distinguish those possibilities before the result becomes a product decision.

### Trace the path from tool availability to task outcome

[7:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=425s)

Returning to the unused search tool, the next question is where the information path broke. Retain the traces, then begin with the context actually available to the agent. Was the tool available at all? If not, stop the investigation there and repair the setup. If it was available, inspect whether the agent called it and what values came back. Tool availability, tool invocation, and useful returned information are separate facts.

These checks answer a narrower question than a final score can. Which prerequisite failed before the information could help? Even after a successful call, one question remains. Did the agent use the returned information in a way that changed the task outcome?

Even a successful tool correction can leave the final result unchanged. Jarmak describes the scenario of an agent finally using a much better tool, yet failing to benefit from its information. That moves the investigation downstream: the integration now works far enough to return useful material, but the agent or configuration still does not turn it into a better result. Keep non-use and failed attempts in view rather than examining only successful calls.

Where did useful information stop helping?
This is a diagnostic path, not a claim that each stage succeeded. Availability and invocation determine which failure to inspect; a tool call alone does not establish an improved outcome.
- Was the tool available? — Inspect the context and tools the agent actually received.
- Repair the setup — A missing tool must be fixed before investigating its use.
- Was the tool called? — Availability does not establish invocation.
- Investigate non-use — Retain the attempt where the available tool was not used.
- What came back? — Inspect the values returned by the tool.
- Was the information used? — A better tool can return information the agent does not use.
- Did the task outcome change? — Check the end result separately from tool-level progress.
- Was the tool available? → Repair the setup (No: stop and fix availability)
- Was the tool available? → Was the tool called? (Yes: inspect invocation)
- Was the tool called? → Investigate non-use (No: inspect the skipped tool)
- Was the tool called? → What came back? (Yes: inspect returned values)
- What came back? → Was the information used? (Trace whether returned information was used)
- Was the information used? → Did the task outcome change? (Check for an end-task benefit)

### Returning 295 files is not the same as finding the right six

[8:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=515s)

The search-tool comparison also exposes a measurement choice. In Jarmak's example, **Claude** used `grep` and returned 295 files—the “entire kitchen sink” approach. All six files in the answer key were present. The other 289 files did not affect recall.

![Scoring audit shows 295 returned files, six reference files found, recall 1.0, precision 0.020339 and an F-beta score of 0.025295 with beta 0.5.](https://ai.engineer/api/talk-essay-media/c5bc0ac0f90f6efb3e323d92f526571341fe7d8fbf89aebdce43b7fd0dd16acb)

The example finds all six reference files among 295 returned files. The slide specifies recall 1.0, precision 0.020339 and an F-beta score of 0.025295 with beta 0.5; relevance beyond the reference set was not established.

Two retrieval measures answer different questions:

- **Recall:** What fraction of the reference files did the search retrieve? Here it found six out of six, for a recall of one.
- **Precision:** What fraction of the returned files matched the reference set? Here six out of 295 matched, or about 2%.

Jarmak describes wanting a balance of precision and recall. The scorer used an F-beta score with beta 0.5, which gives precision more weight than recall.

The business objective decides which tradeoff matters. A search that returns every possibly relevant item can be appropriate under a recall-focused objective and poor under a precision-sensitive one. Before adopting a single benchmark score, ask what its validator measures and why that criterion was chosen. A higher score answers the validator's question; it only answers yours if those questions align.

### A warning must be able to stop work

[10:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=605s)

Evaluation overlaps with operational safety when a detected failure is supposed to block further action. In Jarmak's experimental software factory, work requests passed through approval queues and required checks, with regular audits. The agent could notice that something was wrong. The missing mechanism was a way to enforce a stop.

Separate the responsibilities:

- **Detection:** An agent or another checking system flags a problem.
- **Enforcement:** Code implements the gate that prevents work from continuing when required conditions are not met.
- **Escalation:** A defined path brings the blocked decision to a human when necessary.

A checker that produces the right warning has only performed the first job. Without enforcement, the workflow can continue through a failure it has already recognized.

Sending every alarm to a person creates a second engineering problem: review capacity. In a separate LLM-as-a-judge system Jarmak cites, a 1% false-alarm rate would produce 10,000 human reviews at the scale under discussion. The percentage sounds small; the queue does not. The design must account for whether domain experts can handle that load and how consequential the flagged cases are.

Alarm quality and downstream operations therefore belong in the same evaluation. A threshold does not become useful merely because it catches problems: its false alarms consume scarce attention, while the consequences of missed problems affect how much risk the organization can accept. The bottleneck may move from automated execution to expert review.

### The retained trace can also be wrong

[12:09](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=729s)

Retaining traces is necessary for diagnosis, but the record itself can be unreliable. Jarmak describes a security experiment in which researchers seeded issues into an agent setup and used a hash check intended to detect tampering. That check passed without noticing the injected issues. An independent comparison of records—including traffic history and model changes—detected all the seeded issues in the described experiment.

![Diagram separates the model, harness, tool and destination from independent traffic and state records; the own-record hash check passed 28 of 28 cases while independent comparison detected 28 of 28 introduced changes.](https://ai.engineer/api/talk-essay-media/95704d71275ca773679446380791aec2eda7c05b522ff6fbf71d9f075bfb258c)

The system’s own record passed its hash check in all 28 cases, while an independent record comparison detected all 28 deliberately introduced changes. The slide limits that result to the inspected record scope; it does not establish execution or destination state.

The comparison asks what each checking method can establish. Here, a passing hash check did not establish that the security-relevant history was sound, while the independent record comparison exposed discrepancies. The talk does not detail the hash coverage or the injection mechanisms, so this is a warning about relying on that particular check, not a general claim that hashes cannot protect records.

Observability needs a trust model of its own. Decide whether the retained record is reliable enough for the system's security needs before using it to explain behavior or certify a result. More traces help only if the relevant events are recorded in a form the investigation can trust.

### Production tests whether the workload proxy was good

[13:21](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=801s)

At Omni, the workload design begins with internal production data: how users actually interact with agents. Those interactions are classified into task types, including Q&A ranging from easy to hard questions, dashboard building, and app building. These categories give the evaluation harness a way to compare changes against recognizable kinds of user work rather than an undifferentiated average.

The purpose is to identify where a system change adds value and where failures appear downstream. Even a carefully constructed internal suite remains a proxy for actual workloads. “Production is truth” is Jarmak's shorthand for the next obligation: after deployment, keep measuring the system in its real environment.

Shadow operation offers a way to gather information when directly changing live behavior would carry a larger blast radius. The choice is risk-dependent: obtain observations from realistic conditions while limiting the effects of an insufficiently trusted change. Offline results guide the deployment decision; measurements in the deployed environment test whether those results carried over.

### Make the next action and the escalation decision explicit

[15:25](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=925s)

The final decision is how much risk to accept before a human must intervene. In her software factory, Jarmak allows some autonomy for lower-risk decisions while making the proposed next action and its scope explicit. When human input is needed, the system identifies why a person should weigh in. That makes escalation a bounded decision about an action, rather than an unspecified request to inspect everything the agent has done.

The distinction to encode is when autonomous deployment is acceptable and when work must stop and escalate. It connects back to the earlier gates: risk criteria decide when human involvement is required, and enforcement must make that requirement real. More autonomy is a choice about which actions the system may take under which conditions.

Jarmak closes by offering reusable skills for evaluation design, harness traceability, and reliability procedures, alongside a roughly 300-page archive that agents can consult. These materials supply procedures to bring into a system. The decisions they support remain concrete: what work is representative, what result matters, what record can be trusted, and when the system must hand the next action to a human.

## Key takeaways

- Define the decision before interpreting the metrics. A run that became 28% cheaper still failed the experiment's objective when the agent continued to ignore the intended tool.
- Evaluate the model, harness, and workload together. Public benchmarks help shortlist candidates; representative internal tasks and repeated runs support decisions about the intended configuration.
- Trace availability, invocation, returned information, and final outcome separately. A better tool can be called successfully without improving the task result.
- Measure the retrieval behavior you need. Finding all six reference files among 295 returned files illustrates why recall alone can miss the value of precision.
- Separate detecting a problem from enforcing a stop, and budget human review capacity. A low false-alarm percentage can still create an unmanageable review queue.
- Check whether the retained record deserves trust, then continue measuring in production. Risk and operating scope should determine when an agent may act autonomously and when it must escalate.

## Resources

- [CodeScaleBench](https://www.sjarmak.ai/projects/codescalebench) — Jarmak built this public benchmark to study agents working across complex codebases and to cover developer tasks beyond bug fixing.
- [Software factories are distributed systems](https://www.sjarmak.ai/writing/software-factories-are-distributed-systems) — Jarmak describes a fix that passed automated review and was marked complete while its commits remained on an unmerged branch.
- [Software Factory Reliability](https://github.com/sjarmak/software-factory-reliability) — Companion repository to Jarmak’s essay, with executable reliability patterns and fault drills for AI software factories.

## Related talks

- [Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust](https://ai.engineer/talks/T3SS931wU0I-agentic-search-vs-vector-search-coding-agents) — Pairs Jarmak’s coding-agent retrieval example with a comparison of agentic search and vector search.
- [How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase](https://ai.engineer/talks/_4Ph2dhmUFI-know-when-ai-agent-work-unsupervised-evan) — Shares Jarmak’s closing question about when an agent can safely work unsupervised.
- [Fixing the PR Bottleneck — Matt Pocock, AIHero](https://ai.engineer/talks/LlgiOCmFG_w-fixing-pr-bottleneck-matt-pocock-aihero) — Connects Jarmak’s discussion of human review capacity to the pull-request bottleneck.

## Slides

### Lies, Damned Lies, and Evals

Stephanie Jarmak introduces a research scientist’s approach to evaluating AI agents: question their behavior rather than assume they deserve trust, especially when the stakes are high.

[0:12](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=12s) [Title slide for Stephanie Jarmak’s talk, subtitled How to trust your agents when the stakes are high, identifying her as an AI Engineer at Omni.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3209f5a0b518ba1fdf62700a12004cf3b7399032b973f24296af419cdad83c3.webp)

Lies, Damned Lies,
and Evals

AIE NYC · OCTOBER 13, 2026

How to trust your agents
when the stakes are high

Stephanie Jarmak
AI Engineer at Omni

sjarmak.ai

01

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Lower cost. The retrieval failure remained.

Observed cost fell 28% on the affected task family, but search still missed relevant evidence and the agent trusted the empty result. The small score change does not establish equivalent quality.

[1:09](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=69s) [Slide reporting 28% lower observed cost, a primary evaluation score change of +0.0048 with t = 0.27, and 3.9% longer elapsed time. A diagram connects search missing relevant evidence to the agent trusting the empty result.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b907dde3f4a88f9817664f12ca771c734d63b773b2b150da4c5f58e3878bca5.webp)

Lower cost. The retrieval failure remained.

OBSERVED COST
−28%
on the affected task family

PRIMARY EVALUATION SCORE
Δ +0.0048
t = 0.27

ELAPSED TIME
+3.9%

Search missed relevant evidence → Agent trusted the empty result

The retrieval failure remained. Little evidence of score improvement does not establish equivalent quality.
Jarmak · Engineering Reliable Coding Agents, ch. 18

02

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### The ranking changes with the harness

On the same 63 non-GPU tasks, Claude Opus 5 leads in OpenHands (57.1% vs 49.2%); GPT-6 Astra leads in PI (60.3% vs 30.2%). With one scored run per task/configuration, compare the setup you will actually use.

[3:06](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=186s) [Slide comparing two models on 63 tasks: OpenHands bars show 57.1% and 49.2%, while PI bars show 30.2% and 60.3%. The takeaway is to compare the configuration you will actually run.](https://assets.ai.engineer/sites/aiecode2025/sha256/04471c1c5f2b510550c34bf6f3101fad0098ad57f903a298d130f1d3a0d3d1c2.webp)

The ranking changes with the harness

Same 63 tasks requiring no GPU. Different harness.

Y-axis ticks: 0%, 25%, 50%, 75%, 100%

OpenHands
Claude Opus 5: 36/63 (57.1%)
GPT-6 Astra: 31/63 (49.2%)

PI
Claude Opus 5: 19/63 (30.2%)
GPT-6 Astra: 38/63 (60.3%)

Legend: Claude Opus 5; GPT-6 Astra

Compare the configuration you will actually run.

One scored run per task / configuration, after infrastructure retries.

Li et al. · Finding the Right Fit
Hardy et al. · Agent Evaluation Reliability

03

OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Public benchmarks and internal evaluation

Use benchmarks to shortlist candidates, then compare them on target work and retest changes against a baseline. Public and internal describe visibility, not purpose; benchmark results depend on the task mix and protocol.

[4:00](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=240s) [Workflow from reference and reported protocol to candidate shortlist, comparison on target work, and product or process change, with a retesting loop. Notes distinguish benchmark visibility from purpose.](https://assets.ai.engineer/sites/aiecode2025/sha256/b851924e84eaac8bfd37ae18813388174e87b0cb0031b95aae0f0ee5f46aeed7.webp)

Public benchmarks and internal evaluation

Benchmark: a standardized reference for comparison

LANDSCAPE AND SHORTLIST
Reference + reported protocol → Candidate shortlist

PRODUCT AND PROCESS IMPROVEMENT
Compare on target work → Product/process change

Results depend on the task mix and evaluation protocol.

Diagnose, change and retest.
Communicate value against a baseline.

Internal benchmarks exist. Public benchmarks can be used internally.

Public and internal describe visibility, not exclusive purposes. The decision determines what evidence we need.

Jarmak · Engineering Reliable Coding Agents, chs. 2–3

04

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Design the experiment around the decision

Task families test coverage; repeated runs test variation; model–harness pairings test interaction effects. Track quality, reliability and people’s time together, and divide total operating cost by independently accepted outcomes.

[5:31](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=331s) [Three columns show task-family coverage, three repeats of the same task, and a model-by-harness configuration matrix. A formula divides total operating cost by independently accepted outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/be8cdcfa47f203447e7caa7404d065e80c2e61252b34b651a531874246ae1f5a.webp)

Design the experiment around the decision

Report quality, reliability, people’s time and total cost together.

TASKS
Workload coverage
Family A | task i
Family B | task j
Family C | task k

REPEATS
Variation between runs
Same task | run 1
Same task | run 2
Same task | run 3

CONFIGURATIONS
How results depend on the pairing
          Harness A | Harness B
Model A | A × A     | A × B
Model B | B × A     | B × B

Total operating cost ÷ Independently accepted outcomes

Repeated runs are not independent tasks. A better relative score does not establish that either system is reliable enough.

Jarmak · Engineering Reliable Coding Agents, chs. 1–2
Hardy et al. · Agent Evaluation Reliability

05

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Trace where evidence becomes a conclusion

The trace separates tool availability, calls, returned results and interpretation. Two proposed tests make search limits explicit or supply known relevant evidence with the same agent and settings; neither has demonstrated a repair.

[7:03](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=423s) [Evidence pipeline from Available through Called and Returned to Interpreted, with search documentation and proposed search-limits and evidence tests. The slide notes that these tests have not demonstrated a repair.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d53b375f335a7edffd77d63ff8844fb413ef59f7758e6464fac75a19dd9c029.webp)

Trace where evidence becomes a conclusion

Available → Called → Returned → Interpreted

RECORD THE SEARCH
Where and how you searched
Gaps or incomplete results
What you couldn’t check

SEARCH-LIMITS TEST
Same empty result,
make the search limits explicit

EVIDENCE TEST
Same agent and settings,
supply known relevant evidence

A trace tells us where to investigate. These proposed tests have not yet demonstrated a repair.

Jarmak · Engineering Reliable Coding Agents, chs. 2 & 18

06

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Better evidence can still go unused

Substitute trusted evidence, then bypass the component that uses it to separate evidence quality from evidence use. Keep non-use and failed attempts in deployment comparisons; the simulation leaves production effects untested.

[7:49](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=469s) [Evidence source flows through a component using the evidence to an outcome. Controls substitute trusted information or bypass the component; deployment comparisons retain non-use and failed attempts. Production effects remain untested.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d4e0f80d07b6ca675a5a0a21f63d081109562b83d159e94f05f031afe171eed.webp)

Better evidence can still go unused

Test the evidence and the agent’s use of it separately.

CONTROL A
Evidence source
Substitute trusted information

CONTROL B
Component using the evidence
Bypass the component using the evidence

Outcome

Deployment comparison
Keep non-use and failed attempts in the result.

Simulation result; production effects remain untested. Tool use alone does not identify the tool’s causal effect.

Alzahrani · Verify Claims, Not Scores
Jarmak · Engineering Reliable Coding Agents, ch. 2

07

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Audit the scoring rule and the answer key

All six reference files appear among 295 results: recall is 1.0, but precision is about 2%. Proposed checks challenge the scorer and answer key, including a valid answer missing from the key. One run does not establish prevalence.

[8:38](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=518s) [Scoring audit of 295 returned files, including six reference files and 289 outside the reference set. Recall is 1.0, precision 0.020339 and F-beta score 0.025295 with beta 0.5. Proposed challenges test the scoring rule and answer key; the record includes scoring rule, reference version and run identity.](https://assets.ai.engineer/sites/aiecode2025/sha256/93897cf3bc48958486abb3048e59c897bd9d8fa299763051b006315aa2b5381a.webp)

Audit the scoring rule and the answer key

RECORDED OUTPUT
295
files returned

6 reference files found
289 outside the reference set

SCORING RULE
RECALL
1.0

PRECISION
0.020339
about 2.0%

ACTUAL F-BETA SCORE
0.025295
beta = 0.5

PROPOSED CHALLENGES
Known-good answer
Empty answer
Return-everything answer
Valid alternative absent from the key

AUDITABLE RESULT RECORD
Scoring rule
Reference version
Run identity

One output, measured against one reference set. The score was precision-sensitive; intent and prevalence are not established.

Jarmak · CodeProbe trial e5d7a4e7 / commit 2495927

08

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

[9:50](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=590s) [Scoring audit: 295 files returned, six reference files found, and 289 outside the reference set. Recall is 1.0, precision 0.020339, and F-beta 0.025295 with beta 0.5. Suggested challenges test the scoring rule and answer key.](https://assets.ai.engineer/sites/aiecode2025/sha256/b15abfd05ecf0de4832092f9e8acb5cc2bee5d37f6589f2b19b4bce9895c7b9a.webp)

Audit the scoring rule and the answer key

RECORDED OUTPUT
295
files returned

6 reference files found
289 outside the reference set

SCORING RULE
RECALL
1.0

PRECISION
0.020339
about 2.0%

ACTUAL F-BETA SCORE
0.025295
beta = 0.5

PROPOSED CHALLENGES
Known-good answer
Empty answer
Return-everything answer
Valid alternative absent from the key

AUDITABLE RESULT RECORD
Scoring rule
Reference version
Run identity

One output, measured against one reference set. The score was precision-sensitive; intent and prevalence are not established.

Jarmak · CodeProbe trial e5d7a4e7 / commit 2495927

08

Interface text:
Engineering the future of AI

### Test whether the gate changes execution

Jarmak proposes breaking approval controls to test whether they block execution, tracking authorized completion, safe stops and repair work separately. Her audit did not establish unauthorized production writes.

[9:53](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=593s) [Flow from request through approval and required checks to execution and observed effect. Proposed tests reject or cancel, remove a required component, or change a proposal after approval. Outcomes distinguish authorized completion, safe stops, and people's repair work; unauthorized production writes were not established.](https://assets.ai.engineer/sites/aiecode2025/sha256/62ca9c9e76bcf5f7f70b5e9eadd5709584991e30ffd44f162f0c562c916dcab8.webp)

Test whether the gate changes execution

My approval-queue audit found paths that allowed execution when required components were missing.

PROPOSED ENFORCEMENT TEST
Request → Approval + required checks → Execution → Observed effect

DELIBERATELY BREAK A CONTROL
Reject or cancel
Remove a required component
Change the proposal after approval

MEASURE OUTCOMES SEPARATELY
Authorized completion
Safe stop
People’s repair work

Deliberately break a control next. Also measure false blocks on legitimate work; this audit did not establish unauthorized production writes.

Jarmak · Engineering Reliable Coding Agents, chs. 4 & 17

09

Engineering the future of AI

### Choose what runs automatically, then measure it

Track missed errors, unnecessary reviews, and the share handled automatically. Review capacity matters: a worked example—not measured traffic—shows a 1% false-alarm rate across 1,000,000 legitimate actions creating 10,000 reviews.

[11:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=671s) [Three questions address missed consequential errors, good work sent to review, and the share needing no human review. A worked example shows a 1% false-alarm rate across 1,000,000 legitimate actions producing 10,000 reviews, explicitly not measured traffic.](https://assets.ai.engineer/sites/aiecode2025/sha256/b241b1ade0fc08a947f6074ade65932a242cd1c5465a741eb292d30da48a9193.webp)

Choose what runs automatically, then measure it

Retrieve existing rules; clarify unresolved requirements; encode where possible.

DEFECT DETECTION
What consequential errors get through?

REVIEW LOAD
What good work goes to review?

SHARE HANDLED AUTOMATICALLY
What fraction needs no human review?

ONE OPERATING CONSEQUENCE
1% × 1,000,000 = 10,000
false-alarm rate × legitimate actions = reviews

Worked example, not measured traffic. Samples enriched for suspected errors do not estimate production error rates.

Liu et al. · Auditing and Repairing LLM-as-Judge Failures
Sehanobish et al. · From Benchmarks to Production (FLINT)

10

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Some evidence needs another observer

With 28 deliberately introduced changes, the own-record hash check passed 28/28 while independent comparison detected 28/28. Broader detection is untested; model traffic does not prove execution or destination state.

[12:06](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=726s) [Model-to-destination flow compares independent model-traffic records, self-written harness records, and resulting-state records. For 28 deliberately introduced changes, the own-record hash check passed 28 of 28 while independent comparison detected 28 of 28. Detection beyond this scope is untested; model traffic does not prove execution or destination state.](https://assets.ai.engineer/sites/aiecode2025/sha256/22b46778dd37a28d49959e2606aff23f765ee08e2f4008de84a6cd9fc55d5225.webp)

Some evidence needs another observer

For security claims, test missed detections, false alarms and blind spots.

Model → Harness → Tool → Destination

Independent record
of model traffic

Self-written
harness record

Record of the
resulting state

Own record passed the hash check
28/28

Independent record comparison detected
28/28

28 deliberately introduced changes. Detection beyond this scope is untested; model traffic does not prove execution or destination state.

Dai et al. · Hearsay

11

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

[13:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=785s) [Model-to-destination diagram compares three record sources. Own-record hash checks passed for 28 of 28 deliberately introduced changes, while independent comparison detected 28 of 28. The slide cautions that broader detection is untested and model traffic does not prove execution or destination state.](https://assets.ai.engineer/sites/aiecode2025/sha256/c611b2c36955a9386d153285f43a90442706928490444caf8b204c20ea6591be.webp)

Some evidence needs another observer

For security claims, test missed detections, false alarms and blind spots.

Model → Harness → Tool → Destination

Independent record
of model traffic

Self-written
harness record

Record of the
resulting state

Own record passed the hash check
28/28

Independent record comparison detected
28/28

28 deliberately introduced changes. Detection beyond this scope is untested; model traffic does not prove execution or destination state.

Dai et al. · Hearsay

11

Interface text:
AI Engineer

### A retrieval upgrade: the experiment I would run

The proposed retrieval test moves from matched work to independent challenge, then parallel runs with isolated state-change tests. Deployment monitoring asks whether scores predict real outcomes, with rollback triggers set in advance.

[13:20](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=800s) [Proposed workflow moves from matched target-work comparison through independent challenge to running alongside current work and isolated state-change tests. Post-deployment checks cover completion, repair, latency, cost, audits, trace-to-outcome comparisons, workload changes, ownership and regression tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/22d8c65256c99ae7fb20482c29a66744d2718fec76887857f9cc7c69e9916e61.webp)

A retrieval upgrade:
the experiment I would run

PROPOSED EVALUATION WORKFLOW
Matched target-work comparison
↓
Independent challenge
↓
Run alongside current work.
Test state changes in isolation.

Can an independent challenger find the decisive evidence?

AFTER DEPLOYMENT
Completion, repair, latency, cost
Audit accepted work and failures
Check traces against real outcomes
Watch workload and version changes
Owner, intervention, regression tests

Do scores predict real outcomes?

Set restriction or rollback triggers before deployment. The monitoring loop and independent-review tests are proposed.

Jarmak, chs. 16–17 · Google SRE: monitoring and alerting

12

Interface text:
Engineering the future of AI

### Make the next autonomy decision explicit

Specify action, scope and human check removed. Evidence and validated controls guide expansion within the tested boundary, continued review, or stopping and escalating. A human owns acceptable risk and conflicting business values.

[15:18](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=918s) [Proposed autonomy decision combines additional action, operating scope and a removed human check. Evidence, validated evaluators, execution controls and a human risk owner inform three choices: expand within the tested boundary, retain review, or stop and escalate.](https://assets.ai.engineer/sites/aiecode2025/sha256/118eb91eb25de8923f64e3be8c4ceaa76122c7c44a5a73c1adfef703741fbaad.webp)

Make the next autonomy decision explicit

PROPOSED AUTONOMY DECISION
Additional action + operating scope + human check removed

EVIDENCE BAR
Representative repeated outcomes
Tolerable failures and total burden

VALIDATED EVALUATOR + EXECUTION CONTROLS
Accept good work, detect defects
and enforce the operating boundary

Human owner: acceptable risk and conflicting business values

Expand within
the tested boundary

Retain review

Stop and escalate

Stop for missing evidence, a failed gate or changed scope.
An action-specific standard to test. Procedures support the work; they do not establish reliability or supply missing evidence.
Jarmak · Engineering Reliable Coding Agents, closing

13

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Evaluation procedures and references

Jarmak points to reusable skills for evaluation design, harness traceability and reliability procedures in Coding Agent Workflows, alongside a book and paper on evaluating the system around the model.

[16:29](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=989s) [References slide with a Coding Agent Workflows QR code and public links. Three procedures are listed: evaluation design, harness traceability and reliability procedures. A book and paper provide further material on reliable coding agents and evaluating the system around the model.](https://assets.ai.engineer/sites/aiecode2025/sha256/00d97faeb1f5a357afb89ea0449c32ebe6686682aa52eb851b54bb64c9bb7d52.webp)

Evaluation procedures and references

Coding Agent Workflows

github.com/sjarmak/coding-agent-workflows

THREE PROCEDURES
Evaluation design
Harness traceability
Reliability procedures

BOOK AND PAPER
Engineering Reliable Coding Agents
Evaluating and Operating the System Around the Model
sjarmak.ai/books/engineering-reliable-coding-agents
arxiv.org/abs/2608.13867

14

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

[16:34](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=994s) [Reference slide with a Coding Agent Workflows QR code, three procedures—evaluation design, harness traceability and reliability procedures—and links to a book and paper on reliable coding agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e8c4c9ed9ecca3d024549fe24464ccec42b6a0f8fa9d381fa7e32de5854e475.webp)

Evaluation procedures and references

Coding Agent Workflows

github.com/sjarmak/coding-agent-workflows

THREE PROCEDURES
Evaluation design
Harness traceability
Reliability procedures

BOOK AND PAPER
Engineering Reliable Coding Agents
Evaluating and Operating the System Around the Model
sjarmak.ai/books/engineering-reliable-coding-agents
arxiv.org/abs/2608.13867

14

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

## Full transcript

[0:12](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=12s) My name is Stephanie Jarmak. I am an AI engineer, um, as of a couple of weeks ago at Omni Analytics, um, where we have a, a platform that, um, has, like, a governed semantic layer to have a shared understanding of business data using our beloved agent, Blobby. Um, before that, I spent a year at Sourcegraph where, um, I was a solutions engineer for Amp, a coding agent, and then I moved into some sort of amorphous type role where I was largely doing, um, their

[0:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=41s) evaluation, uh, platform. I built a very large public benchmark called CodeScaleBench to understand how agents are working across very complex code bases. Um, and then prior to that, I was an astronomer. Uh, so I come from this from a research scientist sort of point of view, where I relentlessly question everything, hence the title of Lies, Damn Lies, and Evals. Um, it says how to trust your agents, but I think largely this talk is mainly about how to not trust your agents.

[1:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=71s) So, uh, here's one example of, uh, a run I had done, um, back at Sourcegraph when I was testing out their code search, um, MCP tool. Um, and this was just to show all of the different sort of numbers that you can have after you run a certain evaluation. W- The-- What I was trying to do was figure out, "Hey, why isn't this coding agent actually using the tool that I'm trying to get it to use?" Um, and so, you know, it ran, I have all the traces and everything, and then I, you know, looked at some of the primary sort of metrics that you're

[1:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=101s) looking at when you evaluate the overall, um, effectiveness of a tool. And it showed we-- Like, I cha-- Basically, what I had done was I changed the initial prompt given, uh, to the agent, and then I ran it, uh, 'cause I wanted it to use the tool. Um, and what I saw was, okay, uh, it was twenty-eight percent cheaper. That's, that's interesting. Um, but, you know, that shouldn't really change any of my decisions around what I'm actually tryin-- what I'm interested in measuring and understanding in, in changes to, um, the actual harness and setup,

[2:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=131s) uh, because it still didn't use the tool, right? So you need to be able to understand, like, what it is you're actually sort of setting this up. The, the twenty-eight percent, um, is maybe an interesting avenue to go explore. Okay, we have some level of, like, experimental space to be figuring out what's the variance and what's going on with the, with the cost measurements. Um, but we still weren't, like, seeing it actually use the tool, um, that it was supposed to. And, uh, basically, we had this whole, like, um, indication that

[2:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=161s) it was receiving certain information and then just trusting that that information wasn't there, and that was purely because it didn't use that tool. And I also wanna talk a bit about the impact of the harness itself. Um, so here's an example, and this was from a third-party source where they had sixty-three or so tasks, um, on Terminal-Bench 4, which is a pretty popular, widely used, um, benchmark for coding agents. And they, you know, you ha-- Like, let's say you're trying to pick which model you're going

[3:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=191s) to, um, put into your harness. Um, so here's a comparison they did, they did with, um, Opus 5 then and Astra, and they saw that, you know, with OpenHands, which is one of the open harnesses, um, Opus 5 beat it out. But then when they swapped the-- just the harness, um, itself, then they saw that Astra did a, a much better job, right? So it was, like, thirty-plus percent sort of different-- difference in the, um, effectiveness of these configurations. So the takeaway here is you want to be

[3:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=221s) actually evaluating the closest, like, configuration that you can that's going to reflect the, the work and the environment that you're going to be measuring because there are so many different variables involved and, you know, you might be con-- uh, you know, might see all these, um, benchmarks and whatnot and see the, the fancy results, but it might not do what you expect it to do within your actual environment. Um, which leads a bit into... I wanna kind of disambiguate and expand a little bit more on how to interpret, um, like, public benchmarks

[4:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=251s) versus what is an evaluation. So an evaluation is you're making measurements to, um, make a decision about your product or process and those changes. And the, the public benchmarks as they exist, um, are really primarily helpful for you to have a broader understanding of, like, the larger landscape and get a short list of what you might be exploring within your own, um, organization or your own, um, sort of tooling. Um, but you don't wanna necessarily over-index too much on what some of those results are because of the spiky

[4:41](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=281s) nature of all of the training of these different models, um, and things like that. So, you know, if you are a law firm, um, you don't wanna necessarily be looking at, like, the Terminal Bench, uh, results unless, of course, you are looking at incorporating coding agents w-with your devs, right? So you wanna make sure that the configuration's reflecting, um, your actual environments. Um, and then the-- You can also, like, think of benchmarks really as just a, um, standardized sort of reference set for

[5:11](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=311s) comparison. So you have these public benchmarks, but then you can also set up benchmarks within your organization to use as, like, setting them up as reflective of actual, um, workloads as possible and just have this sort of iterative flywheel process where you are, um, increasingly sort of hill climbing to, to optimize for your, for your configurations.

[5:33](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=333s) And, uh, you want to approach this, uh, to the best of your ability as a, as a scientist, kind of. Um, when-- Like, once you sort of shortlist what your con-- uh, various configurations might be, you wanna look at what are the different variables we're gonna think about changing, um, so that we're actually going to be, like, the, the results will reflect what our system is, right? So you can think of workload coverage So you wanna pick tasks that are gonna actually be reflective, and you could cluster them into different families and things like that. Like when I was making the coding benchmark,

[6:04](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=364s) um, didn't wanna just... Uh, the reason I built out a whole different one was because at the time it was largely like SWE-bench was just bug, uh, fixing. But there's a whole other different swath of tasks that are relevant to developers across the software development life cycle. And the way that you integrate different tooling into those different, like, workloads is gonna potentially be different. So, like, a model might be more effective at code understanding than it is at actually fixing a bug, for example. And then you wanna try to see, okay, how many, like, different runs do I wanna

[6:34](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=394s) do? Um, because as we all know, these systems are non-deterministic, and so you might see some level of variation, um, across run to run. Um, and then as I had shown before with, like, that different configuration of the model and the harness, you wanna think about how is-- how are the different sort of, um, combinations of potential changes going to have these, um, interaction type effects, right? And h-how is that going to potentially be, um, influencing downstream effects.

[7:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=425s) And you also want to be able to gather as much observability data as possible. So, you know, retain all of your traces and everything. Um, so this sort of goes back to one of my-- the first examples when I was trying to figure out, like, what is happening, why, why is it not, um, calling this tool? So first, you know, you want to sort of break it down into what is the sort of entire, uh, context available to the, to the agent. So it... Was that tool even available, right? 'Cause if it wasn't, then you, okay, you stop there,

[7:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=455s) and you f-- you fix that, um, particular, you know, scenario. And then you see was it called, um, et cetera, or like, what, what were the values returned? So you, like, basically keep going along the, the pipeline of available, um, information. Um, and this one, um, the... Like, so as you're going along that pipeline, let's say, uh, I do this improvement, right, for this, uh, tool. It finally uses the much better tool. Um, but then the

[8:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=485s) actual, like, outcome that you care about, the end of the task, like, that still doesn't change, right? So there are scenarios too where you can have this beautiful tool that you're, like, really optimizing for, um, but then the agent, the configuration or, or whatnot, for whatever reason, which then is your job to go in and, and figure out and fix, um, it's not leveraging, um, that information. So you wanna keep all of these different, um, things in mind, like the non-use and the failed attempts, um, and everything, um, when you're trying to basically be a detective and

[8:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=515s) understanding what is actually going on, um, in your system. And, uh, you also want to, uh, really control what the measurement is that you're making. Um, so in this example, um, when I was doing the, uh, tests of, um, the comparisons of different sort of, uh, search tools, right? I was interested in optimizing for, like, a better, basically like an F1 score balancing for precision in addition to

[9:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=545s) recall. Um, so in this example, it was with, like, grep, right? So Claude or whatever, going out and basically pulling in the entire kitchen sink. Um, so in this example, Claude went out and got, like, two hundred and ninety-five files returned when it only really needed six. So, you know, if I was somebody who in this, uh, scenario was like, "Okay, all I really care about is that it pulls in any possibly relevant piece of information," then maybe a recall of one is sufficient. But in my scenario, I was interested in optimizing for a completely

[9:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=575s) different, um, type of value. Um, so this is relevant too for when you're looking at those, like, single number scores in public benchmarks. Like, what was the actual validator measuring, and why did they consider that to be important? Um, 'cause what, what you might want is going to be dependent entirely, like, on your business needs. And, um, there's, like, a interesting sort of overlap that kind of happens when you're thinking about evals, um, and then sort of quality gates and, um, reviews and

[10:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=605s) things like that. So I just sort of wanted to touch on that a little bit here, um, in terms of, like, the trust and the safety. Um, so I have, like, a software factory that I've been experimenting with having, um, these different approval, uh, queues essentially of, I want to automate as much as possible with, uh, for the work to get done, but I also do not trust these systems, like, at all. Um, so we'll have these, uh, regular audits. Um, so, uh, as an example, um, basically, I will have, like, a re--

[10:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=635s) put through the request for some work to get done, um, and then it, it, um, has approval and required checks. But, uh, basically what was happening was the agent would notice things were going wrong, but then there was no mechanism in place for it to actually, like, enforce, uh, a stopping situation. So you need to sort of decouple what's happening with, um, an agent or some other system like flagging something versus, um, actually

[11:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=665s) enforcing that, um, like gate within the code itself, and then how that is, uh, eventually potentially escalated to a human. Um, 'cause this sort of goes into-- you need to also think about balancing that human review load, right? So if you have for-- and this, this comes from, um, basically auditing and repairing LLM-as-a-judge, um, failures that were happening, um, in a separate system. Um, but they were seeing like a one percent, uh, false alarm

[11:35](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=695s) rate, and if all of those like, um, basically the alarms that are raised go to a human, that would result in ten thousand reviews. And like, does your organization actually have the capacity for domain experts to be going and doing that? Um, so it's, you know, you need to think about where that bottleneck might be and how you, um, like, change what the sort of alarm rates Um, tend to be, and, you know, what-- how consequential are those, are those, um, alarms going to impact

[12:05](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=725s) the downstream effects?

[12:09](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=729s) And, uh, we wanna also consider that, um, you know, the actual agent trace itself can sometimes not be, um, reliable or, like, trustworthy. So in this particular example, um, researchers were doing, um, some checks to basically seed in, um, different sort of, like, security issues, um, that were provided to the agent. And, um, they had, like, a hash check in place to check for, like, any sort of tampering with that

[12:38](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=758s) system. Um, and with, uh, using that setup, basically, uh, everything passed. It didn't, it didn't notice, um, any of those, um, injected security issues. So-- But when they had an independent, um, sort of, uh, record comparison, um, take a look at basically all of the traffic history and the model, um, uh, changes and things like that, then they were able to detect everything. So you wanna make sure that, like, depending on the security needs of your system and how you're actually

[13:08](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=788s) measuring and, and observing, um, what's going on, that you have sufficient sort of, um, trust in, uh, the, the actual record that you're retaining of what happened.

[13:21](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=801s) And so this is basically just, like, an overview of, um, like, going back to that, um, original, um, example of, you know, how we're trying to think about, um, evaluating realistic sort of work, um, associated with your particular, um, task loads. So you want to identify, like, a target work comparison. So, um, I'm doing a lot of this right now at, um, Omni and working on their evaluation harness. So we're, um, looking at our

[13:51](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=831s) internal production data and seeing how, uh, our users are interacting with the agents and sort of, uh, having a classification of the different types of, um, things that can be done. So in our example, it's, um, Q&A, which may be easy questions to harder questions, or it could be dashboard building and app building and things like that. And so there's all these different sort of ways of classifying the actual tests. Um, and then you want to, um, be, like, identifying, um,

[14:22](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=862s) are these changes actually-- like, that we're, um, putting into the system actually, uh, adding value or where are the failures in downstream? Um, and we're doing all of this, right? And trying to come up with representative proxies of actual, uh, workloads, but ultimately, like, production is truth. So after you deploy some of these changes, then you need to also have measurement systems in place to get all of that relevant data in the actual environment that, that your system is being deployed in. And you can do

[14:52](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=892s) this in shadow and whatnot as well to, to gain some of that information if, if some of it would be kind of, um, less trustworthy to, to change, and you have a higher blast radius, uh, radius of, of effects and things like that. Um, but ultimately, you wanna be, like, essentially optimizing for as much observability, um, and realistic environments for where your system is going to be deployed for you to be able to trust some of these, um, values. So you want

[15:22](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=922s) to, um,

[15:25](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=925s) identify what, what level of, um, sort of risk you are going to be, um, willing to accept. Um, so the way I have it sort of set up within my own, um, systems, um, is I-- within my software factory, like, as an example, um, for, like, giving the agent some level of, um, autonomy and making some of the lower risk, um, decisions, is, um,

[15:55](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=955s) uh, basically, they will identify what the next, um, action would, like, be, like, why it's needed from a human to, um, be able to weigh in on that, what the actual operating scope is. Um, and then, you know, what, um, like, potential, um, suggested sort of human check removed, like, given the available, um, sort of spaces. But, um, you want to be able to come up with those different sort of boundaries of when is it

[16:25](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=985s) safe to autonomously, autonomously deploy versus, um, stop and escalate to the human. Um, so I have a GitHub repo with a bunch of the different, um, skills, um, that I've worked on. Um, there's a few in there that have, uh, direct, uh, relevance to, um, evaluation design and harness traceability and reliability procedures. And I also have this, like, monstrous three hundred-page archive tome that, you know, you can point your agents at.

[16:55](https://www.youtube.com/watch?v=A3FL_zTQKU0&t=1015s) Um, you can point your agents at all of this, right? Um, to, like, pull in some of those best practices into your own systems. So thank you.
