AI Engineer World's Fair 2026
Why Your Agent Disagrees With Itself (And What To Do About It)
Loading slides…
Read the talk
Why Your Agent Disagrees With Itself (And What To Do About It)
Repeated verdict changes can expose missing policy or evidence. Active learning, semantic memory, and episodic memory turn that disagreement into targeted feedback.
From a talk by Diane Lin
Before you start: Familiarity with LLM classification, model evaluation, and the distinction between training and inference will help.
The same input, a different decision
What should you do when the same model receives the same input and changes its decision? Diane Lin approaches this problem through both learning research and security operations. Her background spans continual learning at Imperial College London, one-shot learning with Josh Tenenbaum at MIT, and work as one of the first three applied scientists on Alexa’s question-answering team. She later researched zero-shot transfer at Vicarious before spending five years applying machine learning to cybersecurity at Zscaler. That experience led her to co-found Culminate to automate security-alert triage. At the time of the recording, she reports that Datadog had acquired Culminate earlier that year and that she was leading development of self-evolving agents. The practical problem is semantic inconsistency: different verdicts, not merely different wording.
Consider a hotel-review classifier whose only permitted labels are positive and negative. Today's LLMs can perform this familiar task well, yet repeated runs of the same review through the same model can occasionally produce opposite labels. Stochastic generation is an obvious initial explanation, but it does not tell you which inputs are vulnerable or how to improve them. A single evaluation run therefore gives an incomplete picture: repeat the evaluation and average its results across runs.
In security operations, the same behavior creates a more consequential problem. An analyst must decide whether an alert describes malicious activity requiring intervention or a benign false alarm that can be ignored. Lin’s example is a failed login to a Gmail account from a suspicious IP. Across repeated agent runs, some alerts remain consistently benign, others remain consistently suspicious, and a third group flips between the two.
A customer facing those changing verdicts must decide which result to trust. In a proof-of-concept vendor comparison, an agent that repeatedly changes its answer can lose credibility against one that gives stable decisions. The engineering question becomes more specific: where does the disagreement concentrate, and what information would resolve it?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The gray zone needs a policy and the right evidence
Lin observes that unstable examples tend to sit near the decision boundary, in a gray zone between classes. An obviously enthusiastic hotel review remains positive across runs. A more ambivalent review can support either interpretation: some wording sounds mildly positive, while other wording sounds negative. Even human experts may disagree.
The missing ingredient can be the hotel’s purpose for assigning labels. If a complaint concerns an experience the hotel cannot improve, one hotel may see no reason to classify it as negative. Another may want that dissatisfaction captured so it can pursue improvements. The label is partly an operational preference, not simply a property of the sentence. Until that preference is explicit, asking for a binary answer leaves the boundary underspecified.
Security triage adds another distinction: an attacker trying to enter an environment is different from an attacker who has entered it. A few blocked attempts may be routine noise for an enterprise that constantly receives such activity. Escalating every attempt could flood the alert queue. But if an attacker guesses the password, passes MFA, and gains access, the response changes substantially. Similar initial behavior can lead to very different outcomes.
Two questions must therefore be resolved: does the customer want notification about blocked attempts, and is there evidence that access succeeded? The first supplies a policy; the second supplies discriminating information. Lin’s broader diagnosis is that these cases also trouble human experts and traditional classifiers. The agent may be exposing ambiguity that already exists in the task. Finding and resolving that ambiguity is the next step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend review effort where the model can learn
Active learning addresses a practical constraint: you have many predictions to assess and too little human attention to inspect them all. An initial model might run in production in monitor mode, producing predictions on unlabeled data. The selection strategy identifies cases likely to be problematic and directs review toward them.
The traditional cycle proceeds as follows:
- Train an initial model and predict labels for an unlabeled dataset, potentially drawn from production.
- Select informative examples. In binary classification, uncertainty sampling can prioritize probabilities near
0.5. A committee can instead prioritize examples on which competing models disagree. - Ask a human to inspect the selected examples, correct labels, and determine whether additional information is needed.
- Add the reviewed labels—and any new features—to the next training round, retrain, and repeat.
The value comes from concentrating effort on cases that teach the model something, rather than reviewing every prediction equally.
For LLM agents, Lin changes two parts of that loop: selection and learning. In her team’s experience, model uncertainty scores were unreliable; confidence did not establish correctness. Disagreement across models or repeated runs provided a more useful signal for requesting human guidance. Repeated sampling of one LLM is her adaptation of the committee idea, rather than a requirement to maintain several separately trained models. Reviewers then label the disputed examples and provide feedback.
A minimal selection function can operate on normalized verdicts rather than generated explanations. Here, the failed-Gmail-login alert is selected because its repeated verdicts differ; a consistently benign alert is left out of this disagreement queue.
python
from collections.abc import Mapping, Sequence
def select_disagreements(
verdicts_by_alert: Mapping[str, Sequence[str]],
) -> list[str]:
return [
alert_id
for alert_id, verdicts in verdicts_by_alert.items()
if len(set(verdicts)) > 1
]
verdicts_by_alert = {
"failed-gmail-login": ["benign", "suspicious", "benign"],
"stable-alert": ["benign", "benign", "benign"],
}
review_queue = select_disagreements(verdicts_by_alert)
This selects a place to investigate; it does not decide which verdict is correct.
After review, fine-tuning remains an option, but Lin describes it as expensive. Her lighter-weight alternative is to augment the agent with semantic memory and episodic memory, allowing feedback to influence later decisions without requiring a new training cycle for every update.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn ambiguity into explicit domain knowledge
Semantic memory stores knowledge that explains how to decide. Return to the hotel review: if the missing information is company preference, add that preference to the knowledge base. Lin’s example policy classifies complaints about matters outside the hotel’s control as positive because the hotel does not want those complaints to demand attention. The rule supplies the operational meaning that the binary labels lacked.
For the security example, the key distinction is whether authentication ultimately succeeds. Password spraying specifically tries one or a small set of passwords across many accounts. Lin’s example customer policy treats a spray without a successful login as benign for triage, and one with a final successful login as malicious. Here, benign expresses the customer’s response policy; failed attempts can still be evidence of an attack.
| Evidence in the example | Customer’s triage label |
|---|---|
| Password spray, no successful login | benign |
| Password spray, final successful login | malicious |
The rule identifies the evidence needed to separate two superficially similar cases. Storing it as domain knowledge sharpens the agent’s decision boundary and gives human labelers the same explicit basis for making consistent decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse past decisions before asking for another review
Sometimes a human has already labeled a case but has not distilled its reasoning into a reusable rule. Episodic memory preserves the case and its decision. When a similar case arrives, the agent can retrieve the earlier example and use its label as a reference. The human intervention happened previously, so online decisions need not wait for someone to write new semantic knowledge.
That mechanism is especially useful for recurring security false positives. Similar noise appears repeatedly, making past reviewed cases valuable references. A new case that falls between familiar groups, however, may still require human attention. Automation handles the recurring cases and leaves review capacity for the examples memory cannot resolve.
| Memory | What it preserves | How it helps |
|---|---|---|
| Episodic | Similar cases and past labels | Reuses earlier decisions |
| Semantic | Explicit policies and domain rules | Explains the decision boundary |
The two forms are complementary. Use episodic memory for recurring situations; if no useful reference exists, or the agent still disagrees after consulting one, send the case to a human. The reviewer can then distill the missing domain knowledge into semantic memory for future decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the feedback loop changes
Lin connects this loop to three operational benefits:
- Consistency: Explicit knowledge and relevant past decisions help stabilize cases near the boundary, supporting user trust.
- Efficient quality control: Reviewers inspect a selected subset of outputs and provide missing information or clearer labels where that effort can be most useful.
- Customer adaptation: Accumulated feedback teaches the agent how a particular customer wants its environment handled.
The review process becomes a way to improve the product while using it, rather than a separate exercise in checking every answer.
Lin reports evaluating 93 cybersecurity alerts three times each. Approximately 25% changed verdict without the proposed solution; approximately 10% remained inconsistent after episodic memory. She describes roughly 15% becoming consistent, a breakdown consistent with an approximately 15-percentage-point reduction across the alert set, although exact counts are not supplied. The measured outcome is repeated-run verdict consistency, not correctness.
The remaining cases either lacked a similar past reference or still produced disagreement after consulting one. Those cases went to human review for additional knowledge to disambiguate the decision and adapt it to the customer’s environment. Lin does not report a numerical consistency or accuracy result after that review.
The practical response to a changing verdict is therefore to investigate the label and the available evidence before assuming the model itself is the problem. Disagreement points to a place where the system can learn. Fine-tuning is one option, but explicit policies and reusable reviewed cases offer another path: resolve what the agent does not know, preserve that resolution, and let subsequent decisions reflect the customer’s environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Burr Settles’s survey explains active-learning selection strategies, including uncertainty sampling and query by committee.
- MITRE ATT&CK: Password SprayingDocumentation
Defines password spraying and describes detection signals and mitigations for attacks across multiple accounts.
Datadog’s overview of automated Cloud SIEM investigations, supporting evidence, verdict recommendations, and activation options.
Full transcript
Loading transcript…