[
  {
    "startMs": 480,
    "text": "Hi, I'm Arius, founder of Pinterse AI."
  },
  {
    "startMs": 4480,
    "text": "Um, this talk is for people who build"
  },
  {
    "startMs": 7520,
    "text": "agents and run their own inference stack"
  },
  {
    "startMs": 10320,
    "text": "or plan to. And if that's you, you"
  },
  {
    "startMs": 13200,
    "text": "associate typically the KB cache, the"
  },
  {
    "startMs": 15040,
    "text": "bird and butter of modern inference"
  },
  {
    "startMs": 17279,
    "text": "engines, usually as a throughput"
  },
  {
    "startMs": 18640,
    "text": "optimization, the keys and values"
  },
  {
    "startMs": 20480,
    "text": "computed per token kept so the model"
  },
  {
    "startMs": 22480,
    "text": "doesn't compute them again. Today, I'll"
  },
  {
    "startMs": 24480,
    "text": "like to show KV cache in a different"
  },
  {
    "startMs": 26720,
    "text": "light. The way agents agent AI agent"
  },
  {
    "startMs": 29679,
    "text": "application could go wrong are keep"
  },
  {
    "startMs": 31679,
    "text": "growing, right? Prom injection,"
  },
  {
    "startMs": 33200,
    "text": "jailbreaks, broken tool calls, sensitive"
  },
  {
    "startMs": 35200,
    "text": "tool calls, sensitive data exposures,"
  },
  {
    "startMs": 37040,
    "text": "hidden back doors and drifts and so"
  },
  {
    "startMs": 39520,
    "text": "forth, right? So the way right now we"
  },
  {
    "startMs": 41600,
    "text": "typically guard against against this"
  },
  {
    "startMs": 43520,
    "text": "numerous ways that an AI agent"
  },
  {
    "startMs": 45120,
    "text": "application could go wrong is by"
  },
  {
    "startMs": 46879,
    "text": "guarding the edge by perhaps passing a"
  },
  {
    "startMs": 49920,
    "text": "judge model that checks the prompts, the"
  },
  {
    "startMs": 52879,
    "text": "inputs and the response outputs against"
  },
  {
    "startMs": 55440,
    "text": "your policy. All right? or you build a"
  },
  {
    "startMs": 58000,
    "text": "classifier for each case. But here is a"
  },
  {
    "startMs": 61199,
    "text": "different light. The model turns every"
  },
  {
    "startMs": 64320,
    "text": "input into a computation before it acts"
  },
  {
    "startMs": 67200,
    "text": "before it decodes before it generates"
  },
  {
    "startMs": 69119,
    "text": "the response before that response"
  },
  {
    "startMs": 70720,
    "text": "becomes another tool call. The KV cache"
  },
  {
    "startMs": 73520,
    "text": "is your engine's record of that"
  },
  {
    "startMs": 75840,
    "text": "computation that modern inference engine"
  },
  {
    "startMs": 78159,
    "text": "for free are using it and reading it."
  },
  {
    "startMs": 80880,
    "text": "And therefore in this talk I would like"
  },
  {
    "startMs": 83439,
    "text": "to suggest as as a complement to the way"
  },
  {
    "startMs": 86720,
    "text": "Are you already using your own"
  },
  {
    "startMs": 87840,
    "text": "guardrails to use this data and"
  },
  {
    "startMs": 90320,
    "text": "integrate it with your KV cache"
  },
  {
    "startMs": 91920,
    "text": "infrastructure? I refer to this as a KB"
  },
  {
    "startMs": 94799,
    "text": "first security signals and basically"
  },
  {
    "startMs": 97520,
    "text": "three reasons that matters, right? So"
  },
  {
    "startMs": 100240,
    "text": "the data is already there. You pick the"
  },
  {
    "startMs": 103040,
    "text": "gems from the pile your inference stack"
  },
  {
    "startMs": 105520,
    "text": "already was throwing away. It covers the"
  },
  {
    "startMs": 108560,
    "text": "guardrails blind spots because whatever"
  },
  {
    "startMs": 111119,
    "text": "slips past those guardrails, it still"
  },
  {
    "startMs": 113439,
    "text": "becomes computation and it stays in"
  },
  {
    "startMs": 115840,
    "text": "place. So it's perfect for data privacy"
  },
  {
    "startMs": 118399,
    "text": "application for cases where you do not"
  },
  {
    "startMs": 120479,
    "text": "have the luxury to read the inputs and"
  },
  {
    "startMs": 122560,
    "text": "outputs. And the recipe is pretty"
  },
  {
    "startMs": 124960,
    "text": "straightforward, right? It has three"
  },
  {
    "startMs": 126479,
    "text": "steps. You learn a baseline for your AI"
  },
  {
    "startMs": 129280,
    "text": "workload, for your AI inference traffic,"
  },
  {
    "startMs": 131520,
    "text": "how your KB page typically look like,"
  },
  {
    "startMs": 134160,
    "text": "and then find the KV pages that are rare"
  },
  {
    "startMs": 138160,
    "text": "against your baseline, typically your"
  },
  {
    "startMs": 140640,
    "text": "anomalies, and then you can even"
  },
  {
    "startMs": 142959,
    "text": "classify and explain those. So first"
  },
  {
    "startMs": 145280,
    "text": "let's start with where in this stack"
  },
  {
    "startMs": 147760,
    "text": "this integration could happen."
  },
  {
    "startMs": 150640,
    "text": "The interesting thing about the KV cache"
  },
  {
    "startMs": 152800,
    "text": "is that in a way existence of those had"
  },
  {
    "startMs": 155680,
    "text": "made the inference in a way stateful"
  },
  {
    "startMs": 158560,
    "text": "right. So for example, things like KBK"
  },
  {
    "startMs": 160720,
    "text": "cache management layers such as LM cache"
  },
  {
    "startMs": 163200,
    "text": "stores them from GPU down to remote"
  },
  {
    "startMs": 166560,
    "text": "storage or LLMD and Nvidia dynamo route"
  },
  {
    "startMs": 169440,
    "text": "each new request towards the worker that"
  },
  {
    "startMs": 171599,
    "text": "already holds perhaps that cached KV"
  },
  {
    "startMs": 174800,
    "text": "page. So there are two integration"
  },
  {
    "startMs": 176720,
    "text": "points you you you can integrate those"
  },
  {
    "startMs": 179920,
    "text": "anomaly detection or those classifiers"
  },
  {
    "startMs": 182319,
    "text": "into either you can implement it"
  },
  {
    "startMs": 184800,
    "text": "directly inside of your inference"
  },
  {
    "startMs": 186159,
    "text": "engines vg so forth. The beauty of this"
  },
  {
    "startMs": 189599,
    "text": "approach is that it's real time and and"
  },
  {
    "startMs": 192560,
    "text": "the the the alert that it would perhaps"
  },
  {
    "startMs": 195280,
    "text": "generate could ride along your future"
  },
  {
    "startMs": 198720,
    "text": "tool calls or your future policies. So"
  },
  {
    "startMs": 202000,
    "text": "you can harden basically your policies"
  },
  {
    "startMs": 204239,
    "text": "or guardrails by just generating the"
  },
  {
    "startMs": 206640,
    "text": "computational anomalies that you observe"
  },
  {
    "startMs": 208959,
    "text": "within your inference engines"
  },
  {
    "startMs": 211519,
    "text": "and perhaps you can integrate it with as"
  },
  {
    "startMs": 214159,
    "text": "a as a way of thread discovery in a"
  },
  {
    "startMs": 216159,
    "text": "sense. So you can integrate with your LM"
  },
  {
    "startMs": 218640,
    "text": "cache. So it reads the stored KB pages"
  },
  {
    "startMs": 222480,
    "text": "and find things that don't belong with"
  },
  {
    "startMs": 224560,
    "text": "the rest. Right? So now I want to show"
  },
  {
    "startMs": 227040,
    "text": "you some numbers and and just a very"
  },
  {
    "startMs": 229360,
    "text": "simple hello world use case. if this is"
  },
  {
    "startMs": 231840,
    "text": "actually working and this is a valid"
  },
  {
    "startMs": 234799,
    "text": "approach and for that let me use a use"
  },
  {
    "startMs": 237519,
    "text": "case that has no text to filter at all a"
  },
  {
    "startMs": 240319,
    "text": "voice agent. So here's a case study of a"
  },
  {
    "startMs": 243680,
    "text": "voice agent. In this case it was a gem"
  },
  {
    "startMs": 245519,
    "text": "for E4B. Our baseline was 72 ordinary"
  },
  {
    "startMs": 248720,
    "text": "clips and spoken four voices. We're"
  },
  {
    "startMs": 251599,
    "text": "looking at 16 token pages. Obviously"
  },
  {
    "startMs": 253519,
    "text": "this granular granularity could change"
  },
  {
    "startMs": 256479,
    "text": "but here we just use shorter pages. So"
  },
  {
    "startMs": 260479,
    "text": "we could have more data in the baseline."
  },
  {
    "startMs": 262079,
    "text": "And then on the test we have different"
  },
  {
    "startMs": 264479,
    "text": "voices and one clean voice and the other"
  },
  {
    "startMs": 267919,
    "text": "one has a hidden instruction within it."
  },
  {
    "startMs": 269919,
    "text": "And as you can as is shown here not only"
  },
  {
    "startMs": 273120,
    "text": "you can detect these cases just using"
  },
  {
    "startMs": 276160,
    "text": "simple anomaly detection but also you"
  },
  {
    "startMs": 278400,
    "text": "would know which part of um which page"
  },
  {
    "startMs": 282800,
    "text": "which part of your request had this"
  },
  {
    "startMs": 285360,
    "text": "anomalous event. We even went further"
  },
  {
    "startMs": 287680,
    "text": "and we trained a model that uses the KD"
  },
  {
    "startMs": 290800,
    "text": "pages as features and we then we're"
  },
  {
    "startMs": 293520,
    "text": "trying to explain it or tag it with"
  },
  {
    "startMs": 295440,
    "text": "things that are useful. In this case,"
  },
  {
    "startMs": 297440,
    "text": "although it wasn't even trained on audio"
  },
  {
    "startMs": 299440,
    "text": "is able to generate correct information"
  },
  {
    "startMs": 301600,
    "text": "about personal information, home"
  },
  {
    "startMs": 303120,
    "text": "address, location data, travel"
  },
  {
    "startMs": 304560,
    "text": "information, and destination."
  },
  {
    "startMs": 309199,
    "text": "So now back to text agents. In this"
  },
  {
    "startMs": 311280,
    "text": "case, it was a quan 3 14 billion served"
  },
  {
    "startMs": 313520,
    "text": "live on 11 cache. The one on the left"
  },
  {
    "startMs": 317600,
    "text": "shows you an idea of how attacks and"
  },
  {
    "startMs": 319919,
    "text": "normal traffic would typically separate"
  },
  {
    "startMs": 321919,
    "text": "from each other. And then the one on the"
  },
  {
    "startMs": 323840,
    "text": "right, which is perhaps more meaningful,"
  },
  {
    "startMs": 326000,
    "text": "it was using the AT bench and hogging"
  },
  {
    "startMs": 329120,
    "text": "face. It has its own baseline of benign"
  },
  {
    "startMs": 332320,
    "text": "agent traces and eight kind of agent"
  },
  {
    "startMs": 335440,
    "text": "failure from injected tools description"
  },
  {
    "startMs": 337680,
    "text": "to misinformation and so forth. We look"
  },
  {
    "startMs": 340639,
    "text": "at the top 10 alerts that anomalous"
  },
  {
    "startMs": 343440,
    "text": "anomaly detection would generate and"
  },
  {
    "startMs": 345039,
    "text": "about seven to eight times of out of 10"
  },
  {
    "startMs": 348320,
    "text": "of these alerts are actually anomalous"
  },
  {
    "startMs": 351440,
    "text": "meaningful and useful anomalous"
  },
  {
    "startMs": 353039,
    "text": "activities. We even went further and we"
  },
  {
    "startMs": 355600,
    "text": "looked at whether there's any kind of"
  },
  {
    "startMs": 358240,
    "text": "anomalous events that stand out more"
  },
  {
    "startMs": 360880,
    "text": "than the rest."
  },
  {
    "startMs": 363680,
    "text": "Um, two make things hard in production."
  },
  {
    "startMs": 366960,
    "text": "First models change finetunes and new"
  },
  {
    "startMs": 371440,
    "text": "releases, but also agents that each run"
  },
  {
    "startMs": 374720,
    "text": "on a different model and routers that"
  },
  {
    "startMs": 377440,
    "text": "pick a specialist model per request."
  },
  {
    "startMs": 380080,
    "text": "That means each model writes its pages"
  },
  {
    "startMs": 382160,
    "text": "its own coordinates. So either you learn"
  },
  {
    "startMs": 385039,
    "text": "every baseline, classifier and explainer"
  },
  {
    "startMs": 388000,
    "text": "which is computationally prohibitive or"
  },
  {
    "startMs": 391360,
    "text": "you have to find a way to map them all"
  },
  {
    "startMs": 394479,
    "text": "into one shared representation space."
  },
  {
    "startMs": 398080,
    "text": "Second real time means moving big"
  },
  {
    "startMs": 401440,
    "text": "objects. For example, a 1256 token chunk"
  },
  {
    "startMs": 405280,
    "text": "of quen 314 billion is about 40"
  },
  {
    "startMs": 408240,
    "text": "megabytes of KV compression and the"
  },
  {
    "startMs": 411280,
    "text": "streaming like hashgen could help but"
  },
  {
    "startMs": 414400,
    "text": "moving that for security analysis and"
  },
  {
    "startMs": 416319,
    "text": "acting in real times is still a slow and"
  },
  {
    "startMs": 418639,
    "text": "that's what we solve at in AI. We are"
  },
  {
    "startMs": 421120,
    "text": "looking for teams building their own"
  },
  {
    "startMs": 422720,
    "text": "inference stacks who treat security as a"
  },
  {
    "startMs": 425280,
    "text": "foundational design decision. I hope"
  },
  {
    "startMs": 427440,
    "text": "this was helpful for you. Thank you so"
  },
  {
    "startMs": 429120,
    "text": "much for listening."
  }
]
