AI Engineer NYC 2026 Online Talks
Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
Slide chapters
22 slide and demo groups · Timestamps are approximate.
Read the talk
Paving the Road to Production: Governed Enterprise AI Agents
Harish Gaggar walks through an enterprise agent platform at Intuit Credit Karma, from registration and authenticated tool access to checkpoints, evaluation records, sensitive-input filtering and context optimization.
From a talk by Harish Gaggar
At a glance
Ideas worth remembering
A paved path should create supporting resources as well as code: workload identity, memory storage, tracking tables and secret management need to be ready when the agent deploys.
The MCP gateway centralizes authentication and routing; reviewed MCP tools constrain the operations and data available behind it.
Policy branches should distinguish execution from auditing: a rejected sensitive action still needs a record, even though it does not run.
Threads preserve conversational continuity, runs identify individual executions, and checkpoints save workflow state at graph nodes.
The demonstrated optimizer reduced one query from 24,000 to 2,400 tokens. That establishes a token reduction for the example, without establishing preserved answer quality or business ROI.
The missing foundations around a working agent
An agent can answer a question locally while leaving its most consequential production questions unresolved. Who authenticated the tool call? Which operations may the tool perform? Can a follow-up recover the previous conversation? Did an API key enter the model prompt—or get copied into a trace? Harish Gaggar, a senior staff software engineer responsible for AI analytics infrastructure and services at Intuit Credit Karma, opens with these gaps between an experiment and an enterprise deployment.
The proposed response is a collection of supporting frameworks, rather than a “magical agent.” LangGraph supplies task-specific nodes and workflow state. An MCP gateway authenticates requests and routes them to custom MCP servers, whose tools perform the operations. Memory, tracking and evaluation, sensitive-data filtering and context optimization handle concerns that a functioning prompt does not resolve by itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Registration creates the repository and its production resources
The paved path starts before implementation. A developer registers the agent’s use case and receives an agent ID. In an internal developer portal, they supply that ID, the intended use case, a framework such as LangGraph or LangChain, a language such as Python or TypeScript, and the skills or prompts they want to use. Those choices become inputs to repository generation.
The generated repository includes the agent framework together with memory, tracking and evaluation, context optimization and PII filtering. This addresses a predictable inconsistency: individual developers may implement only the pieces their coding assistant suggests, or concentrate on one or two frameworks. Supplying the surrounding components at creation time gives agents a consistent starting point and makes their results easier to inspect centrally.
Development then moves through a concrete release sequence. The developer clones the repository, builds a Docker image and tests query inputs and outputs with the supporting storage. Once the agent works locally, a pull request requires two approvals. The release process registers the agent centrally and provides a Langfuse tracing URL; CI/CD builds the production container and scans it for vulnerabilities. After security approval, the image enters the registry and an internal orchestrator deploys it into a Kubernetes namespace. Scanning is a release control, not proof that an image contains no vulnerability.
The portal also provisions the workload identity, memory database, tracking and evaluation tables, and secret management. That provisioning matters at deployment: the agent should already have its identity binding and know which resources to use. The orchestrator can deploy an agent with its supporting resources prepared, rather than leave each team to connect identity, storage and tracing afterward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reviewed tools behind a centralized MCP gateway
Custom MCP servers provide a place to control what tools expose. Gaggar recommends building them internally for an air-gapped environment and describes using a FastMCP wrapper to define the tools. Each tool undergoes internal security review before production use. Filters, limits and excluded fields can then constrain what the tool returns, including information that should never reach the model. The decision is about the operations and data exposed by a tool, not merely whether its server runs inside the company.
Both users and service accounts must authenticate through the MCP gateway. At the first query, a coding assistant or agent sends an authentication request. The gateway saves session state in Redis and returns a JWT access token to the client. This establishes a session before the client requests a tool operation.
For the tool request, the client supplies the JWT as a bearer token and addresses a namespace identifying the intended MCP server—for example, an Artifactory or GitHub namespace. The gateway uses that namespace to select the server and attaches the user token to the forwarded request. The MCP server calls the downstream service, handles the described token exchange and returns data within the user’s authorized scope. It then packages the response for the gateway to return to the client.
Authentication and routing therefore happen on the same required path: client, gateway, MCP server, downstream endpoint. The intended design prohibits clients from bypassing the gateway to call those endpoints directly. A central entry point makes authentication consistent across local coding tools and deployed agents, while the reviewed server tools control the operations available behind it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate operational logs, request traces and aggregate metrics
A centralized gateway also creates a useful place to observe failures. Its monitoring separates three kinds of evidence, each answering a different operational question:
- Operation logs: a store such as ELK, Loki or CloudWatch records events that help investigate failures such as an unsuccessful token refresh.
- Request traces: OpenTelemetry collects and processes trace data for Tempo. A trace helps identify which tool was slow, whether a request originated in an agent or an IDE, and where a particular call failed.
- Aggregate metrics: Prometheus stores measurements such as a tool’s P95 latency and changes in authentication rejection rates. These reveal patterns across calls rather than the history of one request.
Grafana supplies the visualization layer for the traces and metrics. The distinction is practical: an authentication rejection spike tells an operator that something changed; a trace and the corresponding operation logs help explain which request failed and why. These gateway records describe access and execution behavior. Later, the agent’s own tracking tables address questions about the answer, model use and cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Classify the operation before allowing it to execute
LangGraph provides loops, branches and state management for the agent workflow. The first stages interpret the question and form a plan, then classify the requested operation: reading information, modifying it, deleting it or performing another sensitive action. That classification changes how the plan proceeds. The plan itself is a subgraph containing the substantive work.
Evidence gathering uses the same authenticated tool path described earlier. The workflow calls the MCP gateway, reaches the appropriate MCP server and retrieves data through a tool. After validating the findings, it checks policy. Read-only operations can proceed to execution and auditing; write or sensitive operations wait for human approval. Approval permits execution. Rejection still produces an audit record, so the record includes decisions that stopped an action as well as actions that ran.
Where does human approval change the outcome? The branch below separates permission to execute from the obligation to record the decision. Both approval and rejection reach the audit log, but only approval allows the sensitive operation to run. This is how the workflow places a human decision between a proposed action and its effect.
Policy determines whether execution needs human approval
Read the diagram as text
- Check policy. Evaluate the operation after validating the gathered evidence.
- Wait for human approval. Required for write or sensitive operations.
- Execute. Run the read-only operation or the approved sensitive operation.
- Record in audit log. Preserve executed actions and rejected decisions.
- Check policy → Execute: Read-only.
- Check policy → Wait for human approval: Write or sensitive.
- Wait for human approval → Execute: Approved.
- Wait for human approval → Record in audit log: Rejected; do not execute.
- Execute → Record in audit log: Record execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Threads, runs and checkpoints preserve different parts of state
Memory management saves and restores workflow state over time. A thread identifies the continuing conversation. The first question creates a run ID, and execution at a graph node saves a checkpoint. A follow-up stays on the same thread but creates another run ID, with checkpoints saved as that run reaches its nodes. Keeping those identifiers distinct lets the system connect multiple executions to one conversation without treating them as one undifferentiated attempt.
The local demonstration uses SQLite; Cloud Spanner is the named production option on GCP. Initially, the queried database contains no data. After submitting a question, a query returns a record containing the thread ID, the question and its execution time. Further queries inspect how many messages belong to that thread. The observable change is a persisted record of the interaction, rather than state held only in the running process.
What can an operator reconstruct from those records? The subsequent queries inspect thread and run IDs, the number of tools involved, the particular MCP tool executed and the progression through LangGraph nodes. Together, those views connect a user interaction to its workflow execution. The demonstration inspects stored state and execution history; it does not stage a crash-and-recovery test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Store the run’s behavior before asking whether it paid off
Agent-level tracking records the submitted question, response, model, token consumption, cost and evaluation information. Gaggar recommends collecting it from day one. In his example, 100 agents have 100 projects with their own tracking and evaluation tables; those records can later be aggregated into dashboards. The purpose is to make questions about performance and return on investment answerable six months later, rather than attempt to reconstruct missing history.
The demonstration starts with six tables. After a query is submitted, the first inspection finds no data yet; a later refresh reveals the tracking row. What identifies that run and its result? The record includes request and session IDs, agent name, question, output, completion status, elapsed time, model and metadata. The delay between submission and visibility is part of the example: an immediately empty result does not establish that tracking failed.
The remaining views answer different questions about that same run:
- Quality checks: eight checks per run, with separate rows for dimensions including success, latency, cost, tool accuracy and safety.
- Evaluation scores: a single scoreboard row collects quality dimensions scored between zero and one.
- Quality metrics: a wide row uses nested fields for response, quality, safety, latency, cost and tool statistics.
- Tool invocations: each call records which MCP tool ran, how long it took and whether it succeeded.
- Agent trajectory: ordered records describe LLM calls and graph states, such as planning and summarization.
These tables establish a structure for inspection, not a validated quality score for the system. The walkthrough does not explain how the quality checks are computed or calibrated. Likewise, aggregated token and cost records can support an ROI investigation, but the demonstration does not measure business ROI. The useful foundation is the ability to relate a result to its model calls, tools, timing and evaluation fields.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filter sensitive input before it spreads into storage and traces
A document upload may contain sensitive information, and a typed query may accidentally include an API key. The PII trimmer classifies data types and assigns severity levels. Gaggar describes blocking and redacting secrets when severity exceeds approximately 90%. That threshold belongs to the filtering rule; it is not a reported detection accuracy.
The intended interception point is ingestion, before the input reaches the model, memory database, tracking and evaluation tables or tracing system. This placement matters because preventing a model call alone would still leave a secret exposed if logging or checkpointing had already copied it elsewhere. The design aims to stop that spread early. Its effectiveness still depends on detection and redaction coverage; the walkthrough does not establish that every sensitive input will be caught.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce the context before paying to process it
Large uploads and accumulated data can make each query expensive. The context optimizer shapes and prunes the payload before it reaches the model. It selects material within a token limit, reranks it for relevance, removes duplicates and compresses the remaining payload. Each operation targets a different source of excess: too much material, poorly prioritized material, repeated material and an unnecessarily large representation.
The first demonstration asks the agent to find all artifacts in JFrog Artifactory. The tracking view begins empty; after submission and refresh, a record reports 24,000 tokens used. Gaggar then enables the optimizer behind the scenes and reruns the same question.
How much does the recorded usage change? The optimized run reports 2,400 tokens, down from 24,000—a 90% reduction. The comparison makes the cost mechanism concrete: the question remains the same while the optimizer changes the material sent onward. The demonstration does not include a paired answer-quality study, so the smaller token count alone cannot establish that the answer retained all the required information.
ts ctx_opt_on input_tokens total_tokens ctx_saved
2026-08-27 05:40:36.692114 UTC true 2460 2958 23106
2026-08-27 05:39:14.895019 UTC false 24588 25130 null
A second example runs through a Slackbot and asks for Docker images across all repositories in Artifactory. Its first recorded run also uses about 24,000 tokens. After enabling the optimizer and repeating the question, the reported usage falls to approximately 1,500 tokens. Gaggar reports savings of 70–78% in most cases, distinguishing those broader observations from the larger reductions in the two demonstrations. Across many agents issuing many daily queries, reducing repeated context could materially reduce token consumption.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the supporting framework part of the first deployment
The closing recommendation returns to the portal’s original purpose: include the supporting frameworks before the agent ships. Every tool request, whether it originates locally or from a deployed agent, should pass through centralized authentication. MCP servers should control which tools can execute the requested operation. Monitoring and audit records should cover those executions, and high-risk actions should retain human approval.
Context optimization belongs in that initial framework too, because cost becomes harder to ignore as usage grows. The paved path makes these concerns repeatable: identity and resources arrive with the deployment; tool access follows a common route; decisions and executions leave records; context is reduced before model use. The engineering decision is to make those capabilities part of creating an agent, rather than rediscover their absence after deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Extends the evaluation discussion by showing how execution traces and human annotations can guide agent changes and train an evaluator.
Read the complete timestamped transcript
16 slide groups · 6 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate.
Governed enterprise AI agents
Harish Gaggar introduces a path from AI agent experiments to enterprise production, drawing on his work owning AI analytics infrastructure at Intuit Credit Karma.
Text in image
Paving the Road to Production: Governed Enterprise AI Agents Harish Gaggar Senior Staff Software Engineer, Credit Karma
- 0:03
Hello, everyone. Um, I would like to walk you through that how we can transition an AI agents from an experiments to the real enterprise environment. So before I begin, a quick introduction about me. My name is Harish Gaggar. I'm a senior staff software engineer at Intuit Credit Karma, where I built and own the AI analytics infrastructure and services. Um, and beside that, um, I'm advisor, uh, author, researchers, um, and also a program committee members for various organizations where
- 0:33
I mentor the young researchers as well. All right. So let's, uh, walk through the, um, agenda here. So first we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems, and how we, uh, can build a build different framework to address that, uh, different challenges here. Um, then how the architecture looks like for it. Um, then, um, how you can build your own MCP servers, um, and
The infrastructure roadmap
The agenda covers production challenges and architecture, MCP servers and gateways, and agent frameworks, memory, evaluation, PII blocking and context optimization, under a governed path scoped, sanitized and audited by design.
Text in image
Table of Contents 01 Challenges 02 Frameworks 03 Architecture 04 MCP Server 05 MCP Gateway 06 MCP Gateway Monitoring 07 Agent Framework 08 Agent Memory 09 Agent Tracking and Evaluation 10 Context PII Block 11 Context Optimizer Governed Path Scoped, sanitized & audited by design
- 1:03
MCP gateway, and then some monitoring on top of that gateway as well. Um, then further, we're gonna talk about what are the different type of agent framework you can have, um, and then, um, some additional framework around managing the memories, tracking and eval, um, how you can block the PII data, um, going before the LLMs, and then context optimizer to save your token, right? So without delay, let's, um, let me walk you through the challenge. Um, of course, like, there are
Missing production controls
Gaggar attributes production failures to missing controls: authenticated routing, MCP tools to run queries, tracking and evaluation, memory, token-saving context optimization, and secret protection.
Text in image
Challenges Enterprise AI agents often fail in production when foundational controls are missing
- 1:33
many agents, uh, uh, generally get failed in productions because their foundation controls are missing. Um, and the key reason behind is that, um, you know, like, oh, most of the people are missing the gateway to make sure that agent can, you know, communicate to the gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers, right? Second, that, um, MCP servers are missing the right tool to execute your query. Third, do we have, like,
- 2:03
enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating their quality response or not, right? Do we have any in-memory management system or not in terms of, let's say, you are doing the interaction with agent for long or for many days, uh, is it able to recall that what you have been asked in the past, right? And then there are other framework, of course, like do you have a context optimizers to save the token or do you have inner production in place to make sure that
- 2:32
your any secrets should not go into LLM or tracking or tracing or any memory DB or so, right? So lot of, lot of open questions. Um, so in order to resolve this, uh, we are not going to, you know, like, build a magical agent here in this demo. Uh, we're gonna talk about, like, different framework that you can build to address this. So first is the agent framework. Um, this is where, like, we use the, like, we choose to use the LangGraph framework, which is provided by LangChain
Design a framework, not just an agent
Task-specific agent nodes use authenticated, customized MCP tools; operations are audited. The framework strips PII and secrets, tracks queries, evaluates responses, persists memory, and trims context to reduce token use.
Text in image
Design framework not just an Agent for solution 01 Agent Framework Task-specific nodes, with Skills, Prompts or internal marketplace Plugins 02 OAuth before every call Each MCP Tools call runs authenticated 03 Customized MCP MCP tools customized to internal security practices 04 Continuous Audit Every operations logged at MCP gateway 05 PII/secrets stripped Sensitive data scrubbed before it reaches to LLM 06 Tracking and Eval Per-query tracking: Model, tokens, latency, response quality 07 Memory Management thread and checkpoints 08 Context Optimizer Trims & Summarization context to cut unnecessary token spend Each layer - scoped, sanitized, audited and measured by design
- 3:02
community. Uh, this is help us in building the task-specific node based on your questions, and which is easy to plug in any, like, uh, marketplace plugins or skills and prompts to make sure that agents, um, respond efficiently based on your query. Further, um, the second part is we are also building the MCP gateway to make sure that whenever agent make the call or any service account from that agent make the call, make sure that they are securely get authenticated to their
- 3:32
MCP gateway, um, and then get redirected to the right MCP servers. Third is the customized MCP, which means that you are building your own MCP servers with the right tool in place to execute your query. And then finally, some audit, uh, on top of what ex-actually operation you're performing, right? And then we talk about some additional framework around, um, you know, like how you can block the PII secrets, how you're gonna store the tracking eval into the cloud, um, and how you're going to persist the memories,
- 4:02
especially, um, any threads and checkpoints on every context. And then also how you can optimize overall your context to make sure that you don't bunch, don't burn too many tokens, uh, when you run your query. All right. So we'll talk about now the agent infrastructure. So the, firstly, we'll go through the local development. Um, the, the way the approach should be is that, um, you know, like, whenever you build an agent, right, first you have to register your agent into some
Build the agent infrastructure
A registered agent gets a portal-created repository and local memory, tracking and evaluation. A PR registers it for Langfuse tracing; CI builds, scans and deploys its image to GKE with identity, memory, tracking, evaluation and secrets.
Text in image
Building Agent Infrastructure Local Development Agent Registration Dev Portal Repository Local Resources Memory Tracking & Eval
Show 2 builds
- 4:17
Build the agent infrastructure — recorded presentation build 1
Text in image
Building Agent Infrastructure Local Development Agent Registration Dev Portal Repository Local Resources Memory Tracking & Eval
- 7:11
Build the agent infrastructure — recorded presentation build 2
Text in image
Building Agent Infrastructure Local Development Agent Registration Dev Portal Repository Local Resources Memory Tracking & Eval Agent Registered with LangFuse URL Deployment Pipeline circleci CI build docker Image Scan Agent POD Deploy GKE Cluster Namespace 1 Agent 1 KSA 1 Namespace 1 Agent 2 KSA 2 Agent Workload Identity and Dependent Resources Agent Memory Track & Eval Agent Secret
- 4:17
- 4:32
system. You usually tell that, "Hey, look, this is my, the use case, and these are the use case going to use for this particular vertical or system." Um, and you feed all those information and generate a ID for your agent, right? Once you have that, then you can have in dev portal, internal portal where you can ingest all the details saying that, "Hey, look, this is my agent use case. This is my agent ID. This is the framework I'm gonna use, either LangGraph, LangChain or so. These are my language I'm gonna use, either Python or TypeScript."
- 5:02
Um, and then you can choose, um, you know, like, your skills or prompts, um, and then you submit your informations, right, to this. Once you submit this, the dev portal is going to create a repository which is going to contains not just the agent framework, like your LangGraph agent framework, but is also going to contains, um, your memory framework, your tracking eval framework, right? And then your context optimizer framework and your PII blocking envir-- uh, framework as well. So the reason
- 5:32
for that is because not necessarily all your developers are taking care of all this framework, right? They just sometime, um, you know, create the framework what generally, like, AI assistant tool are generally support, right? Or they stick to one or two framework, right? So we just wanna make sure that when you roll out such framework, make sure they are consistent across the company, um, and they can take care of, um, and they can be centralized in one place so you can visualize the results as well, right?
- 6:02
So once you have that repository, so developers basically, you know, clone that repository, they can build their agent as a Docker image into containers And test their agent input and output through the query, and they can also test this on their local memory, maybe Cloud Spanner Evaluator here. And you can also store this tracking eval in some of your DB, maybe SQLite or maybe BigQuery, um, to see, to make sure that agents are fully working on your local, right? So once your agents are running in local, um, then you're ready to go ahead and generate a
- 6:32
PR, and then you can, once you get your two PR approval, then it's go to the CI/CD pipeline, right? So the PR is basically going to, firstly, is going to, uh, register your agent into some centralized agent registry system, which provide a Langfuse URL to it to make sure that you see all the traces. And then the CI/CD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system. Uh, it will scan that image, make sure that there's no vulnerability. Um, if everything looks great,
- 7:02
security approved, you can move that e-image back into the registry, right? Further, you have your own deployment platform or orchestrator platform, which going to deploy that agent into the Kubernetes cluster in a given namespace, right? So the good part here is that this dev portal is not just creating the repository, um, right? It also do create, um, bunch of other production resources. For example, the workload identity, um, right, and then memory DB or
- 7:32
tracking and eval tables, and then agent secret management, right? Um, so it based on whatever the name we provide, it not just, again, like, not just repository, it create all these different resources as well. So what happen is when this internal orchestrator, when it deploy the agents, the agent already know that, "Hey, what workload identity I supposed to work with," right? That binding should already happen. So once this case to GSA binding happens, the agent will start communicating to all these resources.
- 8:02
Once everything in place, um, agent is further going to communicate to rest of your endpoint services like for example, AI gateway or MCP servers or Langfuse for tracing or some other API endpoints, right? So now firstly, we talk about the MCP server as well. So we talk about architecture, now we're going to the different component. So MCP server is something that you can definitely use something which is outside available. But if you're working on air gap environment, it's highly recommended that you build your own
Scoped tools through MCP servers
A FastMCP server exposes typed tools for REST APIs, with filters, limits and blocked fields. The speaker calls for security review before production and gateway authentication for users or service accounts.
Text in image
MCP server A protocol layer that turns REST API into typed, permission-scoped tools an agent can safely call • A FastMCP server exposes a set of typed tools • Server enforces request filters, limits, and blocked fields. • Supports user or service-account identity via MCP Gateway MCP client MCP MCP server
- 8:32
MCP servers. So, um, yeah, uh, and you can use the, um, I will not go with the definition, everyone knows about it. So the way the framework has been built here is that w-you, you can leverage the FastMCPServer package, um, or wrapper to, um, you know, build that MCP servers, and you can create a different set of tools inside that FastMCP wrapper to construct your, um, you know, the overall servers. And then you make sure that, you know, like every tool when you
- 9:02
define an MCP server, they had to be fully reviewed by your own internal security team to make sure that only authorized tool can be go in production. And that's how you can, you know, enforce some filters, limits and block the fields, which is really not required to go to LLM and get you the information which is you're not supposed to get, right? Um, and third part is like a, of course, you need to make sure that you build your MCP server in a way that whenever the traffic is
- 9:32
coming from your MCP client to MCP gateway to MCP servers, either is coming from user or service account, there has to be, uh, authenticated to the MCP gateway first, and so that they can be access your MCP tool, um, to get that right information. All right. Um, now the gateway part. So the how the gateway works, so this is again, you can build it your own internal MCP gateway if you are again working on your air gap environment. So the way it work is you are working on Claude Assist- uh, like local,
Authenticate and route through a gateway
Clients authenticate through the gateway, which stores sessions in Redis, unwraps JWE tokens and routes user identity to MCP servers. A server exchanges the ID token for scoped access; clients cannot reach servers directly.
Text in image
MCP Gateway Centralized authentication and routing hub that securely connects Coding assistance tools/Agents to enterprise services Redis Grafana MCP Gateway 5. Namespace Lookup 6. Unwrap JWE 7. Attach User Token Rate Limit Backend MCP Servers Artifactory MCP Github MCP 1. One Time OAuth 2. User Session State 3. JWE Access Token 4. tool/call ->namespace/mcp ->Auth Bearer <JWE> 8. Exchange id_token (Short Lived/User scoped access token) + MCP Jfrog API call 9. Jfrog user scoped response 10. MCP response 11. Gateway Return final Payload to Client user’s OIDC id_token JFrog GitHub * MCP server never faces the client directly
- 10:02
you know, code assistance tool or maybe your agent, right? So the moment you submit your first query to your agent, it will send that one-time authentication request to the MCP gateway, right? MCP gateway recognized, uh, and take your request. First it will save your session state into the Redis, right? And then it will, um, you know, return back the JWT access token back to the, your client, right? Once client will receive that JWT token, um, it will know that what MCP server needs to call, right? So it
- 10:32
will submit that tool/call request on a given namespace/MCP, meaning, let's say, artifactory or githubs/MCP with the auth bearer, uh, JWT, um, token, um, into that. So once MCP gateway, um, receive that, uh, it will understand that what namespace, meaning MCP server needs to look up for, and then it will unwrap that JWT token, and then it will attach that user token in that request and send it to the respective MCP servers. Now, once MCP
- 11:02
server receive it, it will make that respective API calls to that endpoint of that, um, you know, uh, to the service, and then it will exchange that ID token, uh, and return the response based on whatever scope that user is authorized to see, right? So once the user receive the token, um, you know, uh, MCP server basically, you know, unwrap that all the data, make the payload which is compatible for MCP gateway to receive. And then finally, the final payload will send back to the client to see the data
- 11:32
points. So what it shows is that, you know, like we build this MCP gateway to make sure that we have a centralized authentication and routing hub that securely connect any coding assistant tool or agents to the endpoint via this MCP gateway and MCP server. There, there should not be direct access provided, um, on any endpoints. There has to go through the MCP gateway through proper authentications, uh, before they're reaching to the MCP server and the endpoints. All right.
Monitor gateway compliance
The gateway sends logs to a log store and traces and metrics through OpenTelemetry to Tempo and Prometheus. Grafana visualizes the telemetry, including per-tool P95 latency and authentication rejection spikes.
Text in image
MCP Gateway Monitoring & Compliance Comprehensive telemetry, distributed tracing, and real-time visualization for governed AI workflows MCP Gateway → OTLP traces + metrics → OpenTelemetry Collector OpenTelemetry Collector → traces → Tempo (traces) → Grafana (visualization & dashboards) OpenTelemetry Collector → metrics → Prometheus (metrics) → Grafana MCP Gateway → operational logs → Log store (e.g. ELK, Loki, CloudWatch)
- 12:03
So now the part is the gateway monitoring, right? We talk about, again, we talk about architecture, we talk about MCP server, MCP gateway. Now it's a monitoring. Again, the AI agent cannot be trusted without continuous monitoring, evaluation and feedback, right? This is why we are building the MCP gateway s-uh, monitoring. So the way the monitoring work is usually, like, the moment you submit any calls to your agent- Uh, it will send all the operation log into the log store. You can use ELK, Loki or CloudWatch also, right?
- 12:33
And generally, this contains information like show me like all last photo ones in my last 10 minutes call, right? And what did or do when the token refresh failed, right? So those kind of requests can go to the log store. Now, there are some, you know, traces and metrics data, um, like real-time data that goes to the OpenTelemetry as well. Um, OpenTelemetry again is open source solutions, which going to collect all your traces, um, and metrics, and is going to process the data and
- 13:02
redirect to the back-end system, right? And in this case, the traces will go to the Tempo. This is again a open source solutions, which going to store all your traces in terms of like, uh, traces in the sense like, you know, like information around like which tool was slow on this request, whether this call is coming from agent or IDE, uh, what was the exact error, um, on this failed call or so, right? So these kind of data are going to store into the Tempo. Oh, my bad. Okay. And then the metrics is about,
- 13:32
uh, the Prometheus is all about the information about like, um, what are P95 latencies for a given tool, right? Um, whether my auth, uh, rejection is spiking on a particular time or not, right? So those kind of metrics is going to store in Prometheus. So once you get all this data point, like traces and metrics, this entire aggregate data back actually save into the Grafana for your visualizations. Right. All right. So
- 14:03
the MCP server gateway and monitoring is covered. Now we're going to talk about the, uh, framework. So the agent framework, right? So we use LangGraph, uh, m- for most of the cases. Um, and it's basically LangGraph, again, is, uh, uh, provided by LangChain community, is provide a graph-based architecture that support, uh, loops, branching, state management for complex multi-agent AI workflows, right? And the way it work is like the moment you submit your first
The agent workflow
The agent fetches and validates evidence before policy routes reads to audit and writes or sensitive operations to human approval. Approved operations execute; both approval outcomes go to audit.
Text in image
Agent Framework Core Workflow Start → Interpret → Classify → Plan agent_subgraph Plan → Fetch evidence (MCP Gateway -> MCP) → Evidence (Validate Findings) → Policy (Risk route) Policy → Read → Audit Policy → Write/Sensitive → Approval (HITL) Approval → Approve → Execute (Approved writes) → Audit Approval → Reject → Audit Audit (summarize + log) → flagEND Gateway-First Security Every tool call authenticated & governed before access Artifactory
- 14:33
query, uh, we do interpret, um, you know, like your question first, like what type of questions it is and what response it's looking for. Um, and based on that, uh, it's create a plan, right? Um, then it will send for classifi- uh, classify, uh, area where it will determine that, hey, whether it is just like a read operations or whether it is just like a, um, you know, modification level changes coming up or whether you really want to delete something, right? Or some sensitive operations.
- 15:02
Based on that, um, it adjusts the overall structure and then plan accordingly. The next step is plan, is actually itself is a subgraph where you do the major of your operations. Meaning that in the plan, like once you know what operation you're performing, it goes to the, like, let's say in the fetch evidence section, where it actually call that whatever the question you ask here, after all interpret and classify, um, it actually go ahead and make the call to the MCP gateway, uh, make
- 15:32
sure they get authenticated, um, and then it make the call to the MCP server using the right tool, um, and get that data, and then generate that evidence, right? Once you validate all your finding, then it will check the policies. Uh, it will check whether it is a read-only, then go ahead and execute, and w- I will save all the record points into our audit logs. But if it is a write and sensitive operations, definitely it will go through the HITL process where you wait for an approval. If it is approved, you go ahead and execute. If it is
- 16:02
get rejected, in either case, it's going to save into the audit log, right? So if we combine all this together, like MCP gateway, MCP server, um, and the agent, right? This is how things looks like. So ultimately, whatever the agent you are building, whatever the call you are submitting, it will go through the MCP gateway for authentications, and then it will redirect to the right MCP servers to execute, uh, you know, your query on a given
One governed execution path
The LangGraph agent sends requests through the MCP Gateway, which authenticates and governs access. It passes a user ID token to git-mcp, which calls GitHub. The slide says every request runs as a real identity through the audited gateway.
Text in image
One Governed Path - Agent - Oauth - Artifactory LangGraph Agent Orchestrates agent workflows MCP + gateway token x-mcp-client-type: agent MCP Gateway Authenticates and governs access user_id_token git-mcp MCP host protocol layer API Call Github Target repository Every request runs as a real identity through audited gateway
- 16:32
tool, uh, so that it make the API call to that endpoint and get the response, right? All right. If you t- if you think from the infrastructure perspective, this is how it looks like. So agent is communicating to the MCP gateway, goes to the MCP server and to the endpoint. All right? So now the, uh, we talked a lot about MCP gateway, MCP servers, um, the monitoring, um, and the agent framework. Now we'll talk about, uh, three and four additional
Infrastructure boundaries
A pipeline builds and scans images, then deploys agent pods to one GKE cluster. Agents use workload identities and call endpoints through an MCP gateway and server in a separate cluster; memory, evaluation and secrets support them.
Text in image
At Infrastructure Level Local Development Agent Registration Dev Portal Repository Local Resources Memory Tracking & Eval Agent Registered with LangFuse URL Deployment Pipeline CI build Image Scan Agent POD Deploy GKE Cluster Namespace 1 Agent 1 KSA 1 Agent 2 KSA 2 Agent Workload Identity and Dependent Resources Agent Memory Track & Eval Agent Secret GKE Cluster Namespace 1 MCP Gateway Authenticates, governs access, Route MCP Server Tools Shared Services LangFuse Agent Tracing AI Gateway LLM Services API Endpoint External Services
Persist and restore agent memory
Each question creates a run ID on a thread. Checkpoints at workflow nodes save state to a shared memory database for restoration across runs; follow-up questions create new runs on the same thread.
Text in image
Agent Memory Management Agent Persistent refers to the ability to save and restore the state of a workflow over time (including HITL), ensuring durability and seamless recovery across runs. Agent Framework Super-Step 1: Start Super-Step 2: Node1 Super-Step 3: Node3, Node2, Node4 Super-Step 4: End Thread Run_ID-1: Checkpoint, Checkpoint, Checkpoint, Checkpoint Run_ID-2: Checkpoint, Checkpoint, Checkpoint, Checkpoint Memory DB
- 17:02
framework that we should have to take care of when building the end-to-end ecosystem, right? One is them is agent memory management. Again, it's, uh, uh, you know, it's a, it's refer-- it's basically a process where you can save and restore the state of your workflow over the time, right? Ensuring that, uh, it's provide you a durability and seamless recovery across, uh, across your every query run. So the way it work is like whenever you submit any query to your agent, um, it
- 17:32
actually generate a thread, right? Uh, on that given thread, uh, when you ask your first question, it will going to create a run ID for that. So the moment you ask some deeper questions, it will go on a particular node in that LangGraph and execute that query. And on that node, it going to save that respective checkpoint here. Right? Now, if you ask a follow-up questions on the same thread, it going to create another run ID. And again, depend on your follow-up question, it go to a specific node
- 18:02
and save the checkpoint over there, right? And all this data for your local development, you can use the SQLite, SQLite DB, uh, but in production, unless in GCP, you can use, um, Cloud Spanner and, um, in respect to databases, you can use in Azure, Azure and AWS as well, right? So just walk you through the quick demo here, um, on the memory. So for example, um, I have, you know, like just build this agent locally and then, um, you know,
- 18:32
trying to run this query on SQLite DB to see if there is any data or not. Um, so I'm just You know, initially I don't have any data. Now I submit my query, you will see that I receive the one record, which is saying that, "Hey, this is my thread ID, this is the question I have been asked, and exact time, um, you know, when this query was executed." Now, if I run the next query to see that, "Hey, how many messages have been produced on a given thread?" Um, you can get this information by running the further queries, right? Now, if I would
- 19:02
like to run to see that, um, you know, like all the metadata level information on my query, um, I can run my query again, and you can see the detailed description around, you know, like how many tool I get involved, what is my thread ID, what is my run ID, um, you know, like, and all those informations. Now, if I want to also see that, "Hey, what tool have been executed when I run this query?" Um, you can go ahead and see the exact tool name as well on that given MCP servers. Now, further I run-- a-also want to see that
Inspect run metadata and tool use
The demo queries the memory database for run metadata without message content, then prepares a query to list the tools used in each run.
Text in image
Supply-Chain Agent ARTIFACTORY · XRAY · CURATION Agent Investigations run as multi-step graph plans. Lookups require Artifactory MCP [illegible] Thread Scan incoming package versions for typosquatting, install-sc[illegible] Risk & compliance Answered by the supply-chain graph — these work even without an Artif[illegible] Exploitable CVEs Why it's in prod Inventory & cleanup Repositories and storage come from your live Artifactory over MCP; the [illegible] Repositories Scan incoming package versions for typosquatting, insta[illegible] Investigation detect_supply_chain_attack ON reference[illegible] • requests@2.31.1 score 1.0: new install scripts: [illegible] indicators: ['[redacted URL]', [illegible] maintainer has no 2FA • liblzma@5.6.1 score 1.0: new install scripts: [illegible] indicators: ['ssh-func']; maintainer account is 4 days [illegible] changing release; New ifunc resolver + binary delta COMPLETED planner: llm audit: [illegible] Tool activity Terminal: # Message counts per thread (no text) sqlite3 -header -column ~/.jfrog-agent/memory.db \ "SELECT t.thread_id, COUNT(m.id) AS msgs, SUM(CASE WHEN m.role='user' THEN 1 ELSE 0 END) AS user_msgs, SUM(CASE WHEN m.role='assistant' THEN 1 ELSE 0 END) AS assistant_msgs, datetime(MAX(m.ts),'unixepoch','localtime') AS last_msg FROM threads t LEFT JOIN messages m ON m.thread_id = t.thread_id GROUP BY t.thread_id ORDER BY MAX(m.ts) DESC;" thread_id msgs user_msgs assistant_msgs last_msg [illegible] 1 1 0 Aug 26, 2026 at 21:12:56 # Per-run metadata from assistant messages (outcome, planner, tools - no content) sqlite3 -header -column ~/.jfrog-agent/memory.db \ "SELECT datetime(ts,'unixepoch','localtime') AS ts, thread_id, json_extract(meta,'$.run_id') AS run_id, json_extract(meta,'$.outcome') AS outcome, json_extract(meta,'$.planner') AS planner, json_extract(meta,'$.request_type') AS request_type, json_extract(meta,'$.affected') AS affected, json_extract(meta,'$.audit') AS audit_file, json_array_length(json_extract(meta,'$.tools')) AS tool_count FROM messages WHERE role='assistant' ORDER BY id DESC LIMIT 20;" # Per-run metadata from assistant messages (outcome, planner, tools - no content) sqlite3 -header -column ~/.jfrog-agent/memory.db \ "SELECT datetime(ts,'unixepoch','localtime') AS ts, thread_id, json_extract(meta,'$.run_id') AS run_id, json_extract(meta,'$.outcome') AS outcome, json_extract(meta,'$.planner') AS planner, json_extract(meta,'$.request_type') AS request_type, json_extract(meta,'$.affected') AS affected, json_extract(meta,'$.audit') AS audit_file, json_array_length(json_extract(meta,'$.tools')) AS tool_count FROM messages WHERE role='assistant' ORDER BY id DESC LIMIT 20;" ts thread_id run_id outcome planner request_type affected audit_file tool_count Aug 26, 2026 at 21:13:23 [illegible] [illegible] completed llm read 4 [illegible] 5 # Tools used per run (from meta only) sqlite3 -header -column ~/.jfrog-agent/memory.db \ "SELECT json_extract(meta,'$.run_id') AS run_id, j.value AS tool FROM messages, json_each(json_extract(meta,'$.tools')) j WHERE role='assistant' ORDER BY messages.ts DESC LIMIT 30;" Interface text: COMMAND: Overview, Agent, Threads INVESTIGATIONS: Workbench, Reachability, Attack detection, Forensics, Reproducibility, License exposure, Retention INVENTORY: Artifact explorer, Promotion gate SECURITY: Xray overlay, Policies, Compliance GOVERNANCE: Approvals, Audit trail, Observability, Agent evaluation View less Connection Artifactory MCP connected Ask a supply-chain question (reachability, forensics, license, retention, inventory) 0:56 / 1:3119:19Demo milestone
- 19:32
how the progression was happened on my LangGraph framework. That also you can see by, uh, running this type of query on your me-memory DB. You can see that here, like different kind of, you know, nodes been called when actually, um, you know, that query has been run. All right. Now, the tracking and eval. So just to make sure that, uh, there is no confusion here, like, uh, we talked about earlier the MCP gateway monitoring, this is like agent level tracking. So we do this
Track and evaluate agent runs
Separate from gateway monitoring, agent tracking records queries and responses, plus model, token use, cost and latency for each LLM call. Quality feedback can be aggregated to review performance over time.
Text in image
Agent Tracking and Eval • Distinguish agent-side LLM telemetry from the gateway’s operational telemetry • Record model, token, cost, and latency for each LLM call • Capture answer-quality feedback and summarize it
- 20:02
because we really want to track that what query you have been submitting to the agent, um, and what the response is coming, what LLM model you have been used, what's the cost of your token, uh, how many token you have been burned, and what's your, you know, evaluation criteria looks like. What is the quality of your response or not, right? So you need to store that data from the day one. And imagine that if you have a thousand plus agent, every agents will have their own, um, you
- 20:32
know, um, agent tracking and eval, um, project and the tables, right? So if you combine, let's say if you have hundred agents, you will have hundred different projects, and each project has their own respective tracking and eval tables. Now, tomorrow, six months down the line, if somebody asks you that how your agent has been performed, what was the response, what was the quality, how much token you've been burned, you can aggregate all this data and you can create executive dashboards around it to see, um, whe- to see the what is the ROI in building so many agents,
- 21:02
right? So let me walk you through the, the tracking and eval, uh, demo here. So, um, here, like, if I look in, uh, this one, so first, like, you know, like if you see that there are six tables are listed here. One is a job tracking table, um, and which is contains most of this, like, information around what query, like what is your query, what's the response and everything. And there are a bunch of other tables as well, like your quality related and your tool
The job-tracking schema
The BigQuery job-tracking schema captures query inputs and outputs, request and session IDs, status and timestamps. The presenter says the data appears after a delay, so he refreshes to see the run.
Text in image
agent_job_tracking This is a partitioned table. Learn more Schema Field name Type Mode ts TIMESTAMP NULLABLE request_id STRING NULLABLE session_id STRING NULLABLE trace_id STRING NULLABLE agent_name STRING NULLABLE experience_id STRING NULLABLE input STRING NULLABLE input_sanitized STRING NULLABLE output STRING NULLABLE output_sanitized STRING NULLABLE status STRING NULLABLE error_message STRING NULLABLE start_ts TIMESTAMP NULLABLE Additional columns: Description, Key, Collation, Default Value, Data Governance Tags, Policy Interface text: Set up billing to upgrade to the full BigQuery experience. Learn more Search BigQuery resources Show starred only Connections agent_tracking_eval agent_eval_scores agent_job_tracking agent_quality_eval agent_quality_metrics_detail agent_tool_invocations agent_trajectory Datasets / agent_tracking_eval / Tables / agent_job_tracking Query · Chat · Open in · Share · Copy · Snapshot · Delete · Export / sync Schema · Details · Preview · Insights · Lineage · Data Profile · Data Quality Filter · Enter property name or value Edit schema · View row access policies
21:10Demo milestone
- 21:32
invocations and then node level executions and all that. So in this case, I'm going to submit my first query on my agent, um, and you will see that, um, there will not be, uh, any data in the first place, uh, because it takes some time, so I will refresh this again. And this time I will see that when I refresh, I do see that for a given query, I get my request ID, my session ID, my agent name, um, you know, what question I've been asked, um, and what was the output,
- 22:02
um, of that, uh, question, uh, whether it has completed or not, and then what is the time it took, what is the model I've been used, um, and then some of the meta le-le, uh, metadata level informations, right? Further, if I look for the quality, again, there is a eight quality checks per run, right? And you can see that this one row each for, you know, like success, like latency cost, tool accuracy, safety, et cetera, like all this has been stored here, right?
Per-run quality metrics
The BigQuery preview shows eight rows for one run, one per quality check, including tool selection accuracy, safety compliance, cost and latency.
Text in image
agent_quality_eval This is a partitioned table. traffic_date: 2026-08-27 eval_run_id: run-b9ed41645463 request_id: run-b9ed41645463 agent_name: jfrog-langgraph-agent metric_name: 1. tool_selection_accuracy 2. parameter_correctness 3. safety_compliance 4. recovery_resilience 5. task_success_rate 6. cost 7. step_efficiency 8. latency Interface text: Set up billing to upgrade to the full BigQuery experience. Learn more Search BigQuery resources Show starred only Connections agent_tracking_eval agent_eval_scores agent_job_tracking agent_quality_eval agent_quality_metrics_detail agent_tool_invocations agent_trajectory bqgcs-177117 / Datasets / agent_tracking_eval / Tables / agent_quality_eval Query · Chat · Open in · Share · Copy · Snapshot Schema · Details · Preview · Insights · Lineage · Data Profile · Data Quality Results per page: 50 1–8 of 8
22:19Demo milestone
- 22:34
Now, the eval scores. This is again, this is going to be a one row. This is going to be a scoreboard for your given run, which is basically varies between zero to one, which is scores, uh, you know, like all the quality dimension in a single row, right? Now, the quality metrics. So again, it's a deep dive on that particular one run. Basically a nested field for the response, quality, safety, latency, cost, uh, and the tool stat in one wide row, right? Now, the tool
- 23:03
invocation. So every tool call, which MCP tool ran, how long it took, and whether it succeeded or not, right? And now the tool traj-- agent trajectory, which is talk about step-by-step story of the run, each LLM call, and the graph state in order, like plan, summarize, et cetera. All right. So now, uh, the next is the agent PII trimmer. This is very, very important because many times happen
Trim PII before the model call
The trimmer scans queries and tool output, then redacts or blocks sensitive data by type and severity before the LLM call. The example blocks a phone-number request and asks the user to retry.
Text in image
Agent PII Trimmer before LLM query User query / Tool output (artifact props, CVE text, build info) → DET → “REDACT” (replace by span → [REDACTED_EMAIL]) → DECIDE (severity ≥ 90?) System prompt: untouched No → LLM → Persist telemetry → offline batch → PII Scanner (alert ≥ 50) Yes → BLOCK: reason names types, never values Data Type | Severity | Default Outcome Private key block, AWS secret key | 100 | Block AWS access key, GitHub, OpenAI, Stripe, Slack, connection string | 95 | Block Bearer token, JWT, GCP key, credential filename | 90 | Block api_key= / password= assignment | 90 | Redact SSN | 100 | Redact* Credit card | 95 | Redact* Passport / DOB / email / phone / IPv4 | 90 | Redact please find phone number [REDACTED_PHONE] at artifact publisher Blocked: personal data detected in input (phone). Remove it and retry — the agent does not process personal data. BLOCKED planner: guard · audit: —
- 23:33
that when you're working in agents or, uh, this thing, um, sometime there is a possibility that there are some secret being ingested as, as a part of your query, right? Let's say you are uploading some doc which has some sens-sensitive informations, um, and sometime while typing, you may have some API keys or so. So we have a framework which define that different sort of data types and their severity level. If your severity is above ninety percent or so, uh, we will block and redact that, uh,
- 24:03
secrets, uh, immediately so that, uh, make sure that that secret is never goes to LLM models, right? Neither it will goes to your memory DB or nor it will go to tracking and eval tables or tracing systems as well. So the goal here is to just block this sensitive data as it is the moment it get ingested, right? Moment, moment you type. All right. Um, the last framework is all about now is the context optimizer. This is one of
Forecast context growth
Before an LLM call, the context optimizer selects within a token limit, reranks, deduplicates and compresses the payload. The chart shows lower mean live token use per task with the layer across five models from four providers.
Text in image
Forecast the budget before context snowballs Context Optimizer Framework User turn → SHAPE → PRUNE → LLM select → rerank → cluster → compress Mean live token cost per task, five models, four providers Y-axis: Mean tokens per task (thousands) Y-axis ticks: 0, 50, 100, 150, 200, 250, 300 Legend: No layer; With layer gpt-4o-mini (OpenAI): 52k; 25k gpt-4.1-mini (OpenAI): 320k; 15k claude-sonnet-4.5 (Anthropic): 187k; 31k gemini-2.5-flash (Google): 277k; 15k nova-pro (Amazon): 73k; 20k
- 24:33
my favorite area where like, um, you know, uh, whenever you submit... It's very important, right? Whenever you submit too many queries, sometime you upload lot of the um, data points and you upload bunch of files systems or so, right? It's very important that time is, um, you know, like you make sure that you save your token in every query. So we build in context optimizer framework where you actually shape your query, and then you prune it before you go to LLM. In the pruning system,
- 25:03
uh, we do select the payload based on that token limit, then we re-rank that, "Hey, which is the most relevant, relevant information that we should target for when you submit this huge giant, you know, context?" And then we dedupe that, uh, remove all the duplicates and create a very unique set of, uh, very curated data like, uh, I mean, the main data for the LLM. And then we finally compress this large payload before it goes to the LLM. So let's see this, uh, demo here.
- 25:33
Uh, this will be quick. So for example, let's say I will submit my first query, "Hey, find me all the artifacts in my JFrog Artifactory." Um, you see that first when bef- even I not submitted any query yet, there is no data here. So the moment I submit this first query,
- 25:51
and if I refresh my query here, um, and then you can see that the record has been, uh, added. So you can see there is a twenty-four thousand token has been used. I will go back a little bit, um, just to show the thing. So you can see this twenty-four thousand token has been used for given questions. Now if I ask, uh, now I enable the context optimizer behind the scene, and I'm rerun the exact same query again. So this time you will see that, uh, you know, for the same query, if I refresh this run,
Compare context-optimizer token use
For the same query, the run with context optimization enabled shows 2,460 input tokens, versus 24,588 without it. Total tokens drop from 25,130 to 2,958.
Text in image
Untitled query SELECT ts, (SELECT value FROM UNNEST(metadata) WHERE key = 'ctx_opt_enabled') AS ctx_opt_on, (SELECT value FROM UNNEST(metadata) WHERE key = 'input_tokens') AS input_tokens, (SELECT value FROM UNNEST(metadata) WHERE key = 'total_tokens') AS total_tokens, (SELECT value FROM UNNEST(metadata) WHERE key = 'ctx_opt_tokens_saved') AS ctx_saved, input FROM `bqgcs-177117.agent_tracking_eval.agent_job_tracking` This query will process 486 B when run. Using on-demand processing quota Query results [illegible]-08-27 05:39:14.895019 UTC | ctx_opt_on: false | input_tokens: 24588 | total_tokens: 25130 | ctx_saved: null Interface text: Set up billing to upgrade to the full BigQuery experience. Learn more Search BigQuery resources Show starred only Connections agent_tracking_eval agent_eval_scores agent_job_tracking agent_quality_eval agent_quality_metrics_detail agent_tool_invocations agent_trajectory Run · Save · Schedule · Share File · Edit · Run · Tools Chat · View all jobs · Save results · Open in Job information · Results · Visualization · JSON · Execution details · Execution graph Results per page: 50 1–1 of 1
25:53Demo milestone
Compare context-optimizer token use
For the same query, the run with context optimization enabled shows 2,460 input tokens, versus 24,588 without it. Total tokens drop from 25,130 to 2,958.
Text in image
Untitled query SELECT ts, (SELECT value FROM UNNEST(metadata) WHERE key = 'ctx_opt_enabled') AS ctx_opt_on, (SELECT value FROM UNNEST(metadata) WHERE key = 'input_tokens') AS input_tokens, This query will process 1.12 KB when run. Using on-demand processing quota Query results Row 1: ts: Aug 27, 2026 at 05:40:36.692114 UTC | ctx_opt_on: true | input_tokens: 2460 | total_tokens: 2958 | ctx_saved: 23106 Row 2: ts: Aug 27, 2026 at 05:39:14.895019 UTC | ctx_opt_on: false | input_tokens: 24588 | total_tokens: 25130 | ctx_saved: null Interface text: Set up billing to upgrade to the full BigQuery experience. Learn more Search BigQuery resources Show starred only Connections agent_tracking_eval agent_eval_scores agent_job_tracking agent_quality_eval agent_quality_metrics_detail agent_tool_invocations agent_trajectory Run · Save · Schedule · Share File · Edit · Run · Tools Chat · View all jobs · Save results · Open in Job information · Results · Visualization · JSON · Execution details · Execution graph Results per page: 50 1–2 of 2
26:20Demo milestone
- 26:22
and you will see that, uh, this time it used just two thousand four hundred token, right? For exact same query. So which is almost like ninety percent saving. I can rerun the same demo on the Slackbot as well, where I'm asking Slackbot, "Hey, tell me all my Docker images across all the repositories in Artifactory." When I run this query, there is no data. Now I'm going to submit my query. So once this query will run, um, and if I run my query again here, and you
A completed Slack agent query
The Slack agent completed a request for every Docker image across all repositories, with vulnerability counts by severity sorted worst to best, but found neither. The run used 24,675 tokens in two LLM calls.
Text in image
JFrog Supply-Chain Agent Your JFrog Cloud trial is wired for Artifactory inventory (repos, search, storage) [illegible] is connected. Xray/Curation/AppTrust are not on the trial API — [illegible] the agent’s supply-chain graph (same engines as the console). Submit these in Slack (work now, no extra JFrog SKU): • Is CVE-2024-0727 reachable in production? • Scan for typosquat / xz-style package behavior • Why is payment-service:319 in prod? • Same source, hash mismatch for payment-service • GPL exposure in payment-api • Optimize retention under regulatory minima Submit these only if Artifactory MCP is logged in (trial): • Which repositories exist? • How much storage are we using? • Find artifacts not downloaded in 180 days Do not submit (will be denied on trial): promote, delete, quarantine, live Xray watch edits. Approvals stay in Streamlit. Copy a line without a leading dash. Harish Gaggar 7:15 AM @JFrog-Agent - List every Docker image across all repositories along with their full vulnerability counts by severity, sorted from worst to best JFrog-Agent APP 7:15 AM Working List every Docker image across all repositories along with their full vulnerabil... outcome: completed The findings do not contain any Docker images or their vulnerability counts by severity. Therefore, I cannot provide the requested information. 24,675 tokens in 2 LLM call(s) · run [illegible] Interface text: # all-jfrog-agent-demo Messages Add canvas Invite teammate Message #all-jfrog-agent-demo
26:45Demo milestone
- 26:52
will see the record get created, it's again used like twenty-four thousand for this, this particular query. Now, what I will do is I will, uh, turn on my optimizer behind the scene, and I rerun the same query in the Slackbot. Um, and then if I run this query again, um, it will show like it exactly used one thousand five hundred almost token, which is again like ninety percent saving. In most of the cases, we get like seventy to seventy-eight percent token saves. So imagine that you have thousands of, um, you know, like,
A completed Slack agent query
The Slack agent completed a request for every Docker image across all repositories, with vulnerability counts by severity sorted worst to best, but found neither. The run used 24,675 tokens in two LLM calls.
Text in image
JFrog Supply-Chain Agent Your JFrog Cloud trial is wired for Artifactory inventory (repos, search, storage) [illegible] is connected. Xray/Curation/AppTrust are not on the trial API — [illegible] the agent’s supply-chain graph (same engines as the console). Submit these in Slack (work now, no extra JFrog SKU): • Is CVE-2024-0727 reachable in production? • Scan for typosquat / xz-style package behavior • Why is payment-service:319 in prod? • Same source, hash mismatch for payment-service • GPL exposure in payment-api • Optimize retention under regulatory minima Submit these only if Artifactory MCP is logged in (trial): • Which repositories exist? • How much storage are we using? • Find artifacts not downloaded in 180 days Do not submit (will be denied on trial): promote, delete, quarantine, live Xray watch edits. Approvals stay in Streamlit. Copy a line without a leading dash. Harish Gaggar 7:15 AM @JFrog-Agent - List every Docker image across all repositories along with their full vulnerability counts by severity, sorted from worst to best JFrog-Agent APP 7:15 AM Working List every Docker image across all repositories along with their full vulnerabil... outcome: completed The findings do not contain any Docker images or their vulnerability counts by severity. Therefore, I cannot provide the requested information. 24,675 tokens in 2 LLM call(s) · run [illegible] Interface text: # all-jfrog-agent-demo Messages Add canvas Invite teammate Message #all-jfrog-agent-demo
26:54Demo milestone
Slack query token comparison
After the speaker enables the optimizer and reruns the Slack query, the records show input tokens falling from 24,592 to 1,557; total tokens fall from 24,675 to 1,640.
Text in image
Untitled query SELECT ts, (SELECT value FROM UNNEST(metadata) WHERE key = 'ctx_opt_enabled') AS ctx_opt_on, This query will process 1.06 KB when run. Query results [illegible]-08-27 14:18:01.493106 UTC ctx_opt_on: true input_tokens: 1557 total_tokens: 1640 ctx_saved: 23141 input: List every Docker image across all repositories along with their full vulnerability counts by severity, sorted from worst to best [illegible]-08-27 14:16:03.895498 UTC ctx_opt_on: false input_tokens: 24592 total_tokens: 24675 ctx_saved: null input: List every Docker image across all repositories along with their full vulnerability counts by severity, sorted from worst to best Interface text: Set up billing to upgrade to the full BigQuery experience. Learn more Run Save Schedule Share File Edit Run Tools Using on-demand processing quota Chat View all jobs Save results Job information Results Visualization JSON Execution details Execution graph
27:10Demo milestone
Six production takeaways
Use an enterprise framework: authenticate tool calls through a gateway, expose APIs as MCP tools, keep humans in control of high-risk actions, support trust with observability, privacy and persistence, and optimize context to manage cost.
Text in image
Key Takeaways 01 Build an enterprise framework, not just an AI agent. 02 Gateway-first security is the foundation. 03 MCP turns APIs into safe, agent-friendly tools. 04 Keep humans in control of high-risk actions. 05 Trust requires observability, privacy, and persistence. 06 Optimize context before it becomes cost. Interface text: 16
- 27:21
um, um, if you have thousands of like agents which is running like, you know, many queries per day, imagine how much token you can save with this framework. All right, so now the key takeaway here is, uh, you know, like, um, for us is like make sure that we build an enterprise framework, um, not just an agent, right? Which makes sure that you have all the framework underneath. Don't try to be in rush to building some agent quickly and deploy, ship it. Make sure that it has to go through
- 27:52
all the framework altogether, so that you should not realize at a later stage that, you know, I should have this framework in place, right? I-I-In the very first place. Second is like your make sure that you have a proper gateway system. Make sure that every tool call request, either you are doing this from your local or through agent, it has to go through some sort of centralized MCP gateway. Uh, make sure it authenticated before it reaching to the MCP servers. And then make sure that your MCP servers take a control
- 28:21
on what tool, uh, you know, uh, y-you should use, uh, to execute a particular query, right? And make sure that you have monitoring in place and complete auditing in place for that tool executions. And of course, you should, uh, you know, keep the always human-in-the-loop process for every, um, every action items, every high-risk items. Uh, make sure that there is a observability, um, and, you know, um, auditing has to be in place. And last but not least, make sure that context optimizer should
- 28:51
always be there to justify your cost, um, um, on your agents, right? Um, so just want to share is that I released my, um, MCP, uh, book recently on Amazon. So feel free to download, it's free in Kindle, um, and it's, uh, hard copy is also available. All right. Um, thank you so much for watching this video. Um, and if you do have any questions, feel free to, you know, like, reach out to me over LinkedIn
- 29:21
or Twitter. Uh, I'm happy to answer. Thank you again.