AI Engineer World's Fair 2026
Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio
Read the talk
Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons
Kanish Manuja explains how provider fallback, streaming, route-specific timeouts and guardrail placement change an LLM gateway’s failure behavior—and why centralized governance need not mean one company-wide traffic path.
From a talk by Kanish Manuja
At a glance
Ideas worth remembering
Prefer per-request provider fallback to blind retries, while using cooldown to keep a repeatedly failing primary out of later request paths.
Streaming commits delivered output to the selected provider. A mid-stream failure cannot be recovered through a transparent provider switch; provision the backup thoroughly for requests that can fail over.
Measure P99 per model and route, set corresponding timeouts and constrain available reasoning settings. Tail hedging launches another request after waiting for a slow primary.
Treat guardrails as fallible services: choose fail-open or fail-closed behavior, bound their runtime, consider fallback checks and place them deliberately relative to generation.
Protect the gateway with granular API keys, bounded queues, load shedding and traffic priorities. Shared governance can coexist with separate gateway deployments.
The system behind “Something went wrong”
“Something went wrong. Please try again” is a familiar ending to an AI interaction. Kanish Manuja, a principal engineer at Twilio, opens with that message because the system behind it can be much more carefully engineered than the message suggests. A gateway may route around provider failures and still reach a point where it cannot recover a response already underway.
An LLM gateway sits between applications and model providers. It handles routing, authentication, fallback, rate limits and governance. Those responsibilities bring four goals into the same request path: availability, latency, guardrails and cost. During degradation, improving one can worsen another. Sending a second model request may preserve availability while adding latency and expense; continuing without a failed guardrail may preserve service while weakening protection.
The gateway needs controls that callers can choose for their use cases. One recovery policy cannot express every application’s willingness to wait, spend or accept a weaker check. Productionizing the gateway means making those choices available before an incident forces them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the healthy provider before exhausting the unhealthy one
With one model provider, its availability limits the application’s availability: its outage becomes your outage. Conventional dependency handling starts with retries, exponential backoff and jitter, then opens a circuit breaker after enough failures. For a fast, inexpensive API, another attempt can absorb a transient fault. A slow, expensive model call consumes the request’s remaining time and spend much faster.
The basic recovery example uses providers A and B. The gateway sends a request to A. If A fails, it sends that request to B. The observable change is that A’s failure no longer immediately becomes an application failure: another provider gets a chance to answer. This per-request fallback avoids spending the entire recovery effort repeatedly calling A while B remains healthy. 3:00
Two dispatch strategies expose different tradeoffs:
- Sequential fallback: Try A, then B after A fails. The request pays for the time spent on A before B can begin.
- Parallel requests: Start both providers together when latency matters enough to justify duplicate work. Manuja describes this as doubling cost: both model calls run even when the primary might have succeeded.
Circuit breaking still has a role. Once A has been failing for some time, remove it from the request path, let it cool down and try reintroducing it after a few minutes. Per-request fallback handles the current failure; cooldown keeps later requests from repeatedly discovering the same unhealthy dependency.
Where should the failure counts live?
- Instance-local counters: Each serving instance keeps its own history in memory. Changing the deployment size changes how failures accumulate across instances, so the same configuration can produce different failover behavior.
- Fleet-wide counters: Shared infrastructure collects failure history across the deployment. This can help the fleet recognize an outage and fail over quickly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fallback stops being transparent when output escapes
A successful HTTP exchange with B is only part of a successful fallback. Providers can differ in tool-calling schemas, token limits and stop reasons even when they expose an OpenAI-compatible API format. A normalization layer can reconcile those differences, but the fallback path still needs testing against the application’s actual expectations. Similar request shapes do not guarantee interchangeable behavior.
Now change the A-to-B example by enabling streaming. A begins generating, and the gateway forwards its output to the client. The user sees progress instead of waiting thirty seconds for a wall of text. If A fails after that output arrives, the gateway cannot transparently replace the response with B’s answer: the client already has part of A’s response, and those tokens cannot be recalled. In this streaming design, the continuing response is committed to A. 5:14
Where does the recovery path change? The diagram separates a failure before output reaches the client from one after streaming begins. The first path can move to B; the second leaves an interrupted response because switching providers cannot undo the text already delivered.
That is the mechanism behind the opening error message. Streaming improves perceived speed by giving away a recovery lever. The tradeoff can be worthwhile—some use cases require streaming—but it must be part of the application’s failure design.
B also needs enough capacity to be a real last line of defense. Teams often provision and test the primary thoroughly while giving the backup less attention. Manuja recommends even more throughput, capacity or headroom for the fallback provider. Once the primary is unavailable, an inadequate backup can turn a recoverable provider failure into an application outage.
The primary provider starts handling the request.
Before delivery, fallback can give B a chance to answer. After A’s output reaches the client, a transparent switch cannot retract it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reasoning model’s normal can be a chat model’s outage
Availability failures announce themselves with errors and pages. Latency failures can stay quiet: a request remains in progress while the customer waits. Mixed workloads make that problem harder to recognize. Manuja’s examples include embedding and classification requests taking less than a second, chat requests taking three seconds and reasoning requests taking much longer. These illustrate different workload expectations rather than universal performance targets.
A gateway-wide latency number blends those expectations. It cannot tell you whether a chat route has become unusually slow or whether the gateway simply served more long-running reasoning requests. Track P99 per model, per route instead. P99 describes the slow tail of a particular request population; choosing the population is as important as choosing the percentile. As Manuja puts it, “A reasoning model’s normal is actually a chat model’s outage.” 7:24
Timeouts need the same specificity: set them per model class and route. Without a timeout, the gateway can continue treating an outstanding request as healthy even when it is no longer serving the application usefully. Manuja identifies missing timeouts as his leading cause of silent outages. A route-specific timeout gives the gateway a point at which waiting becomes failure and recovery can begin.
Reasoning and router models add another source of variation. Manuja reports production behavior where the same prompt could take two to sixty seconds and P99 suddenly rose to sixty seconds without an identified explanation. That experience concerns the systems he encountered, rather than a universal range for reasoning models. His first recommendation is to fix the reasoning level per route. Some controls may be unavailable: in many cases, he notes, temperature cannot be set to zero. Router models also choose underlying models behind an abstraction, so constrain the request settings you can control to avoid adding unnecessary variation.
Tail hedging offers a different response to a slow primary. Start the primary normally, then launch another request after the primary has spent enough time outstanding to trigger a delay threshold. Unlike starting two requests immediately, this duplicates work only after waiting. Unlike failure-triggered fallback, it acts while the primary is still running. The aim is to reduce the P99 tail, while accepting the cost of additional model work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Guardrails need failure policies and their own time budgets
Guardrails check for prompt injection, personally identifiable information and toxic output—including the practical problem of a model swearing at customers. Those checks add services to the system, and those services can fail too. Their failure forces an explicit choice:
- Fail open: Serve the request despite the unavailable check. Availability continues with reduced protection.
- Fail closed: Block the request because the check cannot run. Protection takes priority over serving it. 10:15
There is no universal answer across checks and applications. A toxicity-filter outage might be tolerable for one use case, allowing requests to continue; another use case may require blocking them. Choose the worst case you can live with as the default. That decision concerns the consequence of an unavailable check, rather than an assumption that the check will always be available.
Give each guardrail a specific time budget and timeout so it cannot become the component that determines how long the entire request takes. Manuja’s intended design keeps the LLM as the rate-determining step. Guardrails can also have fallback arrangements: secondary providers, secondary checks or cached decisions can preserve service when the primary guardrail provider is down.
Placement changes both latency and what the check can inspect:
- Pre-hook: Check the input before the model runs. Manuja considers this probably the safest placement, but its runtime adds serial latency before generation.
- Parallel check: Run the guardrail concurrently with model work. This overlaps their runtimes and is his preferred latency-saving option for structured output, with a recommendation to avoid streaming that output.
- Post-hook: Inspect output after generation. This placement suits output monitoring and auditing.
Why does parallel placement save time, and why is streaming awkward? The diagram shows the input splitting into two concurrent paths. The check no longer has to finish before model work starts. But streaming can let delivery get ahead of the checking decision—the same irreversible delivery problem that limited provider fallback.
Starts both paths in the parallel placement.
Concurrent work removes the serial wait before generation. Streaming can expose output while the guardrail decision is still pending.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The gateway becomes another dependency
Routing around provider and guardrail failures does not remove the gateway’s own failure modes. It is another dependency in the request path. Shared limits are one way it can spread trouble: a noisy tenant can consume capacity needed by unrelated requests. Segregate API keys as granularly as possible by route and use case so those workloads do not unnecessarily share the same limits.
A retry storm creates a different problem. Repeated attempts pile more work onto an already stressed service, and scaling out alone is insufficient as a recovery strategy. The gateway needs load shedding: the ability to stop accepting more work under overload. Its web-server queues must be bounded, rather than letting requests accumulate without a limit. 13:44
Traffic prioritization can determine which work survives that pressure. Under load, the most important use cases should continue receiving service rather than competing indiscriminately with every retry. Put load shedding into runbooks and game days so operators know the gateway supports it and can use it during an incident.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Centralize governance without concentrating all traffic
The final architectural question is whether the entire company needs one central gateway deployment at all. A common request path can become a single point of failure. Before putting every application behind it, identify the requirement that motivated the design. Manuja’s experience is that teams often want centralized governance: consistent cost tracking, rate-limit management and related controls. 14:34
Those controls can live in plugins or shared custom code while gateways remain decentralized. One team can manage the gateway system without operating one company-wide deployment. Even a deployment distributed across instances can remain a shared dependency for every caller; distributing instances does not by itself separate the applications’ request paths. Explore separate deployments with shared governance before making all company traffic depend on the same gateway deployment.
The ending brings those decisions back to the customer. Manuja asks the audience to “prevent one incident.” The next step is small enough to act on: find one route without a timeout, one backup without enough headroom or one gateway queue that can grow without a bound, and fix the failure before customers encounter it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A separate account of a restaurant order-taking prototype comparing a speech-to-text → LLM → text-to-speech pipeline with an end-to-end realtime audio approach. Useful for extending the talk’s latency, cost and control tradeoffs to voice applications; its conclusions reflect that prototype and the models used then.
Related talks
- The State of Model Routing — NVIDIA, Cognition, OpenRouter
Continues the model-routing subject introduced by provider fallback and router-model latency.
- $1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpintero
A companion on guardrail models for readers considering the cost and implementation of checks placed around generation.
Read the complete timestamped transcript
- 0:01
[music]
- 0:13
I'm Kanesh Manuja. I'm a principal
- 0:15
engineer at Twilio.
- 0:18
Let's start with a quick show of hands.
- 0:20
Who here has seen the message, something
- 0:23
went wrong. Please try again.
- 0:27
Well, we have a few lucky ones and a few
- 0:30
that have had a good lunch. Um, so
- 0:34
behind that simple message is actually a
- 0:37
system that is very complex
- 0:40
that serves you that message despite the
- 0:42
model providers being down.
- 0:45
And that's what we're going to
- 0:46
productionize today or discuss
- 0:48
productionizing today.
- 0:50
So what is an LM gateway? An LLM gateway
- 0:53
is an entry point or a middleware
- 0:55
between your apps and the model
- 0:57
providers behind them. It does a bunch
- 1:00
of things. Routing, authentication,
- 1:02
fallback, rate limits, all kinds of
- 1:04
governance that you can think of.
- 1:08
And right at the heart of the gateway is
- 1:11
a fight between four things. It's
- 1:13
availability, latency, your guardrails
- 1:17
and costs.
- 1:18
In case of a degradation, you cannot
- 1:21
maximize all four. You need to pick what
- 1:24
you want. So with this talk, if you use
- 1:29
an LLM gateway, I want you I want to
- 1:32
help you to make that trade-off for your
- 1:34
use case. And if you design a gateway, I
- 1:37
want you to design or provide those
- 1:39
levers to your callers and customers u
- 1:43
so that your customers are happy.
- 1:46
Let's start with availability.
- 1:50
If you have a single model provider,
- 1:54
their ceiling is your ceiling. Their
- 1:57
outage is your outage.
- 2:03
So in typical software engineering, the
- 2:06
way you tackle unreliable dependency is
- 2:08
by retrying.
- 2:11
Retrying with exponential backoffs, with
- 2:14
jitters. And when all of that fails, you
- 2:17
have a circuit breaker that trips after
- 2:19
you've seen sufficient failures and you
- 2:21
stop calling the damn thing.
- 2:24
This is not enough for LLMs. LLMs are
- 2:27
very different compared to your fast
- 2:29
cheap APIs that you retry on. Retrying
- 2:33
an LLM API eats into your latency budget
- 2:37
really fast. And also tripping over a
- 2:40
circuit breaker when you have another
- 2:43
perfectly fine model provider to route
- 2:45
to doesn't make sense. You should use
- 2:47
the second model provider. And third, as
- 2:51
I said, the calls are slow and
- 2:53
expensive. So blind retries just
- 2:56
multiply your cost and your tail
- 2:57
latencies.
- 3:00
So what is a better idea here? It is
- 3:03
actually a per request fallback. What
- 3:06
that means is you can actually try model
- 3:08
provider A and then in sequence try
- 3:11
model provider B if your request to
- 3:13
model provider A fails. Another option
- 3:16
to consider here is you can fire
- 3:18
requests to both the providers in
- 3:19
parallel. But that's only if you're
- 3:21
highly highly obsessed with latencies
- 3:25
because that's just going to double your
- 3:26
cost.
- 3:28
Some of the similar circuit breaking
- 3:30
patterns apply here to LMS as well. If
- 3:34
you know that your primary has been
- 3:37
failing for some time, it doesn't make
- 3:39
sense to try it again. You put it, you
- 3:42
take it out of the load balancer or your
- 3:45
request path and put it in a cool down
- 3:48
and then after a few minutes have
- 3:50
passed, try putting that back again.
- 3:53
One interesting choice that you have to
- 3:56
make here is where your failure counts
- 3:58
live.
- 4:00
You can decide to have the failure
- 4:01
counts live in memory on the instances
- 4:04
that are serving your traffic or you can
- 4:07
have shared infra where your failure
- 4:11
counts are shared across the fleet.
- 4:13
There are trade-offs.
- 4:15
If you want quick failovers, then
- 4:18
fleetwide helps. And with instance uh
- 4:22
with local state counters the issue that
- 4:25
you run into is whenever you change your
- 4:26
deployment size your configuration and
- 4:29
your expectations change. So something
- 4:32
to consider.
- 4:35
What that clean diagram did not really
- 4:37
show you are some of the other gotchas
- 4:39
that I'm going to discuss. So fallbacks
- 4:41
are not transparent.
- 4:43
While the industry is converging on an
- 4:45
OpenAI API compatible format, I would
- 4:49
say there are still nuances. So you need
- 4:50
to really test your fallbacks well. They
- 4:53
can have differences in your tool
- 4:55
calling schemas, token limits, stop
- 4:57
reasons and what have you. So with LM
- 4:59
gateways, you can have a normalization
- 5:02
layer that can ensure that you can do
- 5:05
cross provider fallbacks as well.
- 5:08
Another thing is streaming
- 5:14
it.
- 5:16
So essentially nobody wants to wait for
- 5:20
30 seconds to have a wall of text appear
- 5:22
in front of them. So there are use cases
- 5:24
where streaming is absolutely required.
- 5:27
But it comes as at a cost. You trade
- 5:29
away your levers. You cannot once you
- 5:31
have decided to go with provider A, you
- 5:34
have to continue going with provider A.
- 5:37
You cannot mid-stream change the
- 5:39
providers. Whatever has been sent to the
- 5:42
client, it's done. And that's where the
- 5:45
something uh went wrong message, that's
- 5:47
the one that you see. It's not because
- 5:50
of laziness. It's by design uh that you
- 5:52
see that and it's one of the trade-offs.
- 5:55
I would like to call out one other thing
- 5:57
where I've seen teams trip over and over
- 6:00
again. They really provision and test
- 6:03
their primary providers really well, but
- 6:07
they the second provider, the fallback
- 6:09
provider doesn't necessarily get the
- 6:10
same level of love. And I would argue
- 6:13
that your throughputs or your capacity
- 6:15
or your headroom should be even higher
- 6:18
for the second provider or the fallback
- 6:21
provider because that's your last line
- 6:23
of defense. If that goes down, your
- 6:25
application goes down.
- 6:30
Let's discuss latencies.
- 6:32
Availability failures are right in your
- 6:34
face. They fail. You get alarmed. You
- 6:38
get paged. But high latencies can be the
- 6:42
quiet ones. And they need to receive
- 6:45
more love um than I would say tuning
- 6:47
your services for just availability.
- 6:54
One thing to call out, a gateway may run
- 6:58
mixed workloads
- 7:00
and you can have embedding embedding
- 7:02
requests that takes just less than a
- 7:04
second. You can have classification
- 7:06
requests that take less than a second.
- 7:08
Uh you have chat requests taking 3
- 7:10
seconds and reasoning requests taking a
- 7:13
long time.
- 7:15
Quick show of hands. If you measure
- 7:18
your aggregate latency for your entire
- 7:20
service.
- 7:22
Well, that was a trick question. Sorry.
- 7:24
You shouldn't. It doesn't make sense.
- 7:25
It's a lie. You should be tracking your
- 7:28
P99 per model per route, not a gateway
- 7:32
wide number. Gateway wide number doesn't
- 7:34
make sense, especially if you're running
- 7:36
mixed workloads. And I hope you're not u
- 7:38
for those who raise your hand. Another
- 7:41
thing that can really I cannot emphasize
- 7:44
this enough is for you to set timeouts
- 7:47
on per model class per route.
- 7:50
That's where that's the number one root
- 7:52
cause of your silent outage. If you
- 7:54
don't have a timeout, your gateway
- 7:57
thinks you're hap your request is being
- 7:58
happily served while it is not. And I'll
- 8:02
leave you with this message for for
- 8:03
latencies. Um, specifically a reasoning
- 8:07
models normal is actually a chat models
- 8:10
outage. So you definitely need to track
- 8:12
latency per route.
- 8:17
Okay, this is the most painful or this
- 8:19
the slide that has given me the most
- 8:20
scarse which is reasoning and router
- 8:24
models. So this is where truly the
- 8:28
latency is unpredictable
- 8:30
and reasoning models they do not give
- 8:34
you
- 8:36
they they're highly undeterministic more
- 8:38
deterministic undeterministic than your
- 8:40
normal models. You cannot set the
- 8:42
temperature to zero in many cases and
- 8:44
the same prompt can take somewhere from
- 8:47
2 seconds to 60 seconds and we've seen
- 8:49
that in production where P99 suddenly
- 8:51
popped to 60 seconds for no good reason.
- 8:54
So that's
- 8:56
while there's no magical solution to it.
- 8:59
I would recommend that you at least
- 9:01
start with fixing the reasoning level
- 9:03
per route. So with router models, they
- 9:07
hide that abstraction behind you. Like
- 9:09
they pick which models to run and I
- 9:12
would highly recommend that you at least
- 9:15
make as much uh you make requests as
- 9:18
determinist deterministic as possible
- 9:20
with an undeterministic system.
- 9:24
Another idea is hedging the tail. You
- 9:28
can have a you can fire another request
- 9:30
if your primary request actually
- 9:32
consumed let's say P90 of your latency
- 9:35
budget.
- 9:37
This can hedge the t this can really
- 9:39
hedge the P99 tail u for for your
- 9:43
services.
- 9:45
All right. This is one of my favorite
- 9:47
ones. Um
- 9:49
to keep your model secure you need to
- 9:52
have guardrails.
- 9:54
And with that, guardrails are necessary
- 9:57
for preventing your services from prompt
- 9:59
injection attacks, keeping PII filters
- 10:03
in place, having toxicity filters,
- 10:05
keeping the LMS to stop swearing at your
- 10:08
customers, all those good things. But
- 10:12
just like a model provider, there are
- 10:14
trade-offs, too. Guardrails are just
- 10:17
like another service that can go down
- 10:20
that can be unreliable and that's where
- 10:23
you need to choose do you fail open or
- 10:26
do you fail close when I say fail open
- 10:29
you can still serve the request even if
- 10:31
your guardrails are down fail close you
- 10:34
block the request and say hey I'm not
- 10:36
available that's the trade-off between
- 10:38
availability and security to certain
- 10:40
extent while there's a no universal
- 10:42
answer it really depends on your use
- 10:45
case you can decide like for example a
- 10:48
toxicity filter if it's not up and
- 10:50
running you can still serve that
- 10:52
request.
- 10:54
So the default choice should be the
- 10:57
worst case that you can live with.
- 11:03
There are a few things that you can
- 11:05
actually do to improve the behavior of
- 11:09
your systems in face of uh you know
- 11:12
guardrails being down and and managing
- 11:14
just unreliability of the guardrails
- 11:16
themselves. So the first is time budget.
- 11:21
Your request should never be bound by
- 11:24
your guardrail timing. It should always
- 11:27
be the LM that is the rate determining
- 11:29
step. So make sure that you have
- 11:32
timeouts in place and those guardrails
- 11:35
run with a specific time budget.
- 11:38
Another important thing is fallback.
- 11:41
You've heard, you probably know and I've
- 11:43
talked about it. We always discuss
- 11:45
fallbacks with regards to model
- 11:47
providers, but guardrails are critical
- 11:50
services too where you can consider
- 11:53
fallbacks, have secondary provider,
- 11:55
secondary checks, cache decisions uh to
- 11:58
keep your service available when a
- 12:01
guardrail provider is down.
- 12:04
Another interesting choice that pops up
- 12:06
with regards to guardrails is the
- 12:08
placement of the guardrails.
- 12:11
Typically, you can place the guardrail
- 12:14
in three ways. You can have a pre- hook
- 12:17
that runs where the guardrail actually
- 12:19
runs on the input. You can and that's
- 12:22
probably the safest uh but it does add
- 12:24
serial latency uh to your requests.
- 12:28
Another one is in parallel. This is one
- 12:30
of my favorites, but just to call out,
- 12:33
streaming wouldn't work well here with
- 12:35
with parallel. So if you're specially
- 12:38
producing structured output, please
- 12:39
don't stream them. Uh try to save your
- 12:41
latencies and run run these guardrails
- 12:44
concurrently for your structured
- 12:45
outputs. Another one is post hooks. The
- 12:48
these are best for um output monitoring,
- 12:52
auditing your outputs and and so forth.
- 12:58
So, so far we've all I've discussed all
- 13:02
the things that can go wrong with
- 13:04
regards to our dependencies.
- 13:07
We haven't discussed that we are
- 13:09
actually adding another dependency in
- 13:11
the request path itself which is the
- 13:13
central or which is the LM gateway
- 13:15
itself. There are a few things where we
- 13:17
have been bitten by u and we've learned
- 13:19
some lessons that I want to share with
- 13:21
you. If you're working on an LLM gateway
- 13:24
or using one, one is shared limits.
- 13:29
Make sure that your API keys are
- 13:31
segregated per route, per use case to
- 13:35
the most granular possible uh to the
- 13:39
most granular thing that you can
- 13:40
imagine. U
- 13:44
having a noisy tenant can be one of the
- 13:47
biggest problems here.
- 13:49
Another thing is load shedding. This is
- 13:52
a feature that you should uh as part of
- 13:54
your runbooks, game days, uh make sure
- 13:56
that the gateway that you're using
- 13:58
supports load shedding because when you
- 14:01
have a retry storm, it becomes really
- 14:03
hard to just scale out. You cannot
- 14:05
simply scale out services that is under
- 14:07
a retry storm and all these web servers
- 14:11
they have an internal queue and they're
- 14:13
configurable. Make sure that they're
- 14:15
bounded and they cannot request they
- 14:18
cannot accept requests that are
- 14:19
unbounded. And if you want to have some
- 14:21
custom logic, you can even have traffic
- 14:24
prioritization here as well to make sure
- 14:26
under load your most important use cases
- 14:29
get served. Well,
- 14:34
last thing that I wanted to discuss is
- 14:37
the whole idea of a central gateway
- 14:38
itself. It is a single point of failure.
- 14:41
So if you're thinking of having a
- 14:43
central gateway for your entire company
- 14:45
for to LLMs, I would recommend rethink
- 14:49
that and see what are the reasons that
- 14:51
you want it. What I've noticed is that
- 14:54
in most scenarios, it's not the central
- 14:56
gateway that they want. They want
- 14:58
centralized governance.
- 15:00
And there is a path forward where you
- 15:02
can actually decentralize the gateway
- 15:05
and still centralize government
- 15:07
governance. So do not try to centralize
- 15:11
your traffic but you can have plugins,
- 15:14
you can have custom code that can
- 15:16
centralize your governance. Uh
- 15:18
governance can be in the form of cost
- 15:20
tracking, rate limit managing management
- 15:23
and there are other solutions possible.
- 15:25
So explore those before you chart on
- 15:28
having one central gateway for your
- 15:30
entire company. It can be managed by a
- 15:33
single team, but I wouldn't recommend
- 15:35
deploying it as a single deployment for
- 15:39
the entire company even though it's
- 15:41
distributed.
- 15:43
With that said, I want to end this talk
- 15:46
on a personal note. So, it is my son's
- 15:49
birthday today and I'm here talking to
- 15:52
strangers about circuit breaking. So the
- 15:56
least you can do for me is please go and
- 15:58
prevent one incident for me and for your
- 16:01
customers. Thank you. If you have any
- 16:04
questions. Yeah.
- 16:21
>> [music]