AI Engineer World's Fair 2026

Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

Read the talk

Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons

Kanish Manuja explains how provider fallback, streaming, route-specific timeouts and guardrail placement change an LLM gateway’s failure behavior—and why centralized governance need not mean one company-wide traffic path.

From a talk by Kanish Manuja

At a glance

Ideas worth remembering

  • Prefer per-request provider fallback to blind retries, while using cooldown to keep a repeatedly failing primary out of later request paths.

  • Streaming commits delivered output to the selected provider. A mid-stream failure cannot be recovered through a transparent provider switch; provision the backup thoroughly for requests that can fail over.

  • Measure P99 per model and route, set corresponding timeouts and constrain available reasoning settings. Tail hedging launches another request after waiting for a slow primary.

  • Treat guardrails as fallible services: choose fail-open or fail-closed behavior, bound their runtime, consider fallback checks and place them deliberately relative to generation.

  • Protect the gateway with granular API keys, bounded queues, load shedding and traffic priorities. Shared governance can coexist with separate gateway deployments.

The system behind “Something went wrong”

“Something went wrong. Please try again” is a familiar ending to an AI interaction. Kanish Manuja, a principal engineer at Twilio, opens with that message because the system behind it can be much more carefully engineered than the message suggests. A gateway may route around provider failures and still reach a point where it cannot recover a response already underway.

An LLM gateway sits between applications and model providers. It handles routing, authentication, fallback, rate limits and governance. Those responsibilities bring four goals into the same request path: availability, latency, guardrails and cost. During degradation, improving one can worsen another. Sending a second model request may preserve availability while adding latency and expense; continuing without a failed guardrail may preserve service while weakening protection.

The gateway needs controls that callers can choose for their use cases. One recovery policy cannot express every application’s willingness to wait, spend or accept a weaker check. Productionizing the gateway means making those choices available before an incident forces them.

0:130:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:13 · section reference included

Use the healthy provider before exhausting the unhealthy one

With one model provider, its availability limits the application’s availability: its outage becomes your outage. Conventional dependency handling starts with retries, exponential backoff and jitter, then opens a circuit breaker after enough failures. For a fast, inexpensive API, another attempt can absorb a transient fault. A slow, expensive model call consumes the request’s remaining time and spend much faster.

The basic recovery example uses providers A and B. The gateway sends a request to A. If A fails, it sends that request to B. The observable change is that A’s failure no longer immediately becomes an application failure: another provider gets a chance to answer. This per-request fallback avoids spending the entire recovery effort repeatedly calling A while B remains healthy. 3:00

Two dispatch strategies expose different tradeoffs:

  • Sequential fallback: Try A, then B after A fails. The request pays for the time spent on A before B can begin.
  • Parallel requests: Start both providers together when latency matters enough to justify duplicate work. Manuja describes this as doubling cost: both model calls run even when the primary might have succeeded.

Circuit breaking still has a role. Once A has been failing for some time, remove it from the request path, let it cool down and try reintroducing it after a few minutes. Per-request fallback handles the current failure; cooldown keeps later requests from repeatedly discovering the same unhealthy dependency.

Where should the failure counts live?

  • Instance-local counters: Each serving instance keeps its own history in memory. Changing the deployment size changes how failures accumulate across instances, so the same configuration can produce different failover behavior.
  • Fleet-wide counters: Shared infrastructure collects failure history across the deployment. This can help the fleet recognize an outage and fail over quickly.
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Fallback stops being transparent when output escapes

A successful HTTP exchange with B is only part of a successful fallback. Providers can differ in tool-calling schemas, token limits and stop reasons even when they expose an OpenAI-compatible API format. A normalization layer can reconcile those differences, but the fallback path still needs testing against the application’s actual expectations. Similar request shapes do not guarantee interchangeable behavior.

Now change the A-to-B example by enabling streaming. A begins generating, and the gateway forwards its output to the client. The user sees progress instead of waiting thirty seconds for a wall of text. If A fails after that output arrives, the gateway cannot transparently replace the response with B’s answer: the client already has part of A’s response, and those tokens cannot be recalled. In this streaming design, the continuing response is committed to A. 5:14

Where does the recovery path change? The diagram separates a failure before output reaches the client from one after streaming begins. The first path can move to B; the second leaves an interrupted response because switching providers cannot undo the text already delivered.

That is the mechanism behind the opening error message. Streaming improves perceived speed by giving away a recovery lever. The tradeoff can be worthwhile—some use cases require streaming—but it must be part of the application’s failure design.

B also needs enough capacity to be a real last line of defense. Teams often provision and test the primary thoroughly while giving the backup less attention. Manuja recommends even more throughput, capacity or headroom for the fallback provider. Once the primary is unavailable, an inadequate backup can turn a recoverable provider failure into an application outage.

How it fits togetherThe same provider failure has two recovery paths

The primary provider starts handling the request.

Before delivery, fallback can give B a chance to answer. After A’s output reaches the client, a transparent switch cannot retract it.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:35 · section reference included

A reasoning model’s normal can be a chat model’s outage

Availability failures announce themselves with errors and pages. Latency failures can stay quiet: a request remains in progress while the customer waits. Mixed workloads make that problem harder to recognize. Manuja’s examples include embedding and classification requests taking less than a second, chat requests taking three seconds and reasoning requests taking much longer. These illustrate different workload expectations rather than universal performance targets.

A gateway-wide latency number blends those expectations. It cannot tell you whether a chat route has become unusually slow or whether the gateway simply served more long-running reasoning requests. Track P99 per model, per route instead. P99 describes the slow tail of a particular request population; choosing the population is as important as choosing the percentile. As Manuja puts it, “A reasoning model’s normal is actually a chat model’s outage.” 7:24

Timeouts need the same specificity: set them per model class and route. Without a timeout, the gateway can continue treating an outstanding request as healthy even when it is no longer serving the application usefully. Manuja identifies missing timeouts as his leading cause of silent outages. A route-specific timeout gives the gateway a point at which waiting becomes failure and recovery can begin.

Reasoning and router models add another source of variation. Manuja reports production behavior where the same prompt could take two to sixty seconds and P99 suddenly rose to sixty seconds without an identified explanation. That experience concerns the systems he encountered, rather than a universal range for reasoning models. His first recommendation is to fix the reasoning level per route. Some controls may be unavailable: in many cases, he notes, temperature cannot be set to zero. Router models also choose underlying models behind an abstraction, so constrain the request settings you can control to avoid adding unnecessary variation.

Tail hedging offers a different response to a slow primary. Start the primary normally, then launch another request after the primary has spent enough time outstanding to trigger a delay threshold. Unlike starting two requests immediately, this duplicates work only after waiting. Unlike failure-triggered fallback, it acts while the primary is still running. The aim is to reduce the P99 tail, while accepting the cost of additional model work.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:30 · section reference included

Guardrails need failure policies and their own time budgets

Guardrails check for prompt injection, personally identifiable information and toxic output—including the practical problem of a model swearing at customers. Those checks add services to the system, and those services can fail too. Their failure forces an explicit choice:

  • Fail open: Serve the request despite the unavailable check. Availability continues with reduced protection.
  • Fail closed: Block the request because the check cannot run. Protection takes priority over serving it. 10:15

There is no universal answer across checks and applications. A toxicity-filter outage might be tolerable for one use case, allowing requests to continue; another use case may require blocking them. Choose the worst case you can live with as the default. That decision concerns the consequence of an unavailable check, rather than an assumption that the check will always be available.

Give each guardrail a specific time budget and timeout so it cannot become the component that determines how long the entire request takes. Manuja’s intended design keeps the LLM as the rate-determining step. Guardrails can also have fallback arrangements: secondary providers, secondary checks or cached decisions can preserve service when the primary guardrail provider is down.

Placement changes both latency and what the check can inspect:

  • Pre-hook: Check the input before the model runs. Manuja considers this probably the safest placement, but its runtime adds serial latency before generation.
  • Parallel check: Run the guardrail concurrently with model work. This overlaps their runtimes and is his preferred latency-saving option for structured output, with a recommendation to avoid streaming that output.
  • Post-hook: Inspect output after generation. This placement suits output monitoring and auditing.

Why does parallel placement save time, and why is streaming awkward? The diagram shows the input splitting into two concurrent paths. The check no longer has to finish before model work starts. But streaming can let delivery get ahead of the checking decision—the same irreversible delivery problem that limited provider fallback.

How it fits togetherParallel guardrails overlap checking with generation

Starts both paths in the parallel placement.

Concurrent work removes the serial wait before generation. Streaming can expose output while the guardrail decision is still pending.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:45 · section reference included

The gateway becomes another dependency

Routing around provider and guardrail failures does not remove the gateway’s own failure modes. It is another dependency in the request path. Shared limits are one way it can spread trouble: a noisy tenant can consume capacity needed by unrelated requests. Segregate API keys as granularly as possible by route and use case so those workloads do not unnecessarily share the same limits.

A retry storm creates a different problem. Repeated attempts pile more work onto an already stressed service, and scaling out alone is insufficient as a recovery strategy. The gateway needs load shedding: the ability to stop accepting more work under overload. Its web-server queues must be bounded, rather than letting requests accumulate without a limit. 13:44

Traffic prioritization can determine which work survives that pressure. Under load, the most important use cases should continue receiving service rather than competing indiscriminately with every retry. Put load shedding into runbooks and game days so operators know the gateway supports it and can use it during an incident.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:59 · section reference included

Centralize governance without concentrating all traffic

The final architectural question is whether the entire company needs one central gateway deployment at all. A common request path can become a single point of failure. Before putting every application behind it, identify the requirement that motivated the design. Manuja’s experience is that teams often want centralized governance: consistent cost tracking, rate-limit management and related controls. 14:34

Those controls can live in plugins or shared custom code while gateways remain decentralized. One team can manage the gateway system without operating one company-wide deployment. Even a deployment distributed across instances can remain a shared dependency for every caller; distributing instances does not by itself separate the applications’ request paths. Explore separate deployments with shared governance before making all company traffic depend on the same gateway deployment.

The ending brings those decisions back to the customer. Manuja asks the audience to “prevent one incident.” The next step is small enough to act on: find one route without a timeout, one backup without enough headroom or one gateway queue that can grow without a bound, and fix the failure before customers encounter it.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:34 · section reference included

Resources

  • A separate account of a restaurant order-taking prototype comparing a speech-to-text → LLM → text-to-speech pipeline with an end-to-end realtime audio approach. Useful for extending the talk’s latency, cost and control tradeoffs to voice applications; its conclusions reflect that prototype and the models used then.

Read the complete timestamped transcript
  1. 0:01

    [music]

  2. 0:13

    I'm Kanesh Manuja. I'm a principal

  3. 0:15

    engineer at Twilio.

  4. 0:18

    Let's start with a quick show of hands.

  5. 0:20

    Who here has seen the message, something

  6. 0:23

    went wrong. Please try again.

  7. 0:27

    Well, we have a few lucky ones and a few

  8. 0:30

    that have had a good lunch. Um, so

  9. 0:34

    behind that simple message is actually a

  10. 0:37

    system that is very complex

  11. 0:40

    that serves you that message despite the

  12. 0:42

    model providers being down.

  13. 0:45

    And that's what we're going to

  14. 0:46

    productionize today or discuss

  15. 0:48

    productionizing today.

  16. 0:50

    So what is an LM gateway? An LLM gateway

  17. 0:53

    is an entry point or a middleware

  18. 0:55

    between your apps and the model

  19. 0:57

    providers behind them. It does a bunch

  20. 1:00

    of things. Routing, authentication,

  21. 1:02

    fallback, rate limits, all kinds of

  22. 1:04

    governance that you can think of.

  23. 1:08

    And right at the heart of the gateway is

  24. 1:11

    a fight between four things. It's

  25. 1:13

    availability, latency, your guardrails

  26. 1:17

    and costs.

  27. 1:18

    In case of a degradation, you cannot

  28. 1:21

    maximize all four. You need to pick what

  29. 1:24

    you want. So with this talk, if you use

  30. 1:29

    an LLM gateway, I want you I want to

  31. 1:32

    help you to make that trade-off for your

  32. 1:34

    use case. And if you design a gateway, I

  33. 1:37

    want you to design or provide those

  34. 1:39

    levers to your callers and customers u

  35. 1:43

    so that your customers are happy.

  36. 1:46

    Let's start with availability.

  37. 1:50

    If you have a single model provider,

  38. 1:54

    their ceiling is your ceiling. Their

  39. 1:57

    outage is your outage.

  40. 2:03

    So in typical software engineering, the

  41. 2:06

    way you tackle unreliable dependency is

  42. 2:08

    by retrying.

  43. 2:11

    Retrying with exponential backoffs, with

  44. 2:14

    jitters. And when all of that fails, you

  45. 2:17

    have a circuit breaker that trips after

  46. 2:19

    you've seen sufficient failures and you

  47. 2:21

    stop calling the damn thing.

  48. 2:24

    This is not enough for LLMs. LLMs are

  49. 2:27

    very different compared to your fast

  50. 2:29

    cheap APIs that you retry on. Retrying

  51. 2:33

    an LLM API eats into your latency budget

  52. 2:37

    really fast. And also tripping over a

  53. 2:40

    circuit breaker when you have another

  54. 2:43

    perfectly fine model provider to route

  55. 2:45

    to doesn't make sense. You should use

  56. 2:47

    the second model provider. And third, as

  57. 2:51

    I said, the calls are slow and

  58. 2:53

    expensive. So blind retries just

  59. 2:56

    multiply your cost and your tail

  60. 2:57

    latencies.

  61. 3:00

    So what is a better idea here? It is

  62. 3:03

    actually a per request fallback. What

  63. 3:06

    that means is you can actually try model

  64. 3:08

    provider A and then in sequence try

  65. 3:11

    model provider B if your request to

  66. 3:13

    model provider A fails. Another option

  67. 3:16

    to consider here is you can fire

  68. 3:18

    requests to both the providers in

  69. 3:19

    parallel. But that's only if you're

  70. 3:21

    highly highly obsessed with latencies

  71. 3:25

    because that's just going to double your

  72. 3:26

    cost.

  73. 3:28

    Some of the similar circuit breaking

  74. 3:30

    patterns apply here to LMS as well. If

  75. 3:34

    you know that your primary has been

  76. 3:37

    failing for some time, it doesn't make

  77. 3:39

    sense to try it again. You put it, you

  78. 3:42

    take it out of the load balancer or your

  79. 3:45

    request path and put it in a cool down

  80. 3:48

    and then after a few minutes have

  81. 3:50

    passed, try putting that back again.

  82. 3:53

    One interesting choice that you have to

  83. 3:56

    make here is where your failure counts

  84. 3:58

    live.

  85. 4:00

    You can decide to have the failure

  86. 4:01

    counts live in memory on the instances

  87. 4:04

    that are serving your traffic or you can

  88. 4:07

    have shared infra where your failure

  89. 4:11

    counts are shared across the fleet.

  90. 4:13

    There are trade-offs.

  91. 4:15

    If you want quick failovers, then

  92. 4:18

    fleetwide helps. And with instance uh

  93. 4:22

    with local state counters the issue that

  94. 4:25

    you run into is whenever you change your

  95. 4:26

    deployment size your configuration and

  96. 4:29

    your expectations change. So something

  97. 4:32

    to consider.

  98. 4:35

    What that clean diagram did not really

  99. 4:37

    show you are some of the other gotchas

  100. 4:39

    that I'm going to discuss. So fallbacks

  101. 4:41

    are not transparent.

  102. 4:43

    While the industry is converging on an

  103. 4:45

    OpenAI API compatible format, I would

  104. 4:49

    say there are still nuances. So you need

  105. 4:50

    to really test your fallbacks well. They

  106. 4:53

    can have differences in your tool

  107. 4:55

    calling schemas, token limits, stop

  108. 4:57

    reasons and what have you. So with LM

  109. 4:59

    gateways, you can have a normalization

  110. 5:02

    layer that can ensure that you can do

  111. 5:05

    cross provider fallbacks as well.

  112. 5:08

    Another thing is streaming

  113. 5:14

    it.

  114. 5:16

    So essentially nobody wants to wait for

  115. 5:20

    30 seconds to have a wall of text appear

  116. 5:22

    in front of them. So there are use cases

  117. 5:24

    where streaming is absolutely required.

  118. 5:27

    But it comes as at a cost. You trade

  119. 5:29

    away your levers. You cannot once you

  120. 5:31

    have decided to go with provider A, you

  121. 5:34

    have to continue going with provider A.

  122. 5:37

    You cannot mid-stream change the

  123. 5:39

    providers. Whatever has been sent to the

  124. 5:42

    client, it's done. And that's where the

  125. 5:45

    something uh went wrong message, that's

  126. 5:47

    the one that you see. It's not because

  127. 5:50

    of laziness. It's by design uh that you

  128. 5:52

    see that and it's one of the trade-offs.

  129. 5:55

    I would like to call out one other thing

  130. 5:57

    where I've seen teams trip over and over

  131. 6:00

    again. They really provision and test

  132. 6:03

    their primary providers really well, but

  133. 6:07

    they the second provider, the fallback

  134. 6:09

    provider doesn't necessarily get the

  135. 6:10

    same level of love. And I would argue

  136. 6:13

    that your throughputs or your capacity

  137. 6:15

    or your headroom should be even higher

  138. 6:18

    for the second provider or the fallback

  139. 6:21

    provider because that's your last line

  140. 6:23

    of defense. If that goes down, your

  141. 6:25

    application goes down.

  142. 6:30

    Let's discuss latencies.

  143. 6:32

    Availability failures are right in your

  144. 6:34

    face. They fail. You get alarmed. You

  145. 6:38

    get paged. But high latencies can be the

  146. 6:42

    quiet ones. And they need to receive

  147. 6:45

    more love um than I would say tuning

  148. 6:47

    your services for just availability.

  149. 6:54

    One thing to call out, a gateway may run

  150. 6:58

    mixed workloads

  151. 7:00

    and you can have embedding embedding

  152. 7:02

    requests that takes just less than a

  153. 7:04

    second. You can have classification

  154. 7:06

    requests that take less than a second.

  155. 7:08

    Uh you have chat requests taking 3

  156. 7:10

    seconds and reasoning requests taking a

  157. 7:13

    long time.

  158. 7:15

    Quick show of hands. If you measure

  159. 7:18

    your aggregate latency for your entire

  160. 7:20

    service.

  161. 7:22

    Well, that was a trick question. Sorry.

  162. 7:24

    You shouldn't. It doesn't make sense.

  163. 7:25

    It's a lie. You should be tracking your

  164. 7:28

    P99 per model per route, not a gateway

  165. 7:32

    wide number. Gateway wide number doesn't

  166. 7:34

    make sense, especially if you're running

  167. 7:36

    mixed workloads. And I hope you're not u

  168. 7:38

    for those who raise your hand. Another

  169. 7:41

    thing that can really I cannot emphasize

  170. 7:44

    this enough is for you to set timeouts

  171. 7:47

    on per model class per route.

  172. 7:50

    That's where that's the number one root

  173. 7:52

    cause of your silent outage. If you

  174. 7:54

    don't have a timeout, your gateway

  175. 7:57

    thinks you're hap your request is being

  176. 7:58

    happily served while it is not. And I'll

  177. 8:02

    leave you with this message for for

  178. 8:03

    latencies. Um, specifically a reasoning

  179. 8:07

    models normal is actually a chat models

  180. 8:10

    outage. So you definitely need to track

  181. 8:12

    latency per route.

  182. 8:17

    Okay, this is the most painful or this

  183. 8:19

    the slide that has given me the most

  184. 8:20

    scarse which is reasoning and router

  185. 8:24

    models. So this is where truly the

  186. 8:28

    latency is unpredictable

  187. 8:30

    and reasoning models they do not give

  188. 8:34

    you

  189. 8:36

    they they're highly undeterministic more

  190. 8:38

    deterministic undeterministic than your

  191. 8:40

    normal models. You cannot set the

  192. 8:42

    temperature to zero in many cases and

  193. 8:44

    the same prompt can take somewhere from

  194. 8:47

    2 seconds to 60 seconds and we've seen

  195. 8:49

    that in production where P99 suddenly

  196. 8:51

    popped to 60 seconds for no good reason.

  197. 8:54

    So that's

  198. 8:56

    while there's no magical solution to it.

  199. 8:59

    I would recommend that you at least

  200. 9:01

    start with fixing the reasoning level

  201. 9:03

    per route. So with router models, they

  202. 9:07

    hide that abstraction behind you. Like

  203. 9:09

    they pick which models to run and I

  204. 9:12

    would highly recommend that you at least

  205. 9:15

    make as much uh you make requests as

  206. 9:18

    determinist deterministic as possible

  207. 9:20

    with an undeterministic system.

  208. 9:24

    Another idea is hedging the tail. You

  209. 9:28

    can have a you can fire another request

  210. 9:30

    if your primary request actually

  211. 9:32

    consumed let's say P90 of your latency

  212. 9:35

    budget.

  213. 9:37

    This can hedge the t this can really

  214. 9:39

    hedge the P99 tail u for for your

  215. 9:43

    services.

  216. 9:45

    All right. This is one of my favorite

  217. 9:47

    ones. Um

  218. 9:49

    to keep your model secure you need to

  219. 9:52

    have guardrails.

  220. 9:54

    And with that, guardrails are necessary

  221. 9:57

    for preventing your services from prompt

  222. 9:59

    injection attacks, keeping PII filters

  223. 10:03

    in place, having toxicity filters,

  224. 10:05

    keeping the LMS to stop swearing at your

  225. 10:08

    customers, all those good things. But

  226. 10:12

    just like a model provider, there are

  227. 10:14

    trade-offs, too. Guardrails are just

  228. 10:17

    like another service that can go down

  229. 10:20

    that can be unreliable and that's where

  230. 10:23

    you need to choose do you fail open or

  231. 10:26

    do you fail close when I say fail open

  232. 10:29

    you can still serve the request even if

  233. 10:31

    your guardrails are down fail close you

  234. 10:34

    block the request and say hey I'm not

  235. 10:36

    available that's the trade-off between

  236. 10:38

    availability and security to certain

  237. 10:40

    extent while there's a no universal

  238. 10:42

    answer it really depends on your use

  239. 10:45

    case you can decide like for example a

  240. 10:48

    toxicity filter if it's not up and

  241. 10:50

    running you can still serve that

  242. 10:52

    request.

  243. 10:54

    So the default choice should be the

  244. 10:57

    worst case that you can live with.

  245. 11:03

    There are a few things that you can

  246. 11:05

    actually do to improve the behavior of

  247. 11:09

    your systems in face of uh you know

  248. 11:12

    guardrails being down and and managing

  249. 11:14

    just unreliability of the guardrails

  250. 11:16

    themselves. So the first is time budget.

  251. 11:21

    Your request should never be bound by

  252. 11:24

    your guardrail timing. It should always

  253. 11:27

    be the LM that is the rate determining

  254. 11:29

    step. So make sure that you have

  255. 11:32

    timeouts in place and those guardrails

  256. 11:35

    run with a specific time budget.

  257. 11:38

    Another important thing is fallback.

  258. 11:41

    You've heard, you probably know and I've

  259. 11:43

    talked about it. We always discuss

  260. 11:45

    fallbacks with regards to model

  261. 11:47

    providers, but guardrails are critical

  262. 11:50

    services too where you can consider

  263. 11:53

    fallbacks, have secondary provider,

  264. 11:55

    secondary checks, cache decisions uh to

  265. 11:58

    keep your service available when a

  266. 12:01

    guardrail provider is down.

  267. 12:04

    Another interesting choice that pops up

  268. 12:06

    with regards to guardrails is the

  269. 12:08

    placement of the guardrails.

  270. 12:11

    Typically, you can place the guardrail

  271. 12:14

    in three ways. You can have a pre- hook

  272. 12:17

    that runs where the guardrail actually

  273. 12:19

    runs on the input. You can and that's

  274. 12:22

    probably the safest uh but it does add

  275. 12:24

    serial latency uh to your requests.

  276. 12:28

    Another one is in parallel. This is one

  277. 12:30

    of my favorites, but just to call out,

  278. 12:33

    streaming wouldn't work well here with

  279. 12:35

    with parallel. So if you're specially

  280. 12:38

    producing structured output, please

  281. 12:39

    don't stream them. Uh try to save your

  282. 12:41

    latencies and run run these guardrails

  283. 12:44

    concurrently for your structured

  284. 12:45

    outputs. Another one is post hooks. The

  285. 12:48

    these are best for um output monitoring,

  286. 12:52

    auditing your outputs and and so forth.

  287. 12:58

    So, so far we've all I've discussed all

  288. 13:02

    the things that can go wrong with

  289. 13:04

    regards to our dependencies.

  290. 13:07

    We haven't discussed that we are

  291. 13:09

    actually adding another dependency in

  292. 13:11

    the request path itself which is the

  293. 13:13

    central or which is the LM gateway

  294. 13:15

    itself. There are a few things where we

  295. 13:17

    have been bitten by u and we've learned

  296. 13:19

    some lessons that I want to share with

  297. 13:21

    you. If you're working on an LLM gateway

  298. 13:24

    or using one, one is shared limits.

  299. 13:29

    Make sure that your API keys are

  300. 13:31

    segregated per route, per use case to

  301. 13:35

    the most granular possible uh to the

  302. 13:39

    most granular thing that you can

  303. 13:40

    imagine. U

  304. 13:44

    having a noisy tenant can be one of the

  305. 13:47

    biggest problems here.

  306. 13:49

    Another thing is load shedding. This is

  307. 13:52

    a feature that you should uh as part of

  308. 13:54

    your runbooks, game days, uh make sure

  309. 13:56

    that the gateway that you're using

  310. 13:58

    supports load shedding because when you

  311. 14:01

    have a retry storm, it becomes really

  312. 14:03

    hard to just scale out. You cannot

  313. 14:05

    simply scale out services that is under

  314. 14:07

    a retry storm and all these web servers

  315. 14:11

    they have an internal queue and they're

  316. 14:13

    configurable. Make sure that they're

  317. 14:15

    bounded and they cannot request they

  318. 14:18

    cannot accept requests that are

  319. 14:19

    unbounded. And if you want to have some

  320. 14:21

    custom logic, you can even have traffic

  321. 14:24

    prioritization here as well to make sure

  322. 14:26

    under load your most important use cases

  323. 14:29

    get served. Well,

  324. 14:34

    last thing that I wanted to discuss is

  325. 14:37

    the whole idea of a central gateway

  326. 14:38

    itself. It is a single point of failure.

  327. 14:41

    So if you're thinking of having a

  328. 14:43

    central gateway for your entire company

  329. 14:45

    for to LLMs, I would recommend rethink

  330. 14:49

    that and see what are the reasons that

  331. 14:51

    you want it. What I've noticed is that

  332. 14:54

    in most scenarios, it's not the central

  333. 14:56

    gateway that they want. They want

  334. 14:58

    centralized governance.

  335. 15:00

    And there is a path forward where you

  336. 15:02

    can actually decentralize the gateway

  337. 15:05

    and still centralize government

  338. 15:07

    governance. So do not try to centralize

  339. 15:11

    your traffic but you can have plugins,

  340. 15:14

    you can have custom code that can

  341. 15:16

    centralize your governance. Uh

  342. 15:18

    governance can be in the form of cost

  343. 15:20

    tracking, rate limit managing management

  344. 15:23

    and there are other solutions possible.

  345. 15:25

    So explore those before you chart on

  346. 15:28

    having one central gateway for your

  347. 15:30

    entire company. It can be managed by a

  348. 15:33

    single team, but I wouldn't recommend

  349. 15:35

    deploying it as a single deployment for

  350. 15:39

    the entire company even though it's

  351. 15:41

    distributed.

  352. 15:43

    With that said, I want to end this talk

  353. 15:46

    on a personal note. So, it is my son's

  354. 15:49

    birthday today and I'm here talking to

  355. 15:52

    strangers about circuit breaking. So the

  356. 15:56

    least you can do for me is please go and

  357. 15:58

    prevent one incident for me and for your

  358. 16:01

    customers. Thank you. If you have any

  359. 16:04

    questions. Yeah.

  360. 16:21

    >> [music]