The Half Life of Agent Infrastructure — Ben Kus, Box

[music]

Hi everyone, I'm Ben Kuss. I'm CTO of

Box and today I'm going to be talking

about uh building for change and

specifically around uh AI agents and how

to continue to adapt uh your

infrastructure as we are all in the

middle of this journey.

Um, so before I get too far, I will

quickly sort of set a little bit of like

who I am and uh sort of what I do. Um,

for Box, I'm CTO and one of my jobs in

my job for my whole career has been to

build enterprise software. And so if

today um you're from a consumer company

or you're uh not involved in enterprise,

I hope that a lot of it is still

relevant. But in many cases um a lot of

the lessons I've learned are enterprise

uh specific.

So um I sort of will highlight um when

I'm thinking and talking about

infrastructure when I'm talking about uh

the kind of challenges that we face um

I'm typically talking about things that

are sort of in a scale of like uh like

for box we have over an exabyte of data

not a gigabyte not a terabyte not a

pabyte but an exabyte um and then

oftentimes we're I'm thinking in the

tens of of millions of users uh the

hundreds of billions of things in our

case files or content or unstructured

content um and then the new stat that is

sort of the the one that we talk about

is uh tokens. Um so we are now in the

ballpark of trillion tokens probably

will be 10 trillion tokens sometime

soon. Um and this of course is uh uh

some of the new and interesting

challenges that this kind of scale

brings.

Um so in my career and um I think maybe

many of us here um we've kind of lived

through these this these technology

changes. And so taking a quick step back

um I started uh my career when the

internet was sort of becoming a thing uh

lived through mobile and sort of this

idea of like you know carrying these

different devices move to the cloud

where you could kind of store and

maintain all your data and then of

course we're all in the middle of this

AI change and I think when you see these

kind of technology disruptions when

you're sort of thinking about this idea

of like all of these kind of have

changed all of our lives um and then uh

and you're thinking about it from the

perspective of a technology leader or a

startup or a engineer. Um, you kind of

see that like these are where like major

companies are born. Um, you see that

like big companies adapt or die. Um,

small companies are here to disrupt

things. They're here to take bets.

They're here to grow. Um, I've had two

startups. I've been acquired twice. Once

in IBM, once in a box. Um, and um, so

that we're in the middle of this kind of

major opportunity.

Um but for um a long tu uh no matter

what you kind of come from and what area

you're at I uh t typically if you were

to ask my advice um and about kind of

what makes it you successful as an

company as an engineering organization

as a technology startup or as a like a

company who has a a technology division

um I would say no matter what there's

kind of three things the first is you

need to build scalable reliable uh

platforms and select the technology that

you care about. Meaning that like

there's a lot of ways to do things, but

get good at something. Get good at that

technology, at that system, at that

stack, and then and then keep going with

that. Um, and then you leverage this

technology so that you do more for your

customers. Um, you build your better

product, you develop the capabilities,

and then you optimize it, make it

better, make it faster, make it cheaper,

make it uh more capable. And this was

sort of the generic enterprise advice,

uh, the generic energy advice that many

many people would follow. And I think

this works really well

except now

I don't know if this is good advice. It

has been across all these major

disruptive changes over time. Unclear.

In fact, I don't think it's good advice

right now because there's a funny thing

happening right now which that didn't

happen in those previous trends which is

that the rate of change is dramatically

higher. Um I and you might say look

technology always is changing like you

know there's other trends things change

a lot but not that much um the internet

is still based on HTTP the uh uh mobile

devices are still um iOS and Android

based and so on and so but nowadays um

other than the fact that like uh

generative AI exists most things that

power it are changing and changing

dramatically.

So last year I was here at this uh at

the AI engineering world fair and I gave

a speech and I said uh after spending a

lot of time on this and thinking through

this I think there's a key to this which

is a gentic uh uh graph-based approach.

The idea was um uh you have a large

language model and these nodes and then

you sort of put them together and you

have the sort of the AI traverse this

graph that you set up you build the

graph. This is the key approach that and

if you use this this is going to really

help you sort of build agents because

what is anything that we do in life it's

a agent it's a workflow um uh it's a

it's a way and then if you have an

intelligent uh uh a agent it can

basically traverse this is the key

approach and I believe that at the time

and a lot of people did and I still love

this approach but nowadays it's sort of

a little bit out of date in fact I

remember um uh a guy came up to me after

my speech last time and he was like the

problem you talked about the the answer

is just this exactly you were speaking

to me thank you so much and and I was

happy I I you know I gave a good speech

and gave somebody some good advice um

and then I remember when we made a

change I was like I wonder what happened

to that guy I wonder if he's here uh

it's uh uh so um but the problem is is

that not that it was wrong that that was

the best approach but a new way emerged

um in fact I started to like look

through like all of the last year's uh

events and actually go to other

conferences like what what did people

talk about a year ago and most of them

were again nothing much wrong all good

speeches all all good ideas but most of

them have now a better way there um and

uh so um and this is be sort of the gist

of of the challenge so if you look at

our journey of the technologies the kind

of things that we care about um I'll

just kind of rapid file here like so

let's say that you want to utilize AI

models and let's just look the last

couple years a long a while ago probably

distant memory now like people would say

train your own my models or maybe fine

tune them like nah that doesn't that's

too why bother just use a frontier model

use something from openi use something

from anthropic use something from from

Gemini um and then that's great but it's

kind of expensive okay great let's just

use openweight models they're pretty

close you can use them you can host them

yourself you can get some good GPUs um

but then uh some companies will come to

you and they'll be like look we just did

this big deal with open orthropic like

can we use our own key or bring our own

model like sure you can do that too but

then nowadays is probably the best

approach is to do an adaptive model

selection where you basically are

picking the big and smaller models what

which model does well and this is kind

of the cool new thing maybe I could give

a talk on that um or let's say you're

building agents like we lot lot of

things here like um I mean the word

agent hasn't really been around for that

long but in that time it used to be like

a singleshot uh LLM response call that

an agent if you feel like it then you

have to say no okay we're chain of

thought reasoning now we're going to

make a graph-based agent system like I

presented last year uh no then it turns

that why are you bothering to make

graphs when you could actually have an

agent just figure out what to do? Make a

plan. That's the new approach. That's

kind of the way that um Claude sort of

uh laid the approach there. And you're

like, okay. And then now it's like,

well, you if you want to use dedicated

sub agents, maybe, but then maybe why

not just make a generic agent and have

it recursively work and then give it

skills. Skills are very generic. They're

super helpful. Um and then maybe now

it's maybe the idea is not just to do

that, but to do it with a agent sandbox

so the agent can write code and execute

it because that's super useful because

agents are great programmers. I'd have

them sort of just live in their own

computer. Um, and then arguably now

that's the best approach or maybe even

we're in the world now of like don't

even bother with any of that. Just bring

your own harness like like like let

people select if they want to use one of

these other systems and then you know

not even just building agents but the

technology around context retrieval

things like um you know in the old world

we were like BM25 and keyword search

that's the way to do things but that's

like distant memory. Uh obviously the

future is is retrieve augmented

generation embeddings approximate

nearest neighbor that's how we're going

to find data. Turns out that doesn't

really scale well and it kind of almost

mimics randomness as you keep going. So

then maybe it's about graphs. It's

difficult to get working well. It's

probably not the best. Um so then it's

about hybrid. You want a lexical and you

want to do semantic search and rank fuse

those together. Arguably not. Arguably

agents are actually way better at

finding data because they can find

things and apply their intelligence to

get to it. So each of these things I

just mentioned is arguably the leading

approach for that moment over time.

If you asked me to give a speech right

now on any one of these, I would pick

the last. I'd pick the adapted models

with dedicated RM style agent and

agentic search powered by hybrid.

But uh is this the end of this journey?

This is not that long of that time here.

And so um my guess is the stuff that

you're learning today likely won't last

that long. Not that it's not wrong, not

that it is not the best answer right

now, but probably something's going to

change. The big thing that changed last

year was in my mind Opus 40 to Opus 45.

When you did that, suddenly you got to a

model that could do instruction

following and really high scale. This is

kind of to me the beginning the epic of

like the new agent models. Uh also uh

hardware is getting better, faster,

cheaper. Maybe we'll start to use more

tokens. Like token usage is off the

charts of course and is that good or

bad? What's going to change there?

Enterprises are adopting things

differently whether or not a company has

decided to go all in on one agent to

rule them all sort of like claude or

maybe codec style agent um or maybe they

want to utilize uh agents from different

platforms and different systems or both.

This is going to affect your lives um in

addition to things like just the new

techniques new interesting uh approaches

new technology to power these things. So

the fact that everybody in is working so

hard on this trillions of dollars

investment is actually leading to a lot

of this change and again it's happening

way faster than than I've ever seen for

sure.

Um, now if you look back, um, it's not

this way with everything else. Like if

you go see some of these other

discussions, like I've given a speech on

some of these topics. I looked at them

come some of a few years old. They're

pretty good. I still think they're very

relevant. You want to talk about large

scale databases about uh identity access

controls, how to scale engineering

teams, how to do multi cloud storage,

probably um these are still relevant

things today. These do not change as

fast despite being high-scale uh

interesting powerful uh technologies.

So the previous wisdom of saying

optimize for specific technologies

go deep switch rarely right this what

this the reason you do that is because

switching is hard migrations suck

whenever you migrate you break something

every time no matter what um it's always

harder than you think even [music] if

you know that um and uh and the

switching cost is basically high so

basically don't do it for most things

you're kind of uh just because

something's better out there that's not

the answer for most infrastructure so

typically If you say half life of an

agent infrastructure,

3 to 5 years, reevali, see what's out

there. We've been using databases like

my SQL databases for a long time. It's

still pretty good and probably need to

replace it soon. But um but now with AI

technologies, arguably the halflife is

measured in months, meaning a few months

after you've adopted what might be the

best possible thing, there's a

significant chance that you're going to

have to replace it coming soon. And this

is I think shocking.

Maybe I see from some of your reactions

that like you're kind of like have

experienced this a little bit. Um but

this is a different aspect of the way

that you build technology.

So if you're an engineer, this really

sucks because the thing that you just

learned and that you're making is now

probably going to be out of date soon.

Uh no engineer I know likes this. Um uh

as a startup, you bet on something.

You're like, we're going to go all in.

We're going to go on the technology this

approach and then we're going to

basically uh uh disrupt somebody which

probably will but then you see and see

now like the first phase of AI companies

are starting to get disrupted by the

next phase. If you're a technology

buyer, you're a leader of a company, you

buy technology, you you you select uh

open source models, you select vendors,

there's a significant chance that

whatever you just bought is not going to

be the approach you're going to invest.

That's you know, good luck doing a

three-year deal like on on things about

about this kind of stuff. Or if you're

VC, uh maybe the coolest best thing that

everybody agrees is the greatest

opportunity is no longer going to be the

opportunity soon um because everything's

changing.

So here's my advice. Get good at

changing.

It's almost silly to say because you

know obviously technology changes.

Obviously it's something that is um you

know built in. Of course we're all going

to change. We've done this for a long

time. Um it's hard. I think it's really

hard and the faster that you do it the

harder it is. Uh when I was going

through that uh like oh yeah we switched

from the graph based agent to a um to

the more looping style deep style agent.

I remember very well the conversation

with the engineer. He just he's like, "I

did it. I got a tic search working and

this approach does deep research. It

does all this stuff just like you

asked." Okay, we're going to switch

rebuild it again in this new technology.

And he's like, "Wait, what?" Like, "It's

working. You did what you're talking."

Yeah, but it's not as capable as we

wanted it to be. Like, what do you mean?

You didn't tell me that before. Like,

and then and then so convince him like,

"Okay, this is a new approach." And

then, you know, he does it and it's

good. Two months later, uh, we we're

we're actually shipping the product uh

on on Tuesday. And then I was like,

"Okay, uh, guys, on Wednesday we're

going to rebuild it again on the new

approach." And they're like, "What are

you talking about?" Like, like, um, then

they'll say, like, it's it's almost hard

on everybody like, "Wait, wait, give me

more time. I'll I'll make the new way

the old way do it better." Um, uh, like,

and then also they're skeptical. Like,

now you say that, but like this is going

to change again, right? Like, who are

you to like make these choices? And the

answer is, yeah, I'm pretty sure it's

going to change again. So, this is, I

think, a leadership problem. It's a

technology problem. It's a morale

problem. It's a team problem. It's a

company problem and if you're not

careful, it is actually can destroy you.

It can destroy a lot of things because

people lose faith, they lose morale.

It's a problem.

So, um if my advice is change um and be

ready for change, how are you going to

do it? Three things to give you.

One, um you just got to prepare people.

Well, this is a kind of a people

challenge. So when you build your teams,

when you talk to them, when you prepare

them, if they're in AI world, you got to

tell them like expect change. It's

normal. It's not a problem. It's not

that you did something wrong. This is is

weirdly like um like helps people like I

have a a technology review team and and

then and then they're like we can't like

change. We don't know. We're not sure.

We can't tell you that in two years from

now this is going to be best. Like

that's okay. Uh we're gonna we're gonna

build these things that change. So just

go with it. You have to pick something.

Um, also whenever possible if you can

build an abstraction so that it lets you

swap out what's underneath. We have an

agent extraction in box and you're able

to go through and be like uh like select

things underneath and the agent still

works the same for the customers but it

it's better underneath. Um, and the idea

is that change is not a mistake and and

I highlight like that's very hard for

most people and and I and I would sort

of just you just I tell them all the

time change is not a mistake. You

wouldn't nobody knew six months ago.

Nobody today will know six months from

now. It's Seems very true. So uh at Box

we are now in the habit of reviewing

every six months no matter what. This is

great technology. We love it. Review in

six months like because uh which is just

completely crazy for everything else

that we're doing. Everything else is

like three years. Um also even though

change is critical you um you need to

define what you mean when change. If you

just change all the time there's a new

paper it's awesome. You know our CEO

Aaron is very active on all the newest

things. He's like check this out. Like

don't change just because of that. Like

don't change just because it's a trend.

Change because you know it matters. And

how do you know it matters? Probably

pitch you on eval sets. If you're

building agents, if you're building AI,

make sure that you know what people

have. You have the ability to give the

same input, expect certain output. Grade

that cost, speed, quality, capabilities.

These are the things that you probably

are going to to be wanting. So for us,

it's easy. Does the new approach work

better for our eval sets? What the

customer cares about? If the answer is

yes, strongly consider switching. If the

answer is no, don't bother like or or

keep working on a little bit of work to

see if you can make sure that you you've

fully explored it. Um and then so the

idea is uh build a system that lets you

be able to change. And then the third uh

and final piece of advice here is um

almost certainly none of us can keep up

with everything. It is very hard. Um I

think I heard uh Andre Kaparthy uh he he

was like everything changes so fast I

can't keep up. and you're like you're

sort of quite famously good at keeping

up and so like what's the hope for

everybody else if if that's the case. Um

and so but then so what you do is you

rely on somebody else. You rely on a

technology, you rely on a vendor, you

rely on a platform. Um you when you

select it and um and then here I think

very use I mean like whenever you

whenever anybody's bought technology in

the past I would had advised them like

look at what they do now. Double check

the road map. Make sure it's good. Make

sure it's on the path you want but just

focus on what's available now. But I

think something else here is um should

do that of course that's most important

thing but like look back how have they

handled change what's their attitude

towards change how can what can you when

you talk to them when you read about

their stuff like what happened six

months ago what happened a year ago how

did they handle that transition many of

the vendors that I really like right now

have reinvented themselves three times

in the last year and I now trust that if

something else comes along they're very

good at this they understand agent

technologies they understand the the

eval sets they understand the the

observability systems and then you can

say ah okay good I hope that they keep

up and then I now my sort of thing I

need to do is just evaluate whether or

not that's a good platform

so um making sure that you have this

sort of platforms that do well is is is

critical um and um if anybody's

interested in unstructured content and

AI associated with it uh Box has a booth

downstairs happy to talk to you about

those kind of things

um and then um I I'll leave you with

this is um I actually I fully bet and I

believe that um a company that's born

this year was born last year um will or

maybe even a company a medium-sized

company or a big company will will they

they'll shoot very high the company that

will dominate tomorrow is is is now born

today. Uh but I kind of bet you that the

technology approach that they have right

now is probably going to change multiple

times before they do that. So

interestingly it's like the challenge

the advice the thought here is build for

change

adaptability arguably that's the moat

that you have

until that changes.

Okay thank you everyone.

[applause]