[
  {
    "startMs": 720,
    "text": "Hi everyone. Uh, my name is Nouha. I'm a senior researcher at Cohere, an associate industry member at Mila, and an adjunct PI at the Allen Institute, Tübingen. I'm so excited to tell you today about how to optimize your agent team and get frontier coding at seventy percent lower the cost. Um, this is a work that I did, uh, at Cohere. All right. So traditionally, there are two ways to make an agent stronger at inference time. First, you can let it think longer, meaning you scale compute sequentially"
  },
  {
    "startMs": 30680,
    "text": "with longer chain of reasoning. And second, you can sample more. You run many independent attempts in parallel, and then you aggregate them. But as agent take on longer and longer tasks, they can navigate large repo, they can edit the code, they can debug their own work over hours. So what can happen here is that the context window may blow up. And so we can see here that, you know, the context window eventually fills up, and the longer it runs, the longer you wait. And sampling more"
  },
  {
    "startMs": 60500,
    "text": "doesn't either fix it, because each attempt here still has to do the whole task on its own, right? So what if we split the work? Um, um, so here in this case, the key-- the lead keeps the task and hands out sub-tasks, and each worker starts with a fresh context. So it can read files, it can run tests, it can dig through logs, et cetera, et cetera, and then it can send back only a compact result. So here the lead keeps the full picture, and it can finish faster if the task is decomposable and"
  },
  {
    "startMs": 90240,
    "text": "agents can work, uh, on independent pieces. And so here we're talking about a new axe, uh, for scaling inference time compute. You work as a team. You give a task to a team of coding agent and watch them do the task for you. But then scaling the team gets expensive very, very quickly. And so y- as you add more agents, your token budget can blow up. Um, and it can actually, um, and, and it raises a, a harder question, actually. So first, how does coordination actually"
  },
  {
    "startMs": 120120,
    "text": "happen? How should we organize the team so that agent can work together, et cetera. And so, you know, this is the question that we set out to answer. How should a team be orchestrated to be both effective and efficient? By effective, I mean it actually solves more of the task, so we get higher reward. And by efficient, I mean it doesn't burn through your budget getting there. All right. So since these models now are extremely capable, I was wondering, what if"
  },
  {
    "startMs": 150140,
    "text": "we throw an extremely challenging coding task as a society of agent and see what they can do? And so we would expect them to automatically form clean groups, split up the work neatly, and build it together, communicate nicely, and coordinate, et cetera, et cetera. But that's not what-- not quite what happened, actually. In reality, what happened, the agent preferred to solve the task on their own. So each went off and built their own version. Even when we asked them to communicate, they mostly discussed things sometimes that didn't"
  },
  {
    "startMs": 180020,
    "text": "matter, and sometimes they kept also stepping on each other toes. They also preferred to communicate back to the lead, et cetera, et cetera. So here we were wondering, okay, how should we design a team so that, you know, we can actually get the best out of it in terms of reward and in terms of cost, and hopefully in terms of latency. And so, you know, we set out to figure out what a multi-agent harness should actually look like, and like we were wondering about the team design. So there are many design choices, and each one changes how well the"
  },
  {
    "startMs": 209900,
    "text": "team does and how much it costs. So first, the organization, right? How do you structure the team? Who leads? And at which stage of the work planning, implementation, or verification do actually workers come in? Um, the second thing, uh, is communication. Should agent talk to each other directly or only report to the lead? Third is the team size. How many workers is worth it? Like should it be one, two, sixteen, hundred, et cetera? And fourth, the models, right? Which model should"
  },
  {
    "startMs": 239960,
    "text": "play which role? Do we need, you know, a strong model everywhere, or, you know, can cheaper ones do the same work, right? And finally here that something that we cared about is the stage of the software engineering, right? Should, should, you know, allo-- we should allocate more workers towards planning, towards implementation, toward verification, et cetera. And so we actually frame team design as a compute allocation problem. So what we cared about is reward per added dollar over the solo agent, right?"
  },
  {
    "startMs": 269840,
    "text": "So starting from a solo agent, we ask where does the next dollar of inference buy the most reward? This is what we cared about. And so here we actually define a metric called efficiency, and it's actually a function for things. First, the stage. It's the, the stage at which the worker is at, whether it's implementation, planning, or verification, or all of them together, we call it P. We also define that as a function of the worker model. Is it like a very capable model? Is there-- Is it a very, you know, uh, you know, less capable"
  },
  {
    "startMs": 299860,
    "text": "model, the number of workers, the lead model, et cetera. Um, and all right. So basically, if we-- to answer this question, we actually design our multi-agent harness as a top-down hierarchy. We build deliberately a minimal harness, a mini suite agent, and it's hierarchical. As I mentioned here, we have a lead, and based on the task, it decides dynamically how big the team should be and, um, you know, who should do what. And so here we made sure that every part of the harness is exposed, so we could change"
  },
  {
    "startMs": 329700,
    "text": "one choice at a time. We keep everything else fixed and see how each one moves reward and cost. Right. And so here more closely, how does a team of coding agent do the work? You can see here every agent works through a bash tool in one shared workspace. So basically, the lead receives the task and hands out work with a tool we call Delegate H. And then the worker sees only its sub-task and reports back to the lead, and it can now also di-- communicate directly to the other worker. And so here, basically,"
  },
  {
    "startMs": 359640,
    "text": "the lead can decide based on the task, \"Oh, actually I need to just, you know, pow an implementation worker or a verification worker,\" et cetera. And so, as I mentioned also, the lead decides how many workers to spout, how to combine, also consolidate the results. It also has the capability to resume a worker, to kill a worker, and to tap in and, for example, supervise and tell the worker that you have made some errors here, fix it, et cetera. And so, yeah, and you may be wondering, okay, what is the lead"
  },
  {
    "startMs": 389328,
    "text": "doing while the other workers are working? We actually have two variation. One is what we call the blocking lead, the other one is asynchronous lead. So the blocking lead actually hands out subtask and waits until every worker reports back. It doesn't do anything, you know, in between. Uh, whereas an async lead, it keeps actually working while, you know, the results keep coming from the other workers. It can supervise what other workers are doing, it can send them feedback, et cetera, et cetera. And so how did we test this? We test this on a"
  },
  {
    "startMs": 419468,
    "text": "benchmark called Program Bench, which is basically a reverse engineering task. So basically, the agent starts from an empty workspace with only the original program executable, which it can run but not read, and it also receives some documentation. So here the goal is to basically probe the program to discover its interface, to discover the edge cases, the behavior, and then write a new implementation that behaves in the same way. And then the task here were extremely challenging. They range from SQLite, sorry, for example, the"
  },
  {
    "startMs": 449188,
    "text": "PHP interpreter and for example, you know, on the leaderboard on the official website on the task FFmpeg, the best model only passes ten percent of the test. So this is also done by, you know, frontier model. So it was quite challenging, and it was a good test bed for, for our, you know, investigation. And so the question here is that, does a team beat a solo agent, yes or no? The answer is yes. It actually can beat a single agent, but at four to seven times"
  },
  {
    "startMs": 479188,
    "text": "the cost. And we can actually see here in the figure that, you know, with a minimal prompt, uh, a team with a blocking lead actually beats the solo agent by almost ten points, but it costs about, it costs about four to seven times as much. Now, you can see that the blocking actually perform better than the asynchronous, but cost higher and it actually takes more time to complete. So what we noticed here is that we noticed that the async lead doesn't know when to wait, when its"
  },
  {
    "startMs": 508948,
    "text": "workers are still exploring. It starts building on its own guesses before the workers finding come back, and then when the reports finally arrive, it has already moved on. And so either it ignores them or it has to go back and redo the work. And sometimes it just decides the task is done while the workers are still actually running. So basically, it decides, okay, done. It submits the work, and so the work of these workers is simply lost. But then, you know, this problem never occurs with a blocking lead because it will"
  },
  {
    "startMs": 538968,
    "text": "wait until the workers report back, and then it actually consolidates the work and do the submission. All right, so the next thing I want to emphasize on is the prompt, minimal. So I, I mentioned, okay, in the earlier results, I-- we used the minimal prompt. What is an earlier prompt? And I can't emphasize on this enough. This is how our agent operates. We give them prompt and instructions, and essentially they do what we ask them. And so the first one, we tried to go very, very minimal. We just told the lead about the task, you know, that it"
  },
  {
    "startMs": 568848,
    "text": "should create, uh, you know, subworkers and agent, and it should inform them about the task and should tell them about the tools. We gave also the lead tools. We wanted to see why we started minimal because we wanted to see how the agent behave on their own. Do they need a lot of guidance or is it, is a minimal prompt enough, et cetera? And so this was very minimal. We also were wondering, okay, what happens if we give it a very heavy prompt? Meaning like, you know, what the model should exactly"
  },
  {
    "startMs": 598648,
    "text": "do. We tell it exactly about different tricks. Follow exactly this plan of try to, you know, delegate agent and, you know, create agent throughout the whole process of the implementation from exploring and planning to implementation. Create also adversarial workers to verify the task, et cetera, et cetera. And actually, that helped a lot. Uh, we can see here, um, you know, compared to the minimal prompt, um, you know, we can see that the solo agent alone jumps almost six point just from better"
  },
  {
    "startMs": 628688,
    "text": "instructions and telling them, \"Okay, don't step on each other toes. Uh, don't, you know, uh, there is this folder right in this folder,\" et cetera. So lots, lots of guidance and lots of prompt tuning, which is not ideal 'cause we spent too much time there. And now these agent are supposed to be smart enough to figure out how to do it. And so we can actually see that the teams both, you know, async and, uh, blocking both go up, um, and we can see that the async lead now catches up with the blocking one, right? So once we tell the lead how to coordinate, it knows when to"
  },
  {
    "startMs": 658708,
    "text": "wait. So this actually really helped address the problem that I talked about earlier. But again, you know, we have a team that works better than the solo, but it's really, really expensive. It costs six to seven times as much as the solo agent. So we were wondering, now that a team beats a solo agent, um, but it costs more, so if a team is a new way to spend inference time compute, where should the next dollar go, basically? Can we actually optimize our agent setup so that actually we get"
  },
  {
    "startMs": 688728,
    "text": "very good reward, but at a very, you know, much cheaper cost? So to answer this question, actually, we took the same lead and gave it workers at only one stage of the software engineering process. So the first setup that we considered, here, the workers are only responsible for the planning and exploration, right? So basically, here, the lead will give them the task, and then each of the workers will go and try to probe the binary to discover, okay, what kind of plan I need to do, and then it will report the plan to"
  },
  {
    "startMs": 718748,
    "text": "the lead, and the lead will take care of the implementation verification. The second one is, you know, workers are only responsible for implementation. And so the lead is the one responsible for coming up with a plan, distributing that, splitting the work between workers. They will do the implementation, report back with the code Actually to the, to the lead. And so basically, and then the lead does the verification on its own. Now, the third actually setup that we had, the lead is responsible for both the planning and"
  },
  {
    "startMs": 748558,
    "text": "the implementation. And then here the workers are responsible to do the verification, which is the final stage, and then they will report back to the lead, and then the lead will decide whether to submit or not. And so, okay, we wanted to see where exactly extra agent will pay off the most. And here are the conclusion of our, our results, like extra agent pay off the most in program bench in planning. So here you can see, okay, going from solo with minimal prompt to solo with PIV. PIV is the heavy"
  },
  {
    "startMs": 778498,
    "text": "prompt. Actually, we get like a jump already, and then we go with a team of heavy prompting. We get extra, you know, increase and gain in performance. But then, now when we allocate more agent towards the planning, right? We can see that we get almost the same performance as the PIV, but at half the cost. So actually, the conclusion here is that, you know, the extra agent pay off the most in planning. Um, uh, at, at, at the task that bottlenecks actually, uh, the, the, the task"
  },
  {
    "startMs": 808518,
    "text": "which is planning in this case, because you need to figure out how exactly, what's the bi-binary behavior and come up with an, um, detailed plan on what should you implement first, et cetera. And so here, yeah, I... Actually, we computed even, you know, the efficiency ratio that I discussed earlier. And we can see that, you know, it dominates the PIV actually on the frontier. So each planning dollar buys almost one point six more reward than any other stage that we have here, whether it's implementation or verification. And actually, we, we, we"
  },
  {
    "startMs": 838298,
    "text": "noticed that, you know, you know, doubling down on the number of agent in the implementation sta-- in the implementation or verification stage doesn't actually really help. We can see that it just increased the cost, but we are actually getting the same performance as the solo, so it's not worth it. And so again, like program bench only exposes a black box executable. So discovery here is the bottleneck. So planners paralyze this probing and it brings, and they bring lots of benefits, right? And so, yes. So this is to explain"
  },
  {
    "startMs": 868258,
    "text": "why planning workers pay off. And we were wondering, why not, for example, verification or implementation. And you know, as I mentioned, program bench give you this binary and its documentation. So before you write anything, you have to work out what the program actually does. So discovery is the real bottleneck here. And you can also think about this in terms of context, right? So we have one agent that's exploring, you know, loan files, uh, it's in its own context with probing and also still misses edge cases, right? And on the other"
  },
  {
    "startMs": 898298,
    "text": "hand, you know, we have several workers each explore a different part of the program in a fresh context, and each one hands back a compact report to the lead. So basically here the lead, you know, learns more than it could do alone. And so now, you know, this is also to explain better this phenomena. And you can-- We actually looked at our data, and we came up with these nice figure based on what we saw. And we spe-specifically looked at, uh, when the lead delegates, right? So here we have planning, we have implementation and"
  },
  {
    "startMs": 927758,
    "text": "verification, and we have all the stages as well. And so every row that we can see here in the figure is one run. Uh, and actually time goes left to right, right. So the yellow is planning, the blue is implementation, the green is verification. And every dot here that you see in the figure is the lead sending work to a worker. And so with planning guidance, all the delegation happens in one burst at the very, very first of, you know, the, the work before any code is written. So basically here the, the workers"
  },
  {
    "startMs": 958038,
    "text": "go explore, and then they report back, and then the lead builds. With the full, um, PIV guidance where, you know, um, there is no, um, specific guidance on where exactly the, the, the number of workers should be allocated. We get the same early burst, but then if you see here, y- actually the lead keeps delegating all the way through. Another eight hundred and thirty calls and they buy nothing. Actually, it just increased the cost. As we saw, it was around five hundred dollar, but it didn't"
  },
  {
    "startMs": 987858,
    "text": "actually improve in performance. So it's the same reward for planning, we can see, but it's two point four times more reward per call that we, we actually measured. And so what pays off here is, is, is one early burst of discovery and not delegating all the way through, right? So why implement workers fall short? Here, first of all, as I mentioned, the plan was the bottleneck, and so if you don't do it right, it's not gonna work out. And so here we have only the lead that explored. It"
  },
  {
    "startMs": 1017898,
    "text": "had limited capacity to discover all, you know, the, the edge cases and the full binary. And so we had a weak foundation, and then basically each worker got like a narrow slice. It doesn't have visibility over the entire task. And basically, they were already doing errors by implementing each feature on, on their own. That's one problem. And on top of that, they were building on a weak foundation. And also we noticed that... So here you might ask me that, you know, why not give workers more context?"
  },
  {
    "startMs": 1047858,
    "text": "We did. But we noticed that when you-- they have more context, actually, they didn't respect the boundaries of their specific task. They get excited, and then they want to build extra features that nobody asked them for it, so they just step on each other toes, et cetera. The other issue that we noticed is that, you know, they are supposed to communicate together to figure out who's doing what to avoid conflicts and duplicated work, but they didn't do that. They actually, they worked alone and eighty-four of messages go, go to the lead directly and nobody sees"
  },
  {
    "startMs": 1077718,
    "text": "who's doing what. And sometimes here what happens is the lead gets overwhelmed, and it doesn't know, doesn't communicate well where the problem is to the other workers, et cetera. So it can be actually quite challenging. And so what happens here is that when, you know, implementation is not, uh, fully correct, you know, the, the, the lead pays for the gap. It actually tries to, you know, make up for that duplicated work and does lots of stitching. And so here what happens is that the latency goes up and also the reward goes down and,"
  },
  {
    "startMs": 1107998,
    "text": "you know, w-- it, it, it's also expensive. Uh, okay. So basically so far I've been talking about program bench, but we also run experiments also on CommitZero. And what I want to say here is that the results, like are, are different and the hierarchy and the, the allocation of the team is different according to the task. Because the bottleneck for program bench was planning, and so allocating more agents there really, really helped. But for CommitZero it was not the"
  },
  {
    "startMs": 1137922,
    "text": "case. The CommitZero, the bottleneck was not planning because basically you're given the skeleton, so the docs are known and the unit tests are actually known to the workers. So the plan is basically given to you. And the, the, the bottleneck was for the, you know, for the reward to go up was the implementation. You need to get the implementation right. And so here when we doubled the number of agent on the implementation stage, this is where we saw the biggest improvement. I'm not gonna present results on this. Please check out the"
  },
  {
    "startMs": 1167762,
    "text": "paper for, for, for these results. You'll find them in the appendix. So the conclusion here is that you need to allocate more agents at the software engineering stage that bottlenecks your task. All right, so planning already, uh, gets us the same reward at half the cost. So, uh, we were wondering, can we, uh, go even cheaper than this? Can we cut the cost even further? And so here we did some experiments, and we focused, for example, on three GPT five point six, uh, models, uh,"
  },
  {
    "startMs": 1197802,
    "text": "Luna, Terra and Sol, uh, in, uh, this, uh, increasing order of capability and cost. And so, like the question here is that what would happen if we keep Sol as the lead, like the strongest model, and then use the cheap models as the workers, right? So remember here we used to have this. This is all the P here is the old Sol, you know, model. By the way, here the, the, the Sol, the basic-- this is the version like of P where the lead dynamically decides on the size of the"
  },
  {
    "startMs": 1227722,
    "text": "team, and it ended up on average creating and spawning five agents, right? And so basically here we have this, we go from P id, we actually allocate more agent towards P. And, uh, basically when we replace the Sol, you know, sub workers with cheap ones like Luna, we actually get almost, you know, same performance but at sixty percent less the cost. Like just we lose a little bit on performance, but it's still good because we're actually"
  },
  {
    "startMs": 1257242,
    "text": "maximizing reward compared to the Solo and then we're, we're getting less cost. And so, you know, now the other way around, we were wondering like, can the lead be cheap, right? Can we just even get a, a cheaper lead? And so already here we can see that, you know, going from like a cheap lead like Solo, Luna, Terra, Sol, we keep increasing in performance and also in cost. And so can we have like this homogeneous team like Luna plus Luna, Terra plus Terra. Already we can see the scaling law"
  },
  {
    "startMs": 1287542,
    "text": "the same. As, uh, the model gets more capable, we get better results. But then actually, you know, we noticed that when we have a strong lead, Sol plus Luna, we can get the same performance as Sol plus Sol, but you know, at much lower the cost. But the other way around, like cheap workers can't make up for a we-weak lead. Basically, the lead sets the ceiling and the workers set the price. Here, Terra with Luna workers still trail Sol working alone fifty-one percent versus fifty-seven percent. So essentially here the lead"
  },
  {
    "startMs": 1317462,
    "text": "sets, you know... Basically, this also matches Meta's meta-reasoning work recently. Like they just mentioned that a better controller raised what the same workers achieved. All right. So, so far the lead decided how many workers to spare, as I mentioned, and it did on average five workers. Now we were wondering what would happen. We want to scale ourselves the number of agent at each stage. So we will impose the team size. Um, so here, uh, we're gonna, you know, work with Sol as a lead, Luna as"
  },
  {
    "startMs": 1347262,
    "text": "planners because this was the best, um, reward cost, um, um, uh, relat-- and we go from two to sixteen planners, right? Um, and so here I can actually look at the cost. You know, the sixteen planner team is thirty-eight percent cheaper than all Sol planning team and actually seventy percent, um, cheaper than workers at every stage here for the PID, which is great 'cause here we-- and it seems we, we-- due to some"
  },
  {
    "startMs": 1377282,
    "text": "budget constraint, we couldn't like scale up the, the number of agent further, but it seems like going up and up and up, obviously there is a ceiling to it, but it's really helping. And the good news is that it's not that expensive. It's not a-- it's not more expensive than the five Sols in, uh, all allocated towards planning. Now we also, uh, try to see whether, uh, scaling up the number, the size of, uh, the team, uh, for the other variants like implementation and verification, and it"
  },
  {
    "startMs": 1407102,
    "text": "didn't help at all. Uh, we already saw that, uh, you know, allocating teams', um, agents to work in these stages, uh, is not helping because of, you know, the foundation was weak and other considerations. And so the last thing we were actually curious about, how do agents communicate, right? And so we gave every agent, uh, a send message tool to reach any other agent actually in the sandbox. And we were curious whether communication, uh, happens directly or, you know, it happens through the lead and"
  },
  {
    "startMs": 1436862,
    "text": "whether communication will lead to better performance or worse. And so basically the first thing is that are they communicating together? Not really. They are actually essentially communi-communicating through the lead. Like eighty-four percent of all messages went from a worker to the lead. Worker to worker, you can see is actually very little, less than one message per task. So even when they can talk to each other, they actually behave like a hierarchy. Like do your piece, you report up, expect the lead to communicate back to the right sub"
  },
  {
    "startMs": 1466822,
    "text": "agent, et cetera. And you can also see this pattern consistent across all models. The lead takes much more, talks, sorry, talks much more to workers than they talk to, to each other. And this is also true for Claude Opus four point seven. Also, we did experiments on Gemini"
  },
  {
    "startMs": 1484122,
    "text": "I mentioned, we were curious about the reward. Does it, you know, does extra communication between workers help or not? In our experiments, we didn't find it helped, and I will explain why. Um, so first already you can see, you know, on the left, you can see that about half of Luna messages never even get through, like, because they're malformed and the harness reject them, et cetera. Um, and so we notice also in the middle figure that Sol writes about three to four times as much message text as, as"
  },
  {
    "startMs": 1513942,
    "text": "Luna. And under the PIV guidance, stronger teams actually, we can see that they tend to send more messages overall with leads steering their workers more, which is good. So we can see that, you know, as the, you know, capability of the model increases, so they message rather more. But then on the right figure, you can see that more talking doesn't mean more reward. For example, for Sol and Tora-- Terra, adding messaging changes nothing basically beyond noise, and Luna teams actually lose four to seven"
  },
  {
    "startMs": 1544042,
    "text": "points, so it just seems distracting to them. And we noticed that communication mostly carries status report to a lead, you know, who already receives each worker results anyway, so it, it was just like an overhead. There was nothing important to be discussed. And I want to be clear here. I don't wanna say that peer messaging is not necessary and we should not do it. I do think it's necessary, and I do think as we scale the number of agent, they should be capable to communicate efficiently together because it doesn't make sense to"
  },
  {
    "startMs": 1573922,
    "text": "all the time communicate through your manager for you to work with your team members. No, it doesn't work like that. I think that it's also a byproduct of how, uh, of the harness that we built and also of the, you know, the task itself. Because basically the lead was splitting the work and making sure that each work is independent of the others, and so they give them di-- or the lead gives them directly the task, and so they are not also aware of other, you know, agent in the same sandbox. And so we gave them"
  },
  {
    "startMs": 1603962,
    "text": "the tool only to communicate together, but they didn't have visibility who exists there. And so they have to do extra efforts, and maybe a message where would have solved the problem, and so would incentivize them more to sort of communicate and to figure out, \"Oh, this actually agent is doing something similar to what I'm doing. Let me communicate.\" Or, \"I need more help. Let, let me reach out to someone else.\" Anyways, we leave this to future work. And so the other thing that we also were curious about is like, you know, whether other, you know, models, frontier"
  },
  {
    "startMs": 1633862,
    "text": "models know how to lead. So we also ran five other models family with the same harness, with identical tools. And so here I'm talking about Opus four point eight point-- four point seven five, also Gemini family. We noticed that GPT five point six teams delegates often, and it gained up to seven points in reward. Claude, on the other hand, actually behaved much, you know, the same, gaining almost six points. But then Opus four point seven rarely delegated and actually"
  },
  {
    "startMs": 1663862,
    "text": "gained very little. Haiku and Gemini, you know, delegated as we-- delegate as GPT five point six, but then gained nothing. And so what I want to say here, the conclusion is that, you know, two things have basically to, to go right. The lead has to choose to delegate. This is important because it's actually deciding on who's gonna do the work, and it has to actually make use of what comes back and at which stage of the task do you also delegate. All right, so here are the takeaways. Three things to remember. The"
  },
  {
    "startMs": 1693862,
    "text": "first thing is that teams are now a real new axe for inference and compute, but the catch here is that they cost several times more. We got up to ten points, which is, you know, great, but still it's very, very expensive. Now, you know, we tried our best to optimize our teams of agent, and we were successful to some extent. Still some work to be done. But essentially here the conclusion is that, you know, where you put agent, uh, matters more"
  },
  {
    "startMs": 1723922,
    "text": "than how many.  So you need to find the bottleneck and scale the team there. This is what matters the most. And finally, the team does not have to all be strong. Like a strong lead with cheap workers can get you near frontier results at actually a fraction of the cost. And so what's next here? These are open questions that we are so curious and excited about. First, here we're excited about this hierarchy versus self-organization. We, in our work, we tested up to sixteen workers in a"
  },
  {
    "startMs": 1753802,
    "text": "hierarchical lead worker structure. But then how should teams of thousands of agent be organized, communicate, and spend compute? Can agents design the team themselves given task, budget, and deadline? We think so, and so, but that requires, you know, careful investigation of the problem. Now, the second thing is that we think that there must be alternative forms of communication. As I mentioned, we observed eighty-four percent of messages going from workers to the lead and only four percent between workers. And"
  },
  {
    "startMs": 1783722,
    "text": "so as... I think projects outgrow, like coding project, you know, they can outgo- grow one lead. Will effective communication here emerge, right, with scale or, like, should it be trained, right? How should the lead adapt to the task? It's not scalable to just... For each task that we have, we spend, uh, you know, a huge amount of time optimizing the prompt and giving very detailed instruction to the lead. That should not be the"
  },
  {
    "startMs": 1813562,
    "text": "case. The lead should figure out how to actually, you know, uh, organize the team, uh, according to the task that it has. Um, now finally, uh, multi-agent safety and security. Um, you know, uh, this is, I can't emphasize on this enough, especially given what happens with the Hugging Face incident. You know, right now we have teams of agent that can run autonomously twenty-four/seven with no supervision. They're quite capable, and we're not"
  },
  {
    "startMs": 1843282,
    "text": "talking anymore about the single agent, but we talking about teams of agent and even swarm of agent. And so here the, the key question is that how do we secure our systems and our infra? And we need to urgently basically figure out ways on how to align these models. All right. With that, I actually would like to thank you for making it to the end. And I want to give a special shout-out to the first author here, Kinan and Daniel, for their hard work on this."
  },
  {
    "startMs": 1873222,
    "text": "And for more information, please check out the paper and the blog using these QR codes. And thank you so much for listening."
  }
]
