AI Engineer World's Fair 2024

Building SOTA Open Weights Tool Use: The Command R Family

Sandra Kublik· Developer Advocate, Cohere15:03

Loading slides…

Read the talk

Building SOTA Open Weights Tool Use: The Command R Family

Command R and R+ connect retrieval, grounded answers, and dependent tool calls, with an application toolkit that makes those model behaviors usable in a chat interface.

From a talk by Sandra Kublik

Before you start: Familiarity with language models, retrieval-augmented generation, and basic API calls will help you follow the discussion.

Open weights for retrieval and sequential reasoning

What if a model could handle advanced retrieval-augmented generation and sequential reasoning, compete with larger proprietary models, and still let you run its weights locally? That is the opening premise of Sandra Kublik’s presentation. She positions Cohere’s family against GPT-4 Turbo and Claude Opus, while emphasizing smaller models and local execution. Local availability is part of the proposition; the talk does not specify the hardware required.

Command R arrived on March 11, 2024, with open weights and a focus on scalable RAG. Command R+ followed on April 4, extending that emphasis to advanced retrieval and tool use. Open weights should not be confused with unrestricted licensing: the original Command R+ model card specifies CC-BY-NC licensing and an acceptable-use addendum. Kublik reports that R+ climbed the LMSYS Arena shortly after release, presenting community reception alongside the models’ intended specializations.

Release timeline listing March 11th for Command R and April 4th for Command R+, with an LMSYS post and leaderboard screenshot on the right.Open full source frame
Command R and Command R+ release dates alongside an LMSYS Arena post.
0:000:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Different applications, different kinds of grounding

Early adoption appeared across model platforms and applications. Kublik reports OpenRouter trending and 150,000 Hugging Face downloads within two weeks of release. HuggingChat’s tools integration used Command R+ as its underlying model, with tools including a document parser, image editor, and calculator. This means R+ powered the application’s tool behavior, rather than serving as a training base for a separate HuggingChat model. Kublik reports that almost half a million developers and researchers were using the R family at the time of the talk. These are historical adoption figures attributed to the presentation.

The examples move from broad adoption to concrete applications: Daniel San’s Coding Assistant and Complexity’s generative search, followed by two bots in Cohere’s Discord community. The bots illustrate two distinct ways to shape an application:

  • Playful behavior: A fine-tuned bot demonstrates model capabilities through a more playful interaction style.
  • Documentation help: A helpful bot grounds its answers in Cohere’s documentation and API information.

The distinction matters because adapting how a model behaves and supplying the evidence it should answer from solve different problems. The accompanying Discord examples show API documentation help beside a Dino Dash game concept.

Two Discord conversations under “R+ in the wild”: a chat endpoint explanation with documentation links on the left, and a Dino Dash game description on the right.Open full source frame
Discord examples pair API documentation help with a Dino Dash game concept.
2:182:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:18 · section reference included

Why retrieved text is not enough

Building a reliable RAG application requires more than putting search results into a prompt. Kublik identifies three difficulties, beginning with prompt sensitivity. The model must know what information to look for, where to look, and how to distinguish retrieved evidence from the conversation it has already had with the user. A prompt that blurs those roles leaves the model to infer which text should govern its answer.

The second difficulty is document-position bias. Kublik describes models paying excessive attention to the beginning of a document instead of consistently finding the most relevant information wherever it appears. She points to retrieval evaluations, including needle-in-the-haystack tests, as ways this weakness becomes visible. Having the right information somewhere in the context does not ensure that the answer will use it.

The third is the conflict between pretrained knowledge and supplied evidence. The model’s parameters may contain an answer that no longer matches the world. Asking who the current president of the United States is illustrates the problem: the task calls for up-to-date information, not simply the name most strongly represented in training. RAG must let external evidence guide an answer when the stored knowledge is stale.

4:204:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:20 · section reference included

Train the retrieval decision, then make the answer inspectable

Cohere’s response is to shape the model’s retrieval behavior through post-training. The intended sequence is:

  1. Decide whether external information is needed. Some requests can be answered from pretrained knowledge.
  2. Operate the retrieval system. Form search queries and retrieve relevant information when the request requires it.
  3. Use the results as grounded context. Bring that evidence into the ongoing conversation when generating the answer.

This makes the decision to retrieve part of the model’s trained behavior, alongside the ability to use what retrieval returns.

Fine-grained citations make that grounding inspectable. A user needs to see where a particular claim came from and whether its source deserves trust. Kublik describes citation behavior as a major training focus and associates it with lower hallucination and more reliable use of context. Citations provide a route to checking an answer; their presence alone does not replace that check.

Kublik reports best-in-class performance for Command R and R+ on standard RAG datasets including KILT. The talk does not supply the tasks, scores, or evaluation configuration needed to interpret that comparison precisely. The broader design claim is qualitative: models small enough to be affordable, with good token efficiency, can cover use cases that would otherwise require a larger pipeline of LLMs.

6:336:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:33 · section reference included

Toolkit supplies the application around the model

Even with retrieval and citation behavior in the model, an application still needs to expose those capabilities. A useful RAG chat interface must retain conversation history, display fine-grained citations, accept document uploads, and connect to tools. Cohere spent time building that interface and open-sourced it on April 24, 2024 as Cohere Toolkit. The repository was subsequently archived on May 14, 2026, so it is a historical application reference rather than an actively maintained starter today.

Toolkit separates the interface, model access, and retrieval so that an application can change one without rebuilding all the others.

ComponentRoleChoices described in the talk
Interface and storageChat UI and conversation historyNext.js application; small SQL database
Model accessConnect the application to Command R modelsCloud providers, Cohere platform, local access, Hugging Face
RetrievalConnect tools and data sourcesLangChain-based example with document uploads and web search

The example retriever is a starting point. The retrieval component can be extended with other tools and data sources, while the model component determines where inference happens.

Diagram showing a NextJS chat UI, Fast API, database icon, retrieval tools, and model modules connected to Cohere Platform, AWS Sagemaker, Azure, and Bedrock.Open full source frame
Toolkit architecture connects the chat interface, retrieval components, and model providers.
8:228:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:22 · section reference included

From retrieval to dependent tool calls

Training a model to operate vector databases and retrieval systems provides a path toward broader enterprise tool use. The next goal is to let the model work with tools beyond retrieval, ideally from their descriptions alone. Kublik presents zero-shot use of arbitrary tools as a direction of training, not a universally achieved capability.

The critical distinction is whether actions depend on one another.

Tool-use patternDependencyExample
Single-stepOne action, or independent actionsSearch for a document; send an email
Multi-stepLater actions use earlier resultsFind → compare → summarize → email

In the multi-step example, the system finds a document, compares it with another document, summarizes the comparison, and emails the result. It cannot produce the comparison summary before it has the comparison, and the email needs the resulting summary. The workflow’s dependencies determine the sequence.

That sequence need not be a fixed march toward completion. The system may retrieve from different sources repeatedly, inspect intermediate results, and correct errors. Kublik describes this as a loop and uses the term agents for software performing a sequence of actions in which each action builds on earlier steps. The relevant capability is not merely issuing several calls; it is using their results to decide what should happen next.

10:2310:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:23 · section reference included

Describe tools, inspect results, revise the plan

The multi-step API, described as released the week before the talk, exposes this behavior through tool descriptions. The developer supplies the available tools, what each does, and its parameters. A user request then gives the model a goal from which to construct a plan. This is the historical API presentation; the relative release date does not establish when multi-step capability first became available.

The execution loop connects planning to evidence:

  1. Select a tool call that advances the user’s request.
  2. Execute the call and return its result to the model, as illustrated by the Python loop on the slide.
  3. Inspect the returned content and decide whether the initial plan still fits.
  4. Continue or revise. If an API returns an error, retrying or changing the plan may be the next action.

In the document workflow, these returned results are what let the model progress from finding documents to comparing them and preparing the email. Retrying and replanning are intended behaviors, not guarantees that an unavailable API or another failure will be recovered from successfully.

Multi-step API slide with a Python code example, a loop processing tool calls, and a final printed answer.Open full source frame
Python example for calling tools and returning their results to the model.

The behavioral instructions live in a large multi-step preamble that Kublik points readers to on Hugging Face. It tells the model how to carry out the task, including how to proceed through intermediate results. The model’s trained prompt structure matters here: the model card warns that departing from its template can reduce performance. Open weights make these instructions and model behavior available for inspection, but do not make the surrounding orchestration irrelevant.

12:3312:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:33 · section reference included

Tool provenance and production economics

Citations carry the grounding principle into tool use. Command R and R+ are trained to produce claims that users can verify through citations, making it possible to explain which tool supplied the information behind a response. For a workflow that crosses several data sources, that connection between a claim and its origin is a practical part of transparency.

Kublik claims Command R+ is competitive with Claude Opus and GPT-4 Turbo while being three to five times cheaper. The talk does not provide the pricing basis, workload, or exact model versions, so this is a historical cost comparison rather than a general deployment estimate. Kublik also reports Command R+ as close to or on par with GPT-4 Turbo on standard complex-reasoning benchmarks, without naming those benchmarks or supplying scores. The production proposition is that competitive capability at lower cost could make repeated tool-use steps more scalable.

The stated direction for upcoming releases is continued work on multi-step capabilities. No release dates or additional specific capabilities are promised: the focus remains improving the model’s ability to carry a task through dependent actions.

13:3714:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:37 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…