AI Engineer World's Fair 2024
Building SOTA Open Weights Tool Use: The Command R Family
Loading slides…
Read the talk
Building SOTA Open Weights Tool Use: The Command R Family
Command R and R+ connect retrieval, grounded answers, and dependent tool calls, with an application toolkit that makes those model behaviors usable in a chat interface.
From a talk by Sandra Kublik
Before you start: Familiarity with language models, retrieval-augmented generation, and basic API calls will help you follow the discussion.
Open weights for retrieval and sequential reasoning
What if a model could handle advanced retrieval-augmented generation and sequential reasoning, compete with larger proprietary models, and still let you run its weights locally? That is the opening premise of Sandra Kublik’s presentation. She positions Cohere’s family against GPT-4 Turbo and Claude Opus, while emphasizing smaller models and local execution. Local availability is part of the proposition; the talk does not specify the hardware required.
Command R arrived on March 11, 2024, with open weights and a focus on scalable RAG. Command R+ followed on April 4, extending that emphasis to advanced retrieval and tool use. Open weights should not be confused with unrestricted licensing: the original Command R+ model card specifies CC-BY-NC licensing and an acceptable-use addendum. Kublik reports that R+ climbed the LMSYS Arena shortly after release, presenting community reception alongside the models’ intended specializations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different applications, different kinds of grounding
Early adoption appeared across model platforms and applications. Kublik reports OpenRouter trending and 150,000 Hugging Face downloads within two weeks of release. HuggingChat’s tools integration used Command R+ as its underlying model, with tools including a document parser, image editor, and calculator. This means R+ powered the application’s tool behavior, rather than serving as a training base for a separate HuggingChat model. Kublik reports that almost half a million developers and researchers were using the R family at the time of the talk. These are historical adoption figures attributed to the presentation.
The examples move from broad adoption to concrete applications: Daniel San’s Coding Assistant and Complexity’s generative search, followed by two bots in Cohere’s Discord community. The bots illustrate two distinct ways to shape an application:
- Playful behavior: A fine-tuned bot demonstrates model capabilities through a more playful interaction style.
- Documentation help: A helpful bot grounds its answers in Cohere’s documentation and API information.
The distinction matters because adapting how a model behaves and supplying the evidence it should answer from solve different problems. The accompanying Discord examples show API documentation help beside a Dino Dash game concept.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why retrieved text is not enough
Building a reliable RAG application requires more than putting search results into a prompt. Kublik identifies three difficulties, beginning with prompt sensitivity. The model must know what information to look for, where to look, and how to distinguish retrieved evidence from the conversation it has already had with the user. A prompt that blurs those roles leaves the model to infer which text should govern its answer.
The second difficulty is document-position bias. Kublik describes models paying excessive attention to the beginning of a document instead of consistently finding the most relevant information wherever it appears. She points to retrieval evaluations, including needle-in-the-haystack tests, as ways this weakness becomes visible. Having the right information somewhere in the context does not ensure that the answer will use it.
The third is the conflict between pretrained knowledge and supplied evidence. The model’s parameters may contain an answer that no longer matches the world. Asking who the current president of the United States is illustrates the problem: the task calls for up-to-date information, not simply the name most strongly represented in training. RAG must let external evidence guide an answer when the stored knowledge is stale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train the retrieval decision, then make the answer inspectable
Cohere’s response is to shape the model’s retrieval behavior through post-training. The intended sequence is:
- Decide whether external information is needed. Some requests can be answered from pretrained knowledge.
- Operate the retrieval system. Form search queries and retrieve relevant information when the request requires it.
- Use the results as grounded context. Bring that evidence into the ongoing conversation when generating the answer.
This makes the decision to retrieve part of the model’s trained behavior, alongside the ability to use what retrieval returns.
Fine-grained citations make that grounding inspectable. A user needs to see where a particular claim came from and whether its source deserves trust. Kublik describes citation behavior as a major training focus and associates it with lower hallucination and more reliable use of context. Citations provide a route to checking an answer; their presence alone does not replace that check.
Kublik reports best-in-class performance for Command R and R+ on standard RAG datasets including KILT. The talk does not supply the tasks, scores, or evaluation configuration needed to interpret that comparison precisely. The broader design claim is qualitative: models small enough to be affordable, with good token efficiency, can cover use cases that would otherwise require a larger pipeline of LLMs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Toolkit supplies the application around the model
Even with retrieval and citation behavior in the model, an application still needs to expose those capabilities. A useful RAG chat interface must retain conversation history, display fine-grained citations, accept document uploads, and connect to tools. Cohere spent time building that interface and open-sourced it on April 24, 2024 as Cohere Toolkit. The repository was subsequently archived on May 14, 2026, so it is a historical application reference rather than an actively maintained starter today.
Toolkit separates the interface, model access, and retrieval so that an application can change one without rebuilding all the others.
| Component | Role | Choices described in the talk |
|---|---|---|
| Interface and storage | Chat UI and conversation history | Next.js application; small SQL database |
| Model access | Connect the application to Command R models | Cloud providers, Cohere platform, local access, Hugging Face |
| Retrieval | Connect tools and data sources | LangChain-based example with document uploads and web search |
The example retriever is a starting point. The retrieval component can be extended with other tools and data sources, while the model component determines where inference happens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From retrieval to dependent tool calls
Training a model to operate vector databases and retrieval systems provides a path toward broader enterprise tool use. The next goal is to let the model work with tools beyond retrieval, ideally from their descriptions alone. Kublik presents zero-shot use of arbitrary tools as a direction of training, not a universally achieved capability.
The critical distinction is whether actions depend on one another.
| Tool-use pattern | Dependency | Example |
|---|---|---|
| Single-step | One action, or independent actions | Search for a document; send an email |
| Multi-step | Later actions use earlier results | Find → compare → summarize → email |
In the multi-step example, the system finds a document, compares it with another document, summarizes the comparison, and emails the result. It cannot produce the comparison summary before it has the comparison, and the email needs the resulting summary. The workflow’s dependencies determine the sequence.
That sequence need not be a fixed march toward completion. The system may retrieve from different sources repeatedly, inspect intermediate results, and correct errors. Kublik describes this as a loop and uses the term agents for software performing a sequence of actions in which each action builds on earlier steps. The relevant capability is not merely issuing several calls; it is using their results to decide what should happen next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Describe tools, inspect results, revise the plan
The multi-step API, described as released the week before the talk, exposes this behavior through tool descriptions. The developer supplies the available tools, what each does, and its parameters. A user request then gives the model a goal from which to construct a plan. This is the historical API presentation; the relative release date does not establish when multi-step capability first became available.
The execution loop connects planning to evidence:
- Select a tool call that advances the user’s request.
- Execute the call and return its result to the model, as illustrated by the Python loop on the slide.
- Inspect the returned content and decide whether the initial plan still fits.
- Continue or revise. If an API returns an error, retrying or changing the plan may be the next action.
In the document workflow, these returned results are what let the model progress from finding documents to comparing them and preparing the email. Retrying and replanning are intended behaviors, not guarantees that an unavailable API or another failure will be recovered from successfully.
The behavioral instructions live in a large multi-step preamble that Kublik points readers to on Hugging Face. It tells the model how to carry out the task, including how to proceed through intermediate results. The model’s trained prompt structure matters here: the model card warns that departing from its template can reduce performance. Open weights make these instructions and model behavior available for inspection, but do not make the surrounding orchestration irrelevant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tool provenance and production economics
Citations carry the grounding principle into tool use. Command R and R+ are trained to produce claims that users can verify through citations, making it possible to explain which tool supplied the information behind a response. For a workflow that crosses several data sources, that connection between a claim and its origin is a practical part of transparency.
Kublik claims Command R+ is competitive with Claude Opus and GPT-4 Turbo while being three to five times cheaper. The talk does not provide the pricing basis, workload, or exact model versions, so this is a historical cost comparison rather than a general deployment estimate. Kublik also reports Command R+ as close to or on par with GPT-4 Turbo on standard complex-reasoning benchmarks, without naming those benchmarks or supplying scores. The production proposition is that competitive capability at lower cost could make repeated tool-use steps more scalable.
The stated direction for upcoming releases is continued work on multi-step capabilities. No release dates or additional specific capabilities are promised: the focus remains improving the model’s ability to carry a task through dependent actions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The March 2024 announcement introducing Command R's retrieval-focused positioning.
Model specifications, licensing, inference examples, and grounding and tool-use guidance for the original Command R+.
An archived RAG application with a Next.js interface, conversation storage, model-provider configuration, and extensible tools.
The benchmark paper explaining shared Wikipedia grounding and evaluation of answers and supporting provenance.
The historical announcement of HuggingChat's tool integration using Command R+.
Further reading
- Multi-step tool use becomes the Chat API defaultDocumentation
The June 2024 changelog documenting the switch to multi-step tool use by default.
Full transcript
Loading transcript…