AI Engineer World's Fair 2024

Cohere for VPs of AI

Vivek Muppalla· Director of Engineering, Cohere16:11

Loading slides…

Read the talk

Cohere for VPs of AI

Enterprise AI depends on more than model quality: retrieval, customization, deployment control and operating cost determine whether a useful prototype can become a production system.

From a talk by Vivek Muppalla

Before you start: Basic familiarity with language models and document search will help; retrieval-augmented generation and reranking are explained as they appear.

What does an enterprise need from a model?

How do you put an AI model to work on business data while keeping control of where it runs? That is the starting problem for Vivek Muppalla’s introduction to Cohere: trustworthy enterprise models, developed with partners, that can deploy across cloud providers rather than requiring customers to commit to one infrastructure ecosystem.

The research background helps explain the product emphasis. Muppalla points to cofounder Aidan Gomez’s coauthorship of Attention Is All You Need, Phil’s background at DeepMind and Oxford, Nils’s expertise in BERT, and Patrick’s coauthorship of Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Generation and retrieval are both central to the resulting lineup.

The generative family has two main choices: Command R, presented as the inexpensive enterprise workhorse, and Command R+, a larger model for more demanding reasoning, tool use and retrieval-augmented generation, or RAG. The retrieval family supplies embeddings and Cohere Rerank, which adds a relevance-ranking step to a retrieval pipeline.

ComponentRole in the application
Command RGeneral enterprise generation at lower cost
Command R+More complex reasoning, tool use and RAG
EmbeddingsRetrieve candidates by semantic similarity
RerankReorder retrieved candidates for the query

This separates two decisions that are easy to conflate: which model should generate the answer, and which information should reach that model.

Three-column slide groups Command R/R+ under generative models and Embed 3 and Rerank 3 under advanced retrieval models, with use cases and language support below.Open full source frame
Cohere’s model lineup: Command R/R+, Embed 3 and Rerank 3.
0:290:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:29 · section reference included

Evaluate the workload, then design for its constraints

Cohere’s development principles begin with enterprise-specific evaluation. Academic benchmarks provide one view of model performance; the stated release goal is an evaluation suite built with partners around health, HR, finance and other customer workloads. Repeated evaluation across model versions is meant to reveal whether a release improves the tasks customers actually need.

The next principle is operating efficiency. The objective is not simply to produce the largest model, but to make the model useful, affordable and straightforward to run. Customization follows from the same practical concern: a broadly capable model may still lack the behavior or domain knowledge a particular enterprise needs. Muppalla describes domain adaptation by retraining a base model on enterprise data, including full retraining, and contrasts that with the then-available self-service option he describes as retraining the last few layers.

Data provenance and privacy form the fourth principle. Muppalla says Cohere applies enterprise standards to its training data, offers indemnification for IP claims, and does not use customer data to train its models. That assurance concerns Cohere’s own model training, distinct from customer-requested adaptation. It is also a historical assurance: the later Enterprise Data Commitments distinguish SaaS training opt-out controls from private and third-party cloud deployments where prompts and generations are not sent to Cohere.

The fifth principle is deployment flexibility: major cloud providers, a customer’s own VPC, or on-premises infrastructure. Together, the five principles turn model selection into a system decision—task performance, operating efficiency, customization, data handling and the location of compute all have to fit the application.

Five orange rows list model performance, efficiency and scalability, customization, data provenance and privacy, and deployment flexibility beside a green panel describing corresponding approaches.Open full source frame
Five enterprise needs paired with Cohere’s approach.
2:282:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:28 · section reference included

RAG citations and multilingual operating cost

The capability priorities follow the enterprise workloads: multilingual support, RAG and tool use for agentic applications. Muppalla names HotpotQA, Bamboogle and Berkeley Function Calling as evaluations where Cohere performs well. These are qualitative performance claims in the presentation, not a basis here for a numerical comparison between models.

A more concrete integration feature is built-in RAG citations. Cohere’s models and APIs can return citations with an answer, reducing the work required to connect generated statements to supplied sources. That does not remove the application’s responsibility to retrieve documents and construct the request; it moves citation generation into the model/API path instead of requiring developers to build that behavior themselves.

For multilingual applications, Muppalla cites strong FLORES performance and attributes lower operating costs partly to the tokenizer. Tokenization matters economically because the amount of text represented by a token affects the token volume processed for a workload. His practical objective is to let a customer use the same model across geographies while controlling total cost of ownership.

4:585:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:58 · section reference included

Finding the right attention paper

The retrieval demonstration makes the pipeline problem tangible. After describing embedding quality on noisy data and low cost as product strengths, Muppalla introduces a search over arXiv papers: when was the attention paper associated with Aidan Gomez published? The query sounds simple, but the records combine titles, dates, author lists and full paper text. That mixture resembles enterprise data much more closely than a collection of uniformly formatted passages.

An embedding represents that mixed content in a vector space. A query asking for a particular author’s paper and its publication date may not align cleanly with the representation of the whole record. A semantically related result is not necessarily the precise document needed to answer the question.

Reranking separates candidate discovery from final relevance ordering. Cohere’s reranker is a cross encoder: it evaluates a query together with each candidate document and reorders the already retrieved set before the generator receives its context. The procedure is:

  1. Retrieve candidate documents using the existing search system.
  2. Score the candidates against the user’s query with the reranker.
  3. Select the most relevant candidates for the generative model’s context.

The important boundary is the retrieved set. Reranking can improve its ordering; it cannot promote a document that the initial search never returned.

The demo compares three columns: lexical search using OpenSearch, embedding search, and Cohere Rerank. All can return results for the transformer-paper query, but the relevant document appears at different positions. In the displayed rerank column, Attention Is All You Need appears first. That ordering matters because a RAG application may pass only a small part of its retrieved set to the generator.

Cohere-Rerank demo with a query above three result columns labeled Lexical Search (OpenSearch), Embeddings Search and Cohere Rerank. Attention Is All You Need appears first in the rerank column.Open full source frame
The reranking demo compares lexical search, embedding search and Cohere Rerank results.

Reranking complements chunking rather than replacing it. Chunking determines the units available for retrieval; reranking helps choose among the resulting candidates. A small Python function makes the handoff explicit: given the arXiv candidate records and scored candidate indices, preserve the records while selecting them in relevance order.

python

from collections.abc import Mapping, Sequence
from typing import Any


def select_context(
    candidates: Sequence[Mapping[str, Any]],
    scored_indices: Sequence[tuple[int, float]],
    limit: int,
) -> list[dict[str, Any]]:
    if limit < 0:
        raise ValueError("limit must be nonnegative")

    ranked = sorted(scored_indices, key=lambda item: item[1], reverse=True)
    return [dict(candidates[index]) for index, _ in ranked[:limit]]

Here, scored_indices contains the reranker’s candidate indices and relevance scores. Keeping each selected record intact preserves its title, authors, date and text for the answer-generation step. The function performs selection; the cross encoder supplies the relevance judgment.

Better selection also changes the cost calculation. If a small set of relevant documents is sufficient, the generator needs fewer input tokens. Muppalla emphasizes input-token expense as a major cost driver, but whether it dominates—and whether reranking lowers the total bill—depends on the workload and the reranker’s own cost. The mechanism is narrower and more useful than a universal savings claim: send less irrelevant context to the generator.

6:306:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:30 · section reference included

Putting the pipeline inside the enterprise

Deployment options extend from Cohere’s managed SaaS API to cloud AI services including SageMaker, Bedrock and OCI, alongside private cloud and on-premises deployments. Muppalla also describes compliance with standards customers request, without naming particular standards in this discussion. The architectural choice is where the model runs relative to the organization’s data, compute and security requirements.

Developer integration is the other half of that deployment story. Muppalla names LangChain and LlamaIndex integrations, then describes an open-source toolkit with connectors for ingesting data. The intended benefit is a shorter path from enterprise data sources to a working application while retaining control over the system being built.

9:329:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:32 · section reference included

Partners, private fine-tuning and customer choice

The first audience question pairs two concerns: how do partnerships with Accenture and McKinsey work, and why do customers choose Cohere over alternatives? The first answer concerns the last mile between a general model and a deployed business application. Enterprises need customization, and established partners bring existing customer relationships and knowledge of the relevant business domains. Muppalla describes their role as co-developing products and helping close those implementation gaps.

For customer choice, his emphasis is control over data and compute. Private deployment is useful not only for inference but also for fine-tuning inside the customer’s own cloud environment, so enterprise data can remain within that ecosystem. He gives HR, healthcare and models adapted to proprietary codebases as examples of application categories where that control matters. These are descriptions of customer needs, rather than named case studies or quantified competitive wins.

11:0811:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:08 · section reference included

When should classification use a specialized model?

The next question asks whether the Classify endpoint is still the recommended way to build a text classifier. Muppalla answers through development stage and production scale rather than treating an endpoint as a universal recommendation. An off-the-shelf generative model offers a fast starting point; moving into production creates a reason to reassess whether a general model is adequate and economical.

SituationDecision to examine
Early prototypeUse a general model to get started quickly
Production workloadCompare quality, operating cost and maintenance
Very high-volume classificationConsider a purpose-built model

Muppalla uses a hypothetical workload of tens of thousands of TPS to motivate a purpose-built classifier, not to report measured throughput. A heavier general model may serve several needs, while a specialized model can be a better fit for a narrow, repeated task. Both remain options; the decision depends on the workload and the cost of maintaining the solution.

A follow-up asks whether training such a classifier still produces a transformer model, just a more specialized one. Muppalla confirms that it does. Specialization here changes the model’s task and operating role; it does not imply abandoning the transformer architecture.

13:2713:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:27 · section reference included

A larger context window, with retrieval still doing useful work

The final question returns to a concrete limitation: an audience member had evaluated Cohere in the previous year and found the input context limits small. Muppalla says the then-current generation offers 128K-token input context windows, with further expansion under investigation.

His assessment is that this capacity should suffice for most applications at that point. Capacity, however, is not a guarantee of answer quality across the entire window. The earlier retrieval demonstration remains relevant: having room for more documents does not make every document useful, and selecting the right context still affects what the model can answer and what the application costs to run.

15:1015:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:10 · section reference included

Resources

From the talk

  • The 2024 release note describes Command R's RAG and tool-use positioning and 128k context capacity.

  • Cohere ToolkitRepository

    An archived RAG application repository with configurable retrieval, model providers and data integrations. Archived May 14, 2026.

Updates since the talk

Transcript download
Full transcript

Loading transcript…