AI Engineer World's Fair 2024

What It Actually Takes to Deploy GenAI Applications to Enterprises

Arjun Bansal· CEO, Log10Trey Doig· CTO, Echo AI21:30

Loading slides…

Read the talk

What It Actually Takes to Deploy GenAI Applications to Enterprises

Echo AI’s customer-conversation pipeline shows how broad analysis becomes useful only when teams can inspect outputs, grade them against customer needs, and turn corrections into better models.

From a talk by Arjun Bansal and Trey Doig

Before you start: Familiarity with LLM prompts, fine-tuning, and basic evaluation metrics will help with the research and implementation sections.

Customer understanding breaks down at scale

How much of what customers tell a company does anyone actually examine? Support, sales, and customer success conversations contain more than the routine complaints that reach a dashboard. Trey Doig introduces Echo AI through an iceberg: familiar issues sit above the waterline, while a much larger body of business information remains underneath.

An iceberg diagram labels revenue opportunities, market insights, competitors, agent behavior, compliance violations, and product issues beneath the waterline.Open full source frame
Customer conversations hold insights beneath the surface.

The barrier is the work required to read those conversations. A small company can rely on its support staff and sales team to notice patterns and relay them to decision-makers. As the company succeeds and conversation volume grows, that same process becomes impractical. The organization gains customers while losing touch with what they are saying every day.

0:000:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From sampling to discovery

The usual response progresses from manual review to increasingly specialized software. Doig uses a sample of roughly 5% of conversations as an illustration of manual review, not a measured industry average. Reviewers check compliance, agent handling, and recurring themes; engineers then pull records across systems to investigate questions retrospectively. Once an issue is understood, the company builds software to track it.

ApproachWhat it examinesMain limitation
Manual reviewA small conversation sampleLeaves most interactions unseen
Retrospective scriptsData gathered for an investigationStarts after a question emerges
Purpose-built trackingKnown signals across conversationsRequires knowing what to seek
Generative analysisConversations for known and emerging themesStill requires trustworthy interpretation

By the time the first three approaches identify a problem, the fire may already be burning. Doig presents generative AI as a way to detect the smoke: he claims it enables 100% conversation coverage and can surface issues the company did not already know to search for. Coverage describes how much gets examined; it does not establish whether the analysis is correct.

2:182:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:18 · section reference included

One message can expose several problems

Consider the customer message about a router that arrived broken. The immediate extraction is contact intent: why the customer is asking for help. The same message also identifies the product and the condition on delivery, suggesting a possible supply-chain problem. Sentiment adds another dimension, covering both the customer and the representative handling the interaction. These are separate views of the same evidence, rather than a single label attached to the conversation.

Highlighted customer chat beside colored tags for contact drivers, product and supply chain, audience, sentiment, intent, and agent performance.Open full source frame
A customer message mapped to multiple categories of insight.

Doig then describes Wine Enthusiast, whose high-end wine refrigerators serve customers including retail locations. A new refrigerator model created a manufacturing problem that Echo AI surfaced through customer conversations. Doig reports that the platform identified the defect in real time, before it could continue for weeks. For a business that depends on expensive purchases and repeat customers, the useful output is an operational finding that can prompt action, not merely a negative-sentiment count.

3:584:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:58 · section reference included

The pipeline starts before the prompt

Echo AI’s processing begins with ordinary integration work:

  1. Connect to contact systems, ticket systems, and other conversation sources.
  2. Gather, normalize, clean, and compress the records for LLM prompts.
  3. Run the conversations through dozens of configurable analysis pipelines.
  4. Let customers specify what matters, initially writing prompts with Echo AI and eventually maintaining those prompts themselves.

The customer’s requirements therefore shape the analysis throughout its life, rather than appearing only in an initial configuration form.

That collaboration leads directly to the enterprise buyer’s question: can these insights be trusted as much as the judgments of business analysts, customer-experience leaders, or sales executives? Doig says Echo AI aims to show a new customer useful insights within seven days. He describes 95% accuracy as an aspiration supported by sampling and ongoing assessment, without defining a benchmark or scoring protocol. The commercial objective is durable trust that leads to renewals.

Log10 supports the work of establishing that trust. Engineers need to understand what their application generated, but solutions and implementation engineers also need to handle corrections while working directly with customers. A customer saying that an answer is wrong must be able to initiate a concrete inspection-and-feedback process.

5:295:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:29 · section reference included

What an accuracy problem looks like in production

Arjun Bansal introduces Log10 as infrastructure for improving application accuracy, with a longer-term goal of systems that improve their own prompts and models. He explicitly describes that goal as something the field has not yet achieved. The immediate problem is more concrete: deploying a prompt without a dependable way to measure its behavior exposes the business to failures that extend beyond an incorrect answer.

The Air Canada chatbot incident illustrates the stakes. A customer received inaccurate advice about retroactive bereavement fares. As the legal account of the decision clarifies, the February 2024 Civil Resolution Tribunal outcome was an award for negligent misrepresentation—not a general ruling that a company must enforce any policy its chatbot invents. Bansal also recounts a dealership chatbot being persuaded to offer a Chevy Tahoe for a dollar; the example establishes an inappropriate offer, not a completed sale.

Other failures are less spectacular but still damaging. A support chatbot suggested playing a game while a frustrated customer waited, missing the emotional context. A retrieval-augmented semantic search system can miss common-sense implications that are not explicit in its retrieved documents. Human review is the deployment gold standard in Bansal’s account, but its time and expense recreate the scaling problem that automated analysis was meant to solve.

7:457:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:45 · section reference included

The evaluator needs evaluation too

Replacing reviewers with an LLM judge introduces another source of error. Bansal identifies several recurring biases:

  • Self-preference: a model favors its own output.
  • Position: the first option receives an advantage because it appears first.
  • Verbosity: a longer answer looks better regardless of its quality.
  • Token diversity: varied wording becomes a proxy for merit.

These biases can make the evaluator respond to presentation rather than the rubric it is supposed to apply.

Log10’s Auto Feedback aims to combine human-review accuracy with the speed and cost of model-based review. In Bansal’s presented comparison, an ordinary judge can concentrate its predictions around one score despite variation in human judgments. Auto Feedback produces predictions that correlate more closely with the actual feedback. The desired property is not simply that the system returns a score, but that its scores track distinctions reviewers care about.

Once that signal is useful, it supports three different workflows: continuous quality monitoring, triage that directs limited human attention to valuable cases, and dataset curation for prompt improvement or fine-tuning. Evaluation becomes an input to improvement, rather than a report generated after deployment.

9:4710:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:47 · section reference included

Build the evaluator around human judgments

The training record contains four things: the application input, its output, the grading rubric, and a human’s feedback. Log10 explores three ways to use those records:

MethodHow human feedback enters the evaluator
Few-shot learningLabeled examples appear in context
Fine-tuningLabeled examples train the evaluator
Synthetic bootstrapping, then fine-tuningSeed labels support synthetic training data

The third approach tries to reduce the amount of human annotation needed before the evaluator becomes useful.

The companion study, Hybrid Evaluation: Scaling human feedback with custom evaluation models, examines TL;DR summary grading. Its experiments move from aggregate to annotator-specific models, from GPT-3.5 to GPT-4, from few-shot learning to fine-tuning, and then add synthetic bootstrapping. Bansal calls the combined result an improvement in evaluation accuracy. More precisely, the companion study reports a 45% reduction in grading RMSE, from 2.12 to 1.16, on its TL;DR summary-rating experiment. This is reduced error in predicting human scores, not a 45-percentage-point gain in classification accuracy.

The companion study’s bootstrapping experiment expands 50 human-labeled seed examples into 600 synthetic examples and comes within 0.045 absolute-error points of the result using 1,000 human-labeled examples. The practical attraction is a smaller initial annotation requirement for this summary-grading task. Bansal then reports that follow-up work using Mistral 7B and Llama 70B Chat matched GPT-4 and GPT-3.5 fine-tuned evaluation accuracy. The presentation does not specify the exact open-model versions or matching conditions.

11:4511:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:45 · section reference included

Put feedback beside logging and debugging

Log10 packages these evaluators into its Auto Feedback module. The surrounding platform provides logging, debugging, and evaluation, alongside prompt optimization and fine-tune management. Bansal describes a one-line integration between the application and its LLM SDK, with support for OpenAI, Anthropic, Gemini, several open-source SDKs, and application frameworks. That placement connects the quality signal to the requests and responses engineers already need to inspect.

13:5214:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:52 · section reference included

A broken TV and a network of downstream analyses

Back in Echo AI, Doig distinguishes codified analysis requirements from generative discovery. After transcription and cleaning, customers can ask a broad question such as why customers are canceling; the system generates an ontology of cancellation reasons rather than requiring every reason to be specified beforehand.

The demo account uses generated conversations. Doig opens one about a customer’s broken TV and examines its summary. Summarization looks like a simple feature, but Echo AI uses summaries in downstream analysis, so mistakes can affect more than the text displayed to a reviewer. The resulting prompt volume and throughput requirements motivate substantial self-hosting and continual training for different customer domains. Other insights beside the summary come from separate models and pipelines.

The Broken TV on Delivery conversation view shows a speaker timeline, transcript messages, and an AI Analysis panel with an overview and categorized insights.Open full source frame
Echo AI displays a conversation transcript alongside generated analysis.

Each insight corresponds to a customer question or requirement. That variation makes quality difficult to manage with one generic notion of a good answer. Engineers and solutions engineers use Log10 to inspect the generated prompts behind the visible results, connecting a customer-facing problem to the request that produced it.

14:5815:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:58 · section reference included

Make a correction easy to capture

Echo AI defines specific criteria for summary quality and uses Auto Feedback to grade each summary against them. In the example, the evaluator deducts one point and provides an explanation. Doig shows how a reviewer can change the point value and accept a human override. The workflow gives the reviewer a starting judgment to inspect rather than an empty scoring form.

At the data-record level, the same operation can preserve the automated judgment while adding the reviewer’s accepted score. This Python function illustrates that separation; calling it constructs a revised record without changing the input record:

python

from copy import deepcopy
from typing import Any


def accept_human_override(
    record: dict[str, Any],
    score: float,
    explanation: str,
) -> dict[str, Any]:
    revised = deepcopy(record)
    revised["human_feedback"] = {
        "score": score,
        "explanation": explanation,
        "status": "accepted",
    }
    return revised

The application input, generated summary, rubric, and automated feedback remain available for inspection. The added human feedback records the correction rather than silently replacing the model’s judgment.

Doig also points to a Mistral example as he explains the goal: collect high-fidelity human feedback with little effort from solutions engineers. Automated feedback continues in the background, while human corrections accumulate through normal review work. He says this has changed Echo AI’s process for building fine-tuning datasets.

17:1017:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:10 · section reference included

Inspect a failed summary, not just its score

The next conversation contains an obvious failure: instead of summarizing the interaction, the output reiterates instructions and resembles a system prompt. Doig opens the underlying record and observes that the output has been graded accordingly. The exact cause is not explained aloud, but the demonstration connects the visible failure, its underlying request, and its quality assessment in one investigation.

Doig reports that this process lets Echo AI track hallucinations and observe model drift with data, where it previously depended heavily on people sampling outputs. The operational goal remains customer trust: detect declining quality, understand individual failures, and retain corrections that can inform subsequent improvements.

18:2518:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:25 · section reference included

From feedback to a reported customer gain

Bansal reports a 20 F1-point improvement for one Echo AI use case using Auto Feedback; the accompanying slide links the result to 20 feedback examples and Log10’s Prompt Optimizer. Its chart compares Baseline, dspy, Fine Tune, and Log10. The presentation defers the detailed result to a case study without specifying the task, baseline score, dataset, or F1 averaging convention.

Slide pairs 20 feedback examples and Log10’s Prompt Optimizer with a reported 20 F1 point accuracy increase, beside a bar chart comparing Baseline, dspy, Fine Tune, and Log10.Open full source frame
Prompt optimization slide reports an increase of 20 F1 points using 20 feedback examples.

Because customer data cannot be shared in the presentation, Log10 also offers a public summarization and summary-grading application, with a live website and resources for adapting the example to other tasks. The example provides a way to explore the feedback workflow without exposing the conversations used in the customer deployment.

19:2219:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:22 · section reference included

Carry the workflow into the application

The UI is not the only entry point. Bansal says its functionality is available programmatically through Python and other-language SDKs, with flexible feedback types. The Log10 Python client and Log10 Cookbook provide the corresponding integration and example resources. His closing implementation sequence is:

  1. Create the evaluation task.
  2. Create feedback for that task.
  3. Run Auto Feedback locally with a simpler model.
  4. Fetch feedback from a more complex model hosted in Log10’s cloud.

This local/cloud workflow describes the platform at the time of the recording; current repository documentation does not establish identical hosted-service availability. The limited-time free access to advanced models mentioned at the close was a historical offer. The implementation path ends where the demo began: application outputs become inspectable records, records receive feedback, and that feedback becomes material for improving prompts and models.

20:1520:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:15 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…