AI Engineer World's Fair 2024

E-Values: Evaluating the Values of AI

Sheila Gulati· Managing Director, Tola CapitalNischal Nadhamuni· Co-Founder and CTO, Klarity28:13

Loading slides…

Read the talk

E-Values: Evaluating AI Against the Goals and Values It Serves

As AI moves from isolated predictions to autonomous workflows, evaluation must connect model behavior to user outcomes, business requirements, and the values those systems express.

From a talk by Sheila Gulati and Nischal Nadhamuni

Before you start: Basic familiarity with machine learning evaluation and large language models is helpful; no knowledge of Klarity is required.

Can we measure performance against our goals?

Can we evaluate an AI system’s performance in relation to the goals we set for it? As more work becomes automated and agents take on more responsibility, that question becomes a prerequisite for understanding what we have built. A system can produce impressive outputs without doing what its users actually need. Evaluation has to connect observed behavior to intended outcomes.

Sheila Gulati and Nischal Nadhamuni approach this problem as an investor and an operator: Gulati founded Tola Capital; Nadhamuni is Klarity’s co-founder and CTO. Gulati opens by noting Klarity’s newly announced $70 million Series B. The company’s ambition is to extend the coordination that software brought to internal business systems into external relationships. Customers and partners still negotiate individual documents, leaving important commitments scattered across one-off agreements. Klarity seeks to make those documents part of an ongoing, automated relationship—a foundation for what Gulati calls exponential organizations.

Gulati’s earlier work at Microsoft included leading database and developer-platform businesses, co-leading enterprise strategy, and advocating for Azure inside a company organized around Windows. That experience shaped Tola’s original thesis: a new generation of applications would be built on the cloud. AI applications now present another platform transition, but with a harder evaluation problem.

Cloud infrastructure favored scale. Data centers required capital that existing search, office-software, or retail businesses could supply. Buyers could compare speeds and feeds, performance, functionality, and price. AI retains many of those constraints—training and inference costs, scarce chips, and specialized engineering talent—but adds a more diverse field of models. Open-source development, academic research, and partnerships with large technology companies create contenders whose usefulness cannot be reduced to infrastructure specifications.

0:270:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:27 · section reference included

From a correct prediction to a useful agent

AI reflects the understanding, intentions, and preferences of the people who build and use it. That becomes more consequential when agents represent individuals. Gulati traces a progression from models consuming the good and bad of internet data, through emergent capabilities and curated datasets, toward an anticipated era of self-learning and self-sufficient systems. Her urgency follows from that trajectory: evaluation needs to improve before full automation makes its deficiencies harder to contain.

For narrow AI, the evaluation question resembles a hammer meeting a nail: did it perform the specified task? An image classifier has a relatively bounded target. Broad AI introduces several dimensions that must be considered together.

DimensionEvaluation question
Capability and intelligenceCan it perform the required reasoning?
Domain understandingDoes it understand this field?
Values alignmentWhose values does it express?
SafetyWhat does avoiding harm mean here?
Context and end-user awarenessDoes it understand whom it serves?
ModalityDoes it work across text, images, video, and their combinations?

The values question includes individual versus societal preferences and present versus future expectations. Safety also depends on context: people can disagree about what constitutes harm. End-user awareness cannot remain an afterthought when the purpose of the system is to serve that user.

Radar chart with overlapping colored polygons across capability/intelligence, domain specificity, values alignment, safety, context and end user awareness, and modality.Open full source frame
Broad AI requires evaluation across six dimensions.

These dimensions enlarge the evaluation problem beyond what simple task scores capture. Gulati frames the next part of the discussion around weaknesses in existing tools; Klarity’s operating experience then shows how those weaknesses appear in a product.

5:085:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:08 · section reference included

A leaderboard is not the user’s workflow

A fixed benchmark creates an optimization target. If the task is to solve for X, a model developer can concentrate on producing the right answer to X without establishing that the model understands the broader problem. Gulati uses an AP exam as the example: passing may indicate knowledge of the subject, or preparation for the kinds of questions that have appeared on that exam. The leaderboard alone does not distinguish them.

One response is dynamic evaluation data. Gulati describes Microsoft Research work that generates unpublished synthetic images and moves objects between them. Changing the scene changes the correct answer, allowing tests of spatial reasoning, visual prompting, and object recognition without repeatedly exposing the same test images. The description closely matches Microsoft’s A Dynamic Benchmark for Image Understanding project. The mechanism is to regenerate the task instances so memorizing a fixed set of answers becomes less useful.

Even without memorization, benchmark success may fail to transfer. A model can perform well on MMLU and still answer a particular business question incorrectly. Gulati points to FinanceBench, crediting Patronus AI and Stanford, as an example of financial questions exposing weaknesses that general benchmarks miss. The practical response is to put the user’s task at the center of evaluation and redesign UX and feedback systems to capture what that user needs. Evaluating the experience is difficult, but omitting it leaves the product’s purpose unmeasured.

7:578:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:57 · section reference included

What a score cannot tell us

Evaluation is a proxy for the task we care about. It seeks evidence about correctness; it does not provide a complete account of what a neural network has learned or how it reaches an answer. Interpretability addresses a different part of that gap by examining internal mechanisms. For Transformer-based LLMs, Gulati describes tracing information through the network to understand how pieces contribute to an answer. She anticipates more of this work alongside reinforcement learning in GPT models.

Technical capability is only part of what a model inherits from its creators. It also reflects values, whether those choices are explicit or not. Personalization makes the issue concrete: a model of Sheila could keep reinforcing Sheila’s beliefs, drawing her further into her existing worldview. The analogy is the news echo chamber, where personalization can intensify polarization. An agent that represents someone must therefore be evaluated for more than how closely it mirrors that person.

Several choices remain open:

  • Appease or challenge: Should the system reinforce the user’s view, question it, or let the user choose?
  • Present or aspirational values: Should it reflect society as it is or the society people hope to create?
  • User or model-maker responsibility: Should individuals select their own values, and which obligations remain with the model’s creators?

Gulati presents these as unresolved design questions. They broaden the definition of success before the discussion turns to a concrete enterprise application.

10:4410:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:44 · section reference included

The document is only the beginning of the task

Klarity automates cognitive, non-repetitive back-office work. These workflows resist a simple fixed sequence of instructions because a person must interpret a document before deciding what to do. Nadhamuni describes roughly eight years of work on this problem, predominantly involving PDFs in enterprise operations.

The tasks include revenue recognition, matching invoices to purchase orders, and processing tax withholdings in different languages. A finance or accounting team must extract meaning from unstructured documents while following a tightly regulated process. Matching requires aligning information across two documents; tax processing may require understanding both unfamiliar layouts and another language.

Document quality adds another layer of difficulty. Rotated pages, poor scans, graphs, images, and tables can all disrupt processing. Simply handing such a document to ChatGPT does not reliably complete the workflow. Nadhamuni reports that Klarity invested tens of millions of dollars in a stack for handling these documents. The accompanying slide shows the settled checklist of document-processing caveats, including searchable text, PDFs, tables, diagrams, and OCR.

Photographed presentation lists five document-processing caveats, with green check marks beside each item.Open full source frame
Document-processing limitations involving searchable text, PDFs, tables, diagrams, and OCR.

Klarity’s customers at the time were primarily B2B SaaS companies, many near the conference venue, with finance and accounting teams as the main users. The company was beginning to expand beyond that base. This customer concentration gave evaluation a specific target: the documents, rules, and outcomes those teams actually handled.

13:3113:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:31 · section reference included

Why familiar ML evaluation stopped being enough

Nadhamuni dates Klarity’s founding to 2016 and describes four pivots between 2016 and 2020, followed by a final move toward finance and accounting that found product-market fit. Rebuilding around generative AI over the preceding couple of years accelerated the business; he reports more than $90 million raised.

The team had spent more than six years working with traditional machine learning. Supervised learning supplied familiar evaluation tools: train/test splits, F1, ROC curves, and shared tasks such as sequence labeling and SQuAD. Initially, it was not obvious why generative AI should require a fundamentally different approach.

Nadhamuni reports that, after replatforming, Klarity had processed more than half a million customer documents, with more than 15 production LLM use cases and more than 10 LLMs under the hood. A single use case could involve several models working together.

The first complication was nondeterministic performance: repeatedly uploading the same cognitively demanding PDF could produce different responses. The next was the expansion of the product itself. Classification, regression, and random forests usually fit established interaction patterns. Generative models made entirely different experiences possible, and those experiences required different definitions of quality.

  • Workflow documentation: Klarity Architect takes a recording of someone doing their job and creates a business requirements document. Nadhamuni describes a typical output as a ten-page Word document with a flowchart and images, comparable in form to a consulting deliverable.
  • Natural-language analytics: A user can ask how a contract population has evolved over time and request a stacked bar chart. The output combines interpretation of the question, analysis, and presentation.
  • Document extraction: A system locates consequential information in tables or legal language buried inside a document. Here, extraction accuracy remains central, but the information can take very different forms.

Each experience creates useful capabilities for customers while making evaluation less reducible to a single predictive score.

15:1815:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:18 · section reference included

When evaluation takes longer than the feature

Faster implementation changed the economics of evaluation. Klarity’s earlier deep-learning workflow required dedicated annotation teams and GPU training. Nadhamuni contrasts that cycle with generative feature development:

Development periodReported feature timingEvaluation consequence
Earlier DNN workflowFive to six monthsTwo to three weeks for evals was acceptable
Generative AI workflowDaysOne to two weeks for evals can become the bottleneck

Nadhamuni reports that Klarity launched Document Chat within 12 hours of ChatGPT’s launch. Evaluation that once occupied a small fraction of development could now dominate it.

Model selection added uncertainty. Nadhamuni says small MMLU differences could matter for Klarity’s demanding cognitive tasks, but the ranking was not dependable: models advertised as better on MMLU sometimes performed worse on internal benchmarks. General benchmark scores could help identify candidates, yet they could not settle which model worked best for the customer workflow.

18:3318:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:33 · section reference included

Test the experience before building the full evaluation

Klarity’s response starts with permission to be imperfect during feature discovery. Nadhamuni describes the company’s evaluation practice as still developing, rather than a solved framework. High-quality evals are expensive, so requiring a complete suite at the start of every experiment can waste effort on experiences users will reject.

A new generative feature is almost its own product-market-fit problem. Document chat, natural-language analytics, and automatic video watching introduce unfamiliar ways of working. A recommendation engine, by comparison, often starts with greater confidence that the interaction pattern is useful. Klarity therefore front-loads user testing and delays substantial evaluation investment until the experience has shown promise. Features that fail at the UX stage can stop there. That sequencing does not remove the requirement to build evals before entering production and scaling.

19:4119:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:41 · section reference included

Work backward from the outcome

Once a feature is headed for production, the direction of reasoning changes. Start with the experience and work backward toward measurements. Choosing F1 first and hoping it correlates with user value reverses that relationship. An individual eval does not have to represent the whole experience, but the evaluation suite should reflect it in aggregate.

Klarity walks down three levels:

  1. Business outcome: Identify the end-user result and the business value the feature should create.
  2. Adoption and utilization: Decide which observable behavior indicates that people use the feature.
  3. Feature health: Measure lower-level properties such as JSON adherence and variability in generated outputs.

For an extraction feature, a small health check could examine whether repeated responses contain a required string field and whether the extracted values agree:

python

import json


def extraction_health(responses: list[str], field: str) -> dict:
    values = []
    for response in responses:
        try:
            record = json.loads(response)
        except json.JSONDecodeError:
            continue
        if isinstance(record, dict) and isinstance(record.get(field), str):
            values.append(record[field])

    return {
        "total_responses": len(responses),
        "valid_records": len(values),
        "distinct_values": len(set(values)),
    }


responses = [
    '{"invoice_number": "INV-104"}',
    '{"invoice_number": "INV-104"}',
    '{"invoice_number": "INV-140"}',
]
print(extraction_health(responses, "invoice_number"))

This illustrative check separates structural validity from output variation. It cannot determine which invoice number is correct: a stable, well-formed answer can still be wrong. That is why health measurements sit below the user outcome in the evaluation stack.

In production, customers supply their own labels and requirements. Klarity annotates customer data during user acceptance testing (UAT) and builds accuracy measures for the particular use case. Matching documents and extracting fields need different definitions of correctness. User feedback then helps close the loop, but silence is not positive feedback. The provider still has to measure rigorously and monitor data drift—for example, whether incoming documents differ substantially from the population already evaluated.

20:5321:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:53 · section reference included

Share evaluation data and constrain the search space

Customer-specific tests are a starting point, but a sufficiently common use case can justify shared evaluation infrastructure. Nadhamuni reports surveying more than six synthetic-data providers and trying several before Klarity built its own generation stack. The investment begins when a use case is large enough and enough customers share it to make customer-agnostic evals worthwhile.

The difficult part is keeping synthetic data representative. Klarity assigns people to monitor whether generated documents resemble customer documents distributionally. Nadhamuni says the company had not made that validation fully automated or quantitative. A second check compares model accuracy: unusually strong or weak performance on synthetic data can indicate that the generated population is not a useful stand-in for real work.

Prompt engineering creates another scaling problem. Klarity has features for free text, tabular extraction, matching, and table composition, each requiring customer-specific prompts. Across a large customer base, Nadhamuni describes the resulting inventory as thousands or tens of thousands of prompts. Automated prompt engineers (APEs) replace manual prompt writing, but introduce their own model-selection problem. Different LLMs can perform differently across APE tasks and customers, multiplying the combinations to evaluate.

Klarity reduces that search space by using one LLM for APE tasks, based on observed commonality in which models perform well. The choice is revisited as new models appear, but it remains fixed while the team iterates elsewhere. Nadhamuni acknowledges that a broader search might recover another percentage point of accuracy; he presents that as a possible gain, not a measured improvement, and judges the grid-search infrastructure too expensive to justify. Fixing one dimension makes the overall project easier to manage.

22:1422:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:14 · section reference included

Evaluate what might become possible next

Production evals measure what the company does now. A separate, lighter set can explore workflows it hopes to support several years ahead. Model capabilities can advance faster than a product roadmap anticipates, so a wish list of future tasks helps a team recognize new opportunities. These exploratory tests do not need MMLU- or BBH-level rigor for workflows no customer has requested yet.

Nadhamuni reports that Klarity formed an initial assessment of GPTV, the vision-capable GPT model discussed in the talk, in about an hour: it seemed good at check marks and perhaps less good at pie charts. The result was a quick organizational mental model of what to investigate next, rather than a production acceptance decision. Lightweight evals serve that discovery purpose alongside the more rigorous tests needed for deployed workflows.

The case study closes with a historical hiring invitation following the Series B: Klarity was recruiting across AI, back-end, front-end, and go-to-market roles.

24:1624:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:16 · section reference included

The responsibility that comes with defining success

Returning to the broader question, Gulati emphasizes the disproportionate influence of people building early in a platform transition. Their choices help establish what the industry will measure and reward. Reinventing misleading benchmarks, adding depth to evaluation, and treating capability, context, and values together are therefore practical responsibilities for the engineering community.

That work benefits from sharing unsuccessful approaches as well as successful ones. Klarity’s account is useful because it exposes decisions, constraints, and unfinished problems. Gulati couples this openness with curiosity, empathy, mutual assistance, and examination of the builders’ own values. Evaluation standards emerge from people making choices together; those people need to inspect their assumptions as well as their models.

Call to Action slide with three vertically connected items: Innovate and Reinvent Benchmarks, Empower the Community, and Introspect Our Value Systems, each with explanatory text.Open full source frame
A call to reinvent benchmarks, empower the community, and introspect value systems.

Gulati invokes Brian Christian’s The Alignment Problem and paraphrases the possibility of AGI as a mirror that reveals which values are distinctively human. Those values also change over time. The question is therefore not only whether AI reflects us accurately, but what kind of society its builders want to help create. Her closing appeal—to remain curious, empathetic, kind, fair, and intelligent—ties the opportunity to create new intelligence to accountability for what that intelligence serves.

25:2625:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:26 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…