AI Engineer Summit 2023

The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex

Chris White· CTO, PrefectBryan Bischof· Head of AI, HexBrittany Walker· Principal, CRV35:16

Loading slides…

Read the talk

The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex

Prefect and Hex show how AI investment becomes product work: remove workflow friction, evaluate concrete tasks, preserve the core experience, and design for failure.

From a talk by Chris White, Bryan Bischof and Brittany Walker

Before you start: Basic familiarity with APIs, SQL, and data workflows will help; no machine learning training experience is required.

Where AI fits in an existing business

Where does AI fit when a startup already has customers demanding features, a product roadmap, and hiring and fundraising to manage? At the AI Engineer Summit 2023, Brittany Walker, a principal at early-stage investor CRV, puts that question to two people building from different starting points. Chris White is CTO of Prefect, which provides a workflow orchestration developer tool and remote orchestration service. Bryan Bischof leads AI at Hex, a data science notebook platform. Both have math PhDs: White studied non-convex optimization before moving through data science into developer tools; Bischof brings roughly a decade of data science and machine learning experience.

Wide view of three people in white armchairs, with small tables between them and audience silhouettes in the foreground.Open full source frame
Three participants seated onstage for the panel discussion.

For Prefect, the connection starts with its users. Data engineers and data scientists already orchestrate data movement, and semantic search, retrieval, and summarization introduce more of that work—including possible changes to ETL around vector databases. Prefect is general purpose, so those use cases do not require an entirely new product category. White then reduces the production problem to a familiar dependency: an expensive, brittle, nondeterministic remote API. Much of the tooling needed to operate data APIs should transfer. The team also creates Marvin, a separate repository that gives it a reason to experiment publicly and keep learning.

Hex starts from the analyst’s work. Data science combines business knowledge, creativity, and sometimes difficult programming. Remembering matplotlib syntax consumes attention without necessarily improving the analysis. Removing that tedium creates room for the parts of the job that require judgment. Bischof also expects AI assistance to become table stakes for data platforms, making the investment both an opportunity to improve the workflow and a requirement for remaining useful.

0:210:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:21 · section reference included

Choose the friction to remove

Prefect gives its two AI efforts different success criteria. Marvin is an open-source project the company maintains rather than sells; its return includes direct conversations with people experimenting with LLMs. White says the head of AI speaks with users multiple times a day. The core product has a narrower test: what happens when a workflow fails, and how quickly can a user understand and recover from it? Dashboard error summaries help users triage failures. Relevant measurements include time to find the right page and time to debug the workflow.

Hex’s charter is to make the product feel magical, but its example is concrete: a long SQL query containing several common table expressions, or CTEs, is awkward to work with. CTE Explode breaks the query into separate notebook cells and chains them together. Bischof describes it as straightforward to build and something he had wanted for eight years. The useful capability is the transformation of an existing working artifact into a form that is easier to inspect and edit.

For example, consider a query that totals paid orders by customer and keeps customers whose total reaches 100:

sql

WITH paid_orders AS (
    SELECT customer_id, amount
    FROM orders
    WHERE status = 'paid'
),
customer_totals AS (
    SELECT customer_id, SUM(amount) AS total
    FROM paid_orders
    GROUP BY customer_id
)
SELECT customer_id, total
FROM customer_totals
WHERE total >= 100;

An exploded arrangement would put the paid-order selection in a cell named paid_orders, the aggregation in a downstream cell named customer_totals, and the final filter in a cell that reads that result. These teaching names and values illustrate the feature Bischof describes: preserve the dependency chain while making its stages individually accessible. This is the kind of small, repeated inconvenience Hex uses to decide where AI assistance belongs.

4:535:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:53 · section reference included

Small teams, distinct returns

Hex begins with one hacker building a promising prototype, then expands to a couple of people. Bischof reports keeping the team around five people despite a roadmap large enough for twenty engineers. That constraint is deliberate: it forces prioritization. Prefect hires Adam to own most AI work while leaving participation open to others; one interested engineer effectively transfers to that team to work on AI full-time.

White reports that Prefect’s small AI additions have already improved core-product time to value, without giving a numerical improvement. Regular conversations with companies beginning to productionize AI provide a separate return. Hex looks for a different signal: how often users reach for Magic. Repeated use across cells and projects would indicate that assistance is becoming part of the workflow. Supporting every part of the platform is therefore a design goal, not a claim that universal adoption has already happened.

7:237:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:23 · section reference included

Build the integration that matters

One of Bischof’s first questions on joining Hex in February is how the team evaluates its AI features. His ML experience makes objective framing and evaluation prerequisites for useful iteration. He recalls a much smaller evaluation-tool market then, compared with his estimate of more than twenty-five companies by the panel. Hex chooses to build its own evaluations because they need deep hooks into the platform and should run as close to production as possible, including using production where feasible.

Vector infrastructure gets a different answer. After six or seven years working with vector search, including FAISS and early Pinecone, Bischof does not want to build another vector database. Hex works with LanceDB, then invests heavily in a customized retrieval implementation for its retrieval-augmented generation, or RAG, pipeline.

ComponentHex’s choiceReason
EvaluationsBuild internallyDeep production integration
Vector databaseUse LanceDBAvoid rebuilding complex infrastructure
Retrieval behaviorCustomize heavilyFit the application’s context needs

Buying the storage layer leaves substantial application engineering to do. The boundary follows where the complexity creates value.

Prefect likewise has no interest in building a vector database. White says its experiments with Chroma and Lance are not yet in production. He describes a common progression: explore with a framework, extract the prompts that work, then call the API directly without the experimental framework in the middle. At that point, existing API tooling may already provide the required monitoring and observability. For the integrations White describes, Prefect uses its existing stack and makes no new tooling purchases.

There are still gaps worth filling. Bischof, a Weights & Biases alumnus, wants experimentation and iteration tightly connected to LLM observability. He is watching Rivet by Ironclad for its approach to experimentation; a strong connection to observability is something he would like to see, rather than an integration he says Hex has adopted. White’s priority is the machine-to-machine interface: natural-language inputs are useful, but backend processes need typed outputs they can consume. He sees function calling as a step toward that contract.

10:2210:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:22 · section reference included

The prototype hides the long tail

For Prefect’s core AI additions, White reports no unexpected tooling or productionization obstacles. The harder problem is organizational enthusiasm. Engineers propose using AI for everything, including monitoring itself. The team has to bring the discussion back to specific sources of user friction and faster access to useful information. Scope control remains an ongoing task.

Hex encounters a related problem in estimates. Early AI experiments can work impressively on a few cases, making a feature look nearly finished. The distance from those successes to robust behavior is much longer than the prototype suggests. Bischof responds by turning enthusiasm into concrete evaluation cases:

  1. Start with the requested capability: what should Magic do?
  2. Ask for a specific prompt and the situation in which it will be used.
  3. Establish the expected behavior for that case.
  4. Ask whether a newly hired data scientist with little company context could do the same task.

That last question tests whether the request is feasible with the available context, rather than treating a broad wish as a specification.

15:4716:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:47 · section reference included

A separate experiment, or part of the product?

A separate repository gives Prefect room to learn without committing its entire product to a new direction. Marvin still creates maintenance obligations, but even low adoption would have taught the team how to build interfaces to LLM APIs. Its actual traction brings new contacts working on AI and adjacent data problems.

The core product needs a sharper boundary. Because Prefect orchestrates arbitrary workflows, helping users write arbitrary code creates effectively unbounded scope. White sees little advantage in competing with OpenAI or GitHub at that task. Prefect explores putting AI directly into its SDK, then rejects the direction as too broad and insufficiently additive. Narrow improvements to the existing workflow offer a clearer reason for the feature to exist.

Hex draws its boundary around a single product experience: Magic augments Hex. During one sprint, the team creates Crystal Ball, an experience Bischof describes as capable, polished, and pleasant to use. But it pulls users away from the core Hex workflow. The CEO identifies that it is creating a separate ecosystem for Magic. Hex kills Crystal Ball despite its appeal, because the goal is to improve Hex for all its users rather than split the experience into two products.

19:1519:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:15 · section reference included

Roll out to users who can expose the limits

Hex begins with a private beta and expands toward public beta as quickly as possible. Its early users are engaged, technically capable, and prepared to work with stochastic behavior inside already complicated data science workflows. The team then broadens participation across technical ability, so the product does not become useful only to the people best equipped to work around its limitations.

Marvin has a simpler release path: the team builds until it is proud of the repository, makes it public, and tweets about it. Prefect’s enterprise features require more care. Financial-services and healthcare customers are especially sensitive to data leaving their environment. White describes opt-in features with tooltips explaining that activation sends some data to a third-party provider. The historical error-summary announcement makes the account distinction precise: existing accounts had to opt in, while new accounts had the feature enabled by default. Consent and disclosure therefore need to be understood at the account level, not reduced to a blanket claim that every deployment was opt-in.

23:3523:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:35 · section reference included

Test the contract and the task

After launch, Bischof uses Magic throughout his working day—including to analyze Magic’s own product performance. That creates a feedback loop between direct experience and conventional product analytics: using the feature exposes friction, while analysis helps locate patterns of success and failure. The recursive quality is amusing, but the method remains ordinary data science applied to the product.

Prefect initially tests untyped string outputs with another LLM performing semantic comparisons. White acknowledges problems with that approach, while finding it useful. Moving toward typed outputs in Marvin makes tests easier to specify: an output has an expected type, and a known case can also have an expected value. Typed handshakes make components easier to compose and test, though structural validity does not establish semantic correctness. This is the panel’s historical Marvin design discussion; the repository now describes Marvin 3.0 built on Pydantic AI, rather than the implementation used in 2023.

Walker then asks about improvement over time, mentioning fine-tuning, RLHF, and RLAIF as possibilities—not techniques either company establishes it uses here. Bischof turns to the opposite problem: behavior can change underneath a deployed application. He recalls a paper associated with Matei Zaharia about changing model behavior. The likely reference is How is ChatGPT’s behavior changing over time?, first submitted in July 2023 and comparing March and June versions, rather than a paper published in June. Bischof reports that his evaluations began failing after a provider-side change even though he had not changed the endpoint he called. An unchanged endpoint need not imply an unchanged model version: OpenAI’s 2023 API announcement documented automatic upgrades of generic model names alongside dated versions. That explains the possibility without identifying Hex’s actual configuration.

The response is to define what performance means for the task. Bischof argues against substituting latent-distance distributions, KL divergence, or BLEU similarity for a clear account of whether an output does what the user needs. His preferred approach is to reduce a task to explicit yes-or-no checks wherever possible. Nuanced tasks may demand substantial work to find those checks; difficulty defining them is a reason to examine the task more closely. He frames this as established ML practice, recalling years of criticism of BLEU for these outputs and advocacy for composability. White connects that process back to orchestration: a changing, nondeterministic experimentation workflow still has structure that can be managed, even when the orchestrator does not choose the evaluation metric.

25:4425:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:44 · section reference included

Make agent workflows observable

When the panelists question each other, Bischof asks what workflow engineering can teach people building agents from DAGs and structured request-response chains. White starts with failure modes:

  • Runaway processes: consider how a workflow can continue beyond its intended bounds.
  • Unusual inputs and outputs: install observability early enough to catch odd behavior near its origin.
  • Caching: use it where it helps, while recognizing that it introduces hazards of its own.
  • Returned code: prioritize monitoring and debuggability when generated code is immediately evaluated or executed.

The operational layer becomes especially valuable as behavior grows harder to inspect inside individual components.

White then asks about hallucinations. A user may notice a strange result or see code fail, but the interface builder still has responsibility for reducing those failures. Bischof describes searching for advice and repeatedly finding recommendations to improve retrieval-augmented generation. That also matches Hex’s experience: Bischof attributes a reduction in Magic’s hallucinations to more carefully designed, application-specific retrieval, without reporting a reduction metric. He does not disclose the mechanism, saying it is absent even from his own book. The useful engineering boundary is clear: retrieval design matters, but this discussion does not provide an algorithm to reproduce or establish that hallucinations have been eliminated.

30:5631:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:56 · section reference included

The work behind the capability

White closes by challenging the reflex to put a chat interface everywhere: “chat is a product. AI is a tool.” His preferred investment is in machine-to-machine interfaces that let developers incorporate AI directly into their systems. That follows the same concern as typed outputs: make the capability usable inside a larger workflow.

Bischof’s closing warning is about the work itself. Exciting capabilities do not make the implementation process uniformly exciting. Much of it is tedious, and substantial data engineering lies between a promising experiment and a useful production feature. He sees that as a continuation of machine learning practice: the result can be rewarding, but getting there requires sustained attention to the unglamorous parts of the system.

33:5934:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:59 · section reference included

Resources

From the talk

  • Python framework for structured outputs and AI workflows, with current installation instructions and examples.

  • Ironclad's introduction to its open-source visual environment for composing, testing and running AI graphs.

  • The official repository for LanceDB's embedded retrieval library.

Updates since the talk

  • Managing MagicArticle

    Hex's November 2023 guide to improving SQL assistance through metadata, table retrieval and controls for string hallucinations.

Transcript download
Full transcript

Loading transcript…