AI Engineer World's Fair 2025
The Billable Hour is Dead; Long Live the Billable Hour?
Loading slides…
Read the talk
The Billable Hour Is Dead; Long Live the Billable Hour?
Classification, retrieval, and structured extraction show how AI can change consulting work—and why enterprise value still depends on schemas, validation, and reliable delivery.
From a talk by Kevin Madura and Mo Bhasin
Before you start: Basic familiarity with LLMs, structured data, and APIs is helpful; the article explains the retrieval and extraction workflows as they arise.
What changes when AI can do more of the work?
What happens to professional services when AI can perform more of the work that clients pay people to do? At AlixPartners, Mo Bhasin and Kevin Madura approach that question through an internal generative AI platform. Bhasin introduces himself as Director of AI Products, with earlier experience in an anomaly-detection startup and data science at Google. He reports two years of development with 20 engineers, reaching 50 deployments and hundreds of users. Madura works with companies, courts, and regulators on understanding emerging technologies. Their examples concern the practical work of consulting: preparing evidence, analyzing it, and delivering a useful answer to a business problem.
METR’s task-horizon research provides an opening signal of changing capabilities. The chart measures the length of tasks an AI can complete at a 50% success rate, with task length expressed as human-expert working time rather than agent runtime. Madura treats the rising curve as a rough proxy for disruption, not a forecast of consulting reliability. Software offers relatively verifiable tasks; knowledge work involves messier evidence and judgments. He expects those differences to make progress less steep and less uniform.
That distinction matters because professional-services firms organize themselves around different ways of extending senior expertise.
| Model | Who directs and performs the work? | Source of capacity |
|---|---|---|
| Junior-led | Senior staff direct larger junior teams | Additional people |
| Senior-led | Experienced practitioners remain directly involved | Domain experience |
| Proposed hybrid | Experienced practitioners direct AI-assisted work | AI extends expert capacity |
Madura describes AlixPartners as senior-led, with practitioners who bring 15–20 years of experience to daily delivery. His forecast is a hybrid in which AI helps reproduce and apply that expertise more widely. Invoking Dwarkesh Patel’s idea of an AI-first firm, he imagines something like 50 copies of a CEO. This is a conceptual picture of scaling knowledge, not a demonstrated ability to duplicate an executive’s judgment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Less preparation, more evidence
An engagement often begins with a pile of PDFs, databases, and spreadsheets. Whether the assignment is an acquisition, an investigation, or due diligence, someone must first ingest, normalize, categorize, and organize the material. Only then can an expert apply a playbook, generate hypotheses, and develop insights. Those activities support the final recommendation: solving the problem for which the client hired the firm.
AI’s clearest immediate effect is to compress that preparation. Madura illustrates the change as preparation falling from 50% of an engagement to perhaps 10–20% in terms of required human effort. The figures express his estimate of the shift, rather than a measured comparison across matched projects.
Consider his contract-review example: 5,000 contracts, each requiring 30 minutes of review. Time and cost force a team to prioritize, perhaps examining only the most valuable 20%. AI changes the feasible scope of extraction: the same questions can be applied across the whole corpus. In Madura’s framing, coverage expands to 100%; that describes which documents receive processing, not whether every extracted answer is correct.
The benefit is more than finishing the original sample faster. A team can examine vendor contracts or the customer base more broadly, identify additional savings opportunities, and leave experts more time for interviews. Reducing preparation can expand the evidence available for judgment, as well as reduce the labor required to assemble it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Individual time savings are not enterprise productivity
Bhasin introduces the deployment examples through a tension: enthusiasm for AI has not reliably translated into organizational results. He cites an 89% figure from the Spring 2025 Fortune/Deloitte CEO Survey. That figure includes organizations exploring, piloting, or implementing agentic AI; it is not a completed-deployment rate. He also cites an NBER finding of no significant impact on earnings or recorded hours.
The other findings concern difficulty converting projects into value. BCG’s figure is 74% of companies struggling to achieve and scale value from AI broadly, rather than GenAI alone. S&P Global reports 42% of companies abandoning most AI initiatives before production, distinct from its separate finding that an average of 46% of projects were scrapped. These describe different populations and outcomes, but motivate the same practical question: which workflows change the organization’s performance? Bhasin’s answer begins by distinguishing employee productivity from enterprise productivity. The three use cases target work that teams depend on, not simply a faster way for an individual to write or search.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Classify against the business taxonomy
The first use case is categorization. A ticket about a repeatedly restarting laptop needs routing to hardware support. In spend analysis, United Airlines belongs under travel. These are small decisions, but organizations need to make them repeatedly across inconsistent text. The older machine-learning workflow required text preparation—stemming and stop-word removal—followed by a classifier such as a support vector machine or Naive Bayes.
Structured outputs replace much of that bespoke classifier-building work with an LLM response constrained to the required format. Bhasin’s example classifies company names, including JD Factors, against the North American Industry Classification System, or NAICS. The output associates the vendor with a code, category, and explanation. He offers other cash management as an illustrative category, rather than establishing an authoritative NAICS assignment for that company.
A schema cannot supply facts the model does not know. For an unfamiliar company, the workflow adds a web query and appends the resulting company information before classification:
- Supply the company name and the target taxonomy, including category descriptions.
- Use a web tool to obtain missing company context.
- Add that context to the classification input.
- Return the category in the required structured format.
This separates two problems: finding out what a company does and mapping that activity to the business’s categories. Bhasin describes the reduced implementation burden as democratizing text classification.
He calls the approach unsupervised, but its deployment is not unchecked. Business partners work with engineers to establish acceptable accuracy, while the taxonomy carries the business context that a generic category list would miss. Successful projects turn skeptical users into internal champions, creating demand for the next application. A reliable classification step also becomes something the team can later chain into an agentic workflow. The output remains stochastic, so that reliability requires continuing validation.
Bhasin reports 95% accuracy categorizing 10,000 vendors, completing in minutes work that would have taken days, at an order of magnitude less cost. The talk does not specify the scoring method, taxonomy depth, runtime conditions, or cost accounting behind that deployment result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Get oriented quickly, then remove information handoffs
The second use case starts with an urgent assignment: 80 GB of internal documents arrive on Friday, and a court filing is due Monday. What did ACME release in 2020? What are its escalation procedures for reporting safety violations? Previously, someone might maintain an Excel index of documents received, missing, and stored in different locations, or attempt to find the answer through SharePoint Search.
The team’s retrieval-augmented generation, or RAG, application is designed to handle hundreds of gigabytes across PowerPoint presentations, documents, Excel files, CSVs, and other formats. Retrieval helps consultants become useful quickly when joining a project with little preparation time. Tool calls to proprietary third-party databases can supplement the document collection.
Those tools also address an organizational bottleneck. A licensed group previously had to retrieve information through a web interface, email it to another team, and wait for that team to analyze the spreadsheet. Bhasin’s team embedded the API specification and taught the LLM to call the API, reducing information-access delays that could otherwise take days. The useful change is the removal of intermediate requests and file handoffs within the authorized information workflow.
RAG then becomes a foundation for additional features, but the prompt box can encourage expectations beyond what retrieval provides. Asking it to reason across every document does not make an ordinary RAG system perform exhaustive analysis. Retrieving useful passages and processing an entire corpus are different operations. Bhasin’s prescription is to build those additional capabilities step by step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The schema defines what matters
The third use case turns unstructured documents into records. Madura uses a roughly 50-page credit agreement: extract the contract parties, maturity dates, and senior lenders instead of asking a reviewer to locate each field manually. Referring to Jason Liu and Pydantic, he describes the reusable pattern as a document plus a schema plus an LLM, surrounded by validation and supporting code. The business value resides in the schema: which facts the engagement needs, and why those facts matter.
A Python schema can make those requested fields explicit while allowing a document to omit information:
python
from datetime import date
from pydantic import BaseModel, ConfigDict, Field
class CreditAgreement(BaseModel):
model_config = ConfigDict(extra="forbid")
contract_parties: list[str] = Field(
description="Names of the parties identified in the agreement."
)
maturity_date: date | None = Field(
description="The stated maturity date, or null if not found."
)
senior_lenders: list[str] = Field(
description="Senior lenders explicitly identified in the agreement."
)
The schema constrains the output’s shape and field types. Checking that those values agree with the document remains a separate validation task. An investigation and an M&A engagement can reuse the same extraction capability while asking for different fields.
Madura describes applying extraction across thousands to hundreds of thousands of documents and reducing review from days or weeks to minutes. He does not give matched workload or runtime conditions for that comparison. The firm combines external products, including Box, with internally developed capabilities; the reusable element is the extraction workflow rather than a single application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use token likelihoods to direct review
To help users decide what to inspect, the internal extraction interface aligns token log probabilities returned by the OpenAI API with structured-output fields. It excludes JSON syntax and field names, retaining the tokens that form the extracted values. For the interest-rate field, the relevant text is LIBOR plus 1% per annum—not the key naming that field or the punctuation surrounding it. This makes the review signal specific to the answer the user needs to assess.
Madura describes aggregating these values as a “geometric mean of the logprobs.” The mathematical distinction matters: a geometric mean of token probabilities is exp(mean(logprobs)), whereas exp(sum(logprobs)) is their product. The spoken description does not settle the exact implementation. Either aggregation is a likelihood-based signal, not a calibrated probability that the extracted fact is correct.
The interface translates its field-level scores into green and yellow indicators. These give users an intuitive route into human review, rather than requiring them to inspect token statistics directly. For nontechnical users, seeing a document become usable structured data can be the moment the capability becomes clear. Making that experience trustworthy still requires substantial validation; the color cue helps direct attention but cannot replace checking the source.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adoption is earned one bug fix at a time
The final requirements are organizational as much as technical. Bhasin emphasizes people skills and close work with the business. The team prototypes in Streamlit and builds in React, showing progress through monthly demos. That cadence makes new capabilities visible, helps colleagues imagine uses for them, and supports continued investment.
Agents, MCP, and new models may attract attention, but Bhasin names NPS and ROI as the metrics that guide the work. Progress is earned through individual bug fixes and partnerships with the people using the system. The platform’s value depends on reliable delivery becoming part of how the firm operates.
Madura closes with a joke: once Excel-powered LLMs actually work, the industry will have reached AGI. It is a fitting handoff after three examples built around ordinary enterprise material—categories, documents, and spreadsheets—whose usefulness depends on getting the details right.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains task horizons measured in human-expert working time at a specified success rate. The live page also contains later chart updates.
Survey report distinguishing exploration, pilots, and implementation of agentic AI among CEOs' organizations.
Further reading
Dwarkesh Patel's speculative essay about how copying and coordinating AI workers could change firms.
- Measuring LLM ConfidenceArticle
Kevin Madura's explanation and code excerpts for matching token likelihoods to extracted fields and directing human review.
OpenAI examples for inspecting token log probabilities and scoring generated outputs.
Build structured extraction workflows with typed schemas, validation, and retries.
Full transcript
Loading transcript…