AI Engineer World's Fair 2026
From Tokenmaxxing to Trusted Throughput — Mingsheng Hong, Ironclad
Loading slides…
Read the talk
From Tokenmaxxing to Trusted Throughput
Mingsheng Hong explains how Ironclad connects AI coding costs to software that survives automated checks, human judgment, and customer use—and why faster generation moves the real bottleneck into review and CI.
From a talk by Mingsheng Hong
At a glance
Ideas worth remembering
Use token dashboards to investigate adoption gaps and unusual bursts, with comparisons grounded in each team’s work. Do not reward consumption.
Measure delivered value alongside cost. Ironclad’s complexity-weighted merged PRs are an evolving proxy, not a complete measure of customer value.
Trusted throughput combines objective checks, human engineering judgment, and evidence from customer use. A merge is only an intermediate delivery measure.
Expect abundant generation to move constraints into review and CI. Measure readiness-to-submission time and retries, remove flaky tests, and resist giant batched PRs.
Bound agent retry loops, arrange prompts for cacheable prefixes, prune growing context, and turn recurring lessons into shared practices.
Buy common infrastructure and build company-specific working knowledge, while evaluating ambiguous agent investments in light of their downstream review and CI costs.
Token usage is a smoke detector, not a leaderboard
Mingsheng Hong, VP of engineering focused on AI at Ironclad, opens with a management failure hiding inside an ordinary dashboard. In a reported story from Amazon, an employee created a voluntary view of AI token usage. Some engineers then competed to reach the top. Hong is explicit that he does not know whether leadership encouraged the competition; the consequential point is that publishing a number can create an incentive even without attaching a formal reward to it.
Ironclad also tracks token usage and cost by team and individual, but treats the dashboard as a “smoke detector.” Surprisingly low usage can expose a pocket where access, training, or confidence is missing. A sudden burst can justify investigating what changed. Neither observation establishes employee performance, and rewarding the largest number would turn consumption—the input cost—into the goal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cost controls come after the adoption hump
This advice has a boundary: it primarily applies after a team has made AI tools readily available and established real usage. A company still provisioning access or persuading engineers to experiment may undermine adoption by introducing aggressive cost controls too early. Ironclad had only crossed that threshold over the preceding couple of quarters, and roughly half the audience identified with the next phase: adoption exists, so spending has become a serious concern.
Adoption itself cannot be solved entirely by executive instruction. Some engineers told Hong that the pride of handcrafting code had been replaced by reviewing “AI slop code.” That objection describes a real change in the work, not mere resistance to a tool. Leaders need to sit with affected teams, understand where the friction comes from, and preserve demanding technical work through which engineers can continue developing judgment and skill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trust grows by testing familiar ground first
Ironclad builds AI products for legal contracting, where lawyers, procurement teams, and other business users need to move contracts faster without losing control of risk. A lawyer may first run conversational search over contracts they already know. Familiar material provides a reference point: if the answers match expectations, the lawyer can cautiously expand to unfamiliar contracts or higher-stakes workflows such as redlining and anomaly detection.
AI-assisted engineering follows the same progression. Generated work has to earn confidence from engineers and leadership before it reaches customers, then survive contact with actual customer use. Hong calls the result trusted throughput: code reviewed and validated internally, then ultimately validated in customer deployments. The objective is therefore better return on token spending, not austerity for its own sake.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure value before trying to minimize cost
Cost measurement starts with the available usage data. A team centered on one coding tool may get enough detail from its vendor dashboard. Ironclad uses several tools, so it built extraction pipelines and dashboards that combine vendor data, then aggregate and break down spending by team and individual. This makes cross-tool usage visible instead of leaving each vendor’s portion isolated.
Moving directly from that dashboard to cost cutting evaluates only one side of the return. Ironclad pairs cost with a proxy for delivered value, then looks for whatever prevents generated work from becoming shipped work. As code generation becomes abundant, those constraints increasingly appear downstream in code review, continuous integration, and merging.
Dashboard comparisons also require local context. A platform infrastructure team may use AI differently from a UI team and derive value through different work. Regular review should answer two practical questions. Adoption gaps: Where is usage unexpectedly low, and does that reveal a problem worth helping with? Anomalies: Where did spending suddenly jump, and was the burst legitimate? The resulting lessons can update shared practices without stack-ranking employees.
Lines of code provide the useful analogy. LOC can describe activity, but maximizing it would punish an engineer who deletes unnecessary code and leaves the system simpler. Token usage has the same shape: worth observing, terrible as a direct performance target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From generated volume to complexity-weighted merges
Hong uses lines of code as an analogy for a metric that counts output without saying whether it forms a useful change. He then describes Ironclad’s measurement evolution from open to merged pull requests. Open pull requests grouped work into proposals and showed a pronounced increase, but proposals can remain experimental or never ship. Merged pull requests moved the metric closer to delivery because the code had at least passed through the path to the codebase.
The next failure appears in a concrete comparison. A 10-line change might represent a concurrency bug that took substantial investigation to find and fix. A thousand-line change might be boilerplate that was easy to generate but expensive to review. Counting each as one merge treats visibly different work as equivalent; counting lines makes the boilerplate look more valuable. Ironclad therefore asks one or two LLMs, using a crafted prompt, to assign each merged PR a T-shirt-sized complexity score and uses that score as a weight.
This is an evolving proxy, not a settled definition of value. The talk supplies neither a reproducible scoring rubric nor numerical weights, and complexity does not necessarily equal customer value. The complexity weighting is intended to distinguish a concurrency fix from boilerplate; the talk does not report evaluated discrimination results, but the remaining gap is why trusted throughput needs qualitative evidence beyond the score.
What evidence makes throughput trusted? The comparison below separates three forms of validation that catch different failures. Automated checks test predefined properties, people judge qualities that resist simple rules, and customer experience reveals whether internally acceptable work holds up in production.
The three forms of evidence behind trusted throughput
Read the diagram as text
- Objective checks. internal validation — Test coverage, predefined security checks, and canary rollout practices test known requirements.
- Human judgment. internal validation — Code and design review assess quality, clarity, maintainability, architectural fit, and security design.
- Customer evidence. external validation — Production incidents, rollbacks, complaints, usability friction, and bug tickets reveal real-world performance.
- Trusted throughput. outcome — High-quality engineering output trusted internally and by customers.
- Objective checks → Trusted throughput: supplies objective evidence.
- Human judgment → Trusted throughput: supplies contextual judgment.
- Customer evidence → Trusted throughput: supplies production evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Abundant generation moves the bottleneck into review and CI
Once AI makes PR creation easier, review and merging become the scarce stages. A tempting workaround is to stop splitting changes. If the regression suite takes an hour, an engineer may reason that 10 small PRs imply 10 hours of waiting and submit one large PR instead. The apparent CI saving creates a review problem: larger diffs demand more human attention, spread that attention thin, and make careful review less likely.
Ironclad uses AI as the first review pass rather than the final authority. Automated review can flag simpler issues such as style problems or missing test coverage before a human sees the change. Human reviewers can then spend their limited attention on code quality, architectural fit, and security design. The engineering team keeps final accountability.
CI creates a second queue. More numerous, smaller PRs increase test demand, while flaky tests force engineers to babysit runs and press retry. Assigning an agent to babysit merely exchanges human waiting for token spending. Both are workarounds around the same unstable pipeline, and the repetitive delay also damages morale.
The useful measurement is end-to-end waiting, not just the nominal duration of one CI run. Ironclad tracks wall-clock time from a PR being ready to submit until it is submitted, plus the number of retries required to pass. If one CI run normally takes an hour but a typical PR takes two or three hours to get through, the unexplained gap is the red flag. That observation directs platform and developer-experience investment toward flaky-test removal and CI capacity.
Where does the extra AI-generated work accumulate? The flow below shows review and CI as parallel downstream constraints rather than a universal ordering. Increasing generation loads both systems; slow CI can then encourage batching, which sends larger, harder diffs back into review.
The bottleneck moves downstream
Read the diagram as text
- Abundant AI code generation. production — Engineers can create and split changes into more pull requests.
- Review queue. validation bottleneck — Human attention becomes scarce as pull-request volume grows.
- CI queue and flaky retries. validation bottleneck — More test runs increase waiting and repeated execution.
- Larger batched PRs. anti-pattern — An attempted workaround reduces CI invocations but makes review harder.
- Merge and delivery. delivery — Work proceeds after the workflow’s required review and CI checks clear.
- Abundant AI code generation → Review queue: more PRs require judgment.
- Abundant AI code generation → CI queue and flaky retries: more PRs require test capacity.
- Review queue → Merge and delivery: clear required review.
- CI queue and flaky retries → Merge and delivery: pass required checks.
- CI queue and flaky retries → Larger batched PRs: slow runs encourage batching.
- Larger batched PRs → Review queue: larger diffs increase review burden.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bound waste, improve the loop, and build only local advantage
The operating framework combines three mechanisms. Guardrails set budgets and quotas, track usage, and define anomalies that trigger attention. Working practices improve how engineers use AI. Learning loops bring leaders and engineers together to review patterns, refine limits, and feed useful discoveries into institutional knowledge. Automated alerts identify known conditions; human review still catches patterns the predefined rules miss.
A test-and-repair agent loop makes the need for guardrails concrete. The agent generates a PR, runs tests, modifies the code or tests after a failure, and retries. A hard limit on loop steps makes the process finite. Hong does not prescribe a universal retry count; the design requirement is to stop a broken workflow from consuming tokens indefinitely.
Two prompt and context practices reduce repeated work. Stable prefixes: when a vendor supports prompt caching, fixed material should come first and varying material afterward—for example, a stable system prompt above the changing user prompt. Context pruning: as a session grows, engineers should summarize or compact accumulated history rather than repeatedly carrying everything forward. Coding tools may automate compaction. Hong presents these practices as improving token efficiency and output quality, without quantifying either effect.
The build-versus-buy rule is deliberately plain: buy common, non-differentiating infrastructure such as IDE and CI capabilities; build the knowledge specific to how the company works. Ironclad’s example is an internal playbook of carefully written prompts for distinct tasks such as small bug fixes, new UI features, and refactoring. Sharing those prompts lets teams reuse and improve their working methods.
The boundary remains ambiguous for systems that mix commodity infrastructure with local workflow. Ironclad was exploring a cloud-based “builder agent” wrapping coding tools while also considering external vendors. Hong leaves that decision unresolved rather than claiming every company should build its own agent.
The closing lesson is to plan for second-order effects. More code generation changes review load, CI demand, pull-request shape, and developer experience. Teams should make build-versus-buy decisions early, invest in downstream capacity before queues force bad workarounds, and keep instrumenting the path from ready PR to customer outcome. Maximizing token ROI means expanding the organization’s ability to validate and deliver software alongside its ability to generate it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The official talk page provides the recording, chapter navigation, timestamped transcript, and concise section summaries for revisiting individual mechanisms.
Related talks
- Leadership in AI-Assisted Engineering
Extends the organizational argument with evidence-based measurement, psychological safety, and attention to bottlenecks across the full software lifecycle.
- ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, eBay
Develops the downstream review problem into a deterministic framework for estimating the human burden created by each pull request.
- How to Quantify AI ROI in Software Engineering (Stanford Study / 120k Devs)
Provides a complementary outcome-oriented approach to AI coding ROI and similarly warns against treating raw token consumption as productivity.
Full transcript
Loading transcript…