Alex Shaw

Alex Shaw is a founding member of the technical staff at the Laude Institute, creator of Harbor, and co-creator of Terminal-Bench. His work gives developers a practical way to measure whether autonomous AI agents can complete realistic computer-based tasks, diagnose their failures, and systematically improve their performance.

Shaw studied computer science, mathematics, and German studies at Brigham Young University. His earlier work included internships on Microsoft’s Azure Service Fabric team and Apple’s computer-vision systems, alongside co-founding Lyvli, a coaching platform for creators. At Google, he worked on advertising recommendations and Conversion Lift experiments, including causal-inference bias and the reliability of controlled experiments.

At Laude, Shaw developed Terminal-Bench with Mike Merrill, initially creating evaluation tasks themselves before incorporating difficult problems from expert contributors. The benchmark uses the command line as a gateway to software development, systems administration, data analysis, and other consequential computer work. Their November 2025 launch of Terminal-Bench 2.0 and Harbor combined more rigorously verified tasks with infrastructure for running evaluations across cloud-hosted containers. Shaw subsequently co-authored the Terminal-Bench research paper, which describes 89 realistic tasks with dedicated environments, human-written solutions, and verification tests.

• Agent development as empirical machine learning. Shaw treats agent behavior as probabilistic: model selection, prompts, tools, and environments interact in ways that cannot be understood through code inspection alone. Repeated agent rollouts produce measurable outcomes, while familiar machine-learning problems such as overfitting and reward hacking become central engineering concerns.

• Realistic, renewable benchmarks. Terminal-Bench emphasizes expert-authored tasks, independently checkable results, and problems that resemble actual work. Shaw sees static leaderboards as vulnerable to contamination and optimization against the test, making stronger verification, transparent submissions, and continually refreshed tasks essential.

• Harbor and sandboxed agent evaluation. Harbor combines an instruction, an isolated computing environment, an agent, and a verifier. Its standardized environments support parallel trials across different models and agents, capture execution trajectories, and can be reused for evaluation, supervised training, or reinforcement learning.

• Agentic MapReduce. Shaw also uses Harbor to distribute operational work across sandboxed agents and consolidate their results. In a public demonstration of distributed execution, he applied this pattern to coding-agent traces: identify human corrections, group recurring mistakes, and turn those failures into better evaluations.