AI benchmarking and evaluation
Artificial Analysis
Artificial Analysis independently benchmarks AI models, agents, inference providers and chips, helping developers and enterprises choose technologies for their applications. Its public comparisons bring model capabilities together with speed and cost. Co-founders Micah Hill-Smith and George Cameron began the project in 2023 and launched it publicly in January 2024. The idea grew from Hill-Smith’s work on a legal AI research assistant: choosing a model for each stage required understanding the trade-offs between accuracy, performance and price. Hill-Smith is CEO and Cameron is chief product officer.
A defining part of its work is measuring the service developers actually receive from a model endpoint. Its performance benchmarks send the same tasks to competing endpoints under concurrent load, measuring response speed, latency and failures. Independent test accounts help check whether providers give benchmark traffic different treatment from ordinary users. The Artificial Analysis Intelligence Index combines evaluations across knowledge work, coding, science and general knowledge. Its AA-Omniscience benchmark examines factual reliability by rewarding correct answers, penalizing hallucinations and leaving abstentions neutral, distinguishing missing knowledge from confident mistakes.
For teams choosing models for specific work, Optima builds custom benchmarks from their tasks and examples or imported agent traces. Teams can evaluate answers and completed deliverables against grading criteria, then compare quality with cost and time per task. Artificial Analysis also provides advisory services, custom benchmarking and API data access to organizations adopting AI and companies building AI technologies. Its business therefore serves both users deciding what to deploy and providers evaluating what they have built.
2 talks
Newest firstLoading more talks and speakers…
2 speakers at AIE
Affiliations reflect their AIE event listings and recorded appearances, not necessarily current employment.

