Model Evaluation & Benchmarks

Pre-deployment testing pattern where frontier AI models are assessed by independent third parties for dangerous autonomo...

AI safety company behind the Claude model family — including Claude Opus, Sonnet, Haiku, and the restricted Claude Mytho...

AI safety research organization focused on detecting and evaluating deceptive capabilities in frontier AI models.

AI coding agent platform for professional software teams, built around a proprietary Context Engine that semantically in...

Recurring dynamic where AI models approach maximum scores on benchmarks, rendering them unable to distinguish between sy...

Open-source Apache-2.0 LLM evaluation framework by Confident AI with 50+ metrics spanning RAG, agents, multi-turn conver...

Cognition's commercial autonomous AI software engineer with full shell and browser access, SaaS and VPC deployment optio...

AI research institute tracking compute trends, model capabilities, and publishing data-driven analyses of AI progress.

Google's open-source CLI wrapping the Agent Development Kit (ADK) to automate the full AI agent development lifecycle —...

METR's primary benchmark measuring frontier AI autonomous software task completion, calibrated against 140 human experts...

A 2,500-question expert-level benchmark curated by ~1,000 specialists to measure AI capabilities where frontier models s...

An open-source LLM evaluation framework by the UK AI Safety Institute with 100+ pre-built evals for safety, coding, reas...

Open-source LLM engineering platform (MIT-licensed, 21k+ GitHub stars) covering observability traces, evaluation, prompt...

Observability and evaluation platform for LLM applications, providing tracing, prompt testing, and experiment comparison...

Contamination-resistant LLM coding benchmark that continuously collects new competitive programming problems from LeetCo...

Open-source MIT-licensed data framework for building RAG and document agent applications on top of LLMs, with 38k+ GitHu...

Nonprofit research org that evaluates frontier AI models for dangerous autonomous capabilities before deployment.

A portable specification for defining AI agent evaluation tasks with standardized environment setup, instructions, and s...

A benchmark of 15,908 multiple-choice questions across 57 academic subjects for evaluating LLM knowledge, now effectivel...

An open-source platform for autonomous AI coding agents with Docker-sandboxed execution, multi-model support, and a Pyth...

Open-source Apache-2.0 evaluation framework for RAG pipelines and LLM applications by ExplodingGradients (YC W24), provi...

AI benchmark suite from METR for evaluating autonomous AI agent capabilities on real-world research engineering tasks.

Nonprofit AI safety research organization focused on AI control, alignment, and pre-deployment evaluation, home to the t...

Persistent sandboxed dev environments for AI agents with git-style state management and built-in SWE-bench integration.

A benchmark evaluating whether AI agents can resolve real-world GitHub issues by generating code patches that pass repos...

Open-source MIT-licensed LLM evaluation and tracing framework by TruEra, now maintained by Snowflake, combining OpenTele...

Open-source benchmarking tool for vector databases, covering 30+ databases with CLI and visual interface; maintained by...

METR's open-source platform for running AI agent evaluations and elicitation research, now deprecated in favor of Inspec...