CentaurBench
The best player is not always the best coach.
Most benchmarks ask which model works best alone. At work, models often help another worker. Those are different skills — and they produce different rankings.
Wrong question: “Which model is best?” · Right question: “Best for which role, on which task?”
Automation
The model completes the task end-to-end on its own.
Augmentation
The model writes assistance text for a fixed GPT-3.5-Turbo worker, which produces the deliverable.
Methodology
Two regimes, one evaluation pipeline
Automation: the model finishes the task itself. Augmentation: the model writes assistance text for a fixed GPT-3.5-Turbo worker. Blind pairwise judges score the deliverables with task rubrics.
Tasks
Seven economically grounded professional tasks
Drawn from 
GDPval and the 
Anthropic Economic Index — except tax preparation, designed by the authors. Select a task to open its prompt brief.
Main results
Automation and augmentation produce different rankings
10-run panel averages. Lower rank is better. In five of seven tasks the regime winners differ; menu planning and tutoring share GPT-5-Mini across both.
5 of 7 tasks
The best model in automation is not the best model in augmentation.
3 of 7 tasks
The unaided worker beats every assisted condition — poor assistance can hurt.
Shared winners
GPT-5-Mini wins both regimes on menu planning and tutoring; other tasks diverge by role.
Role-swap scatter
Each point is a model on reverse rank (inverted mean rank — higher = better). Points above the diagonal are stronger in augmentation than in automation.
Augmentation · Mean Rank Across Runs
Automation · Mean Rank Across Runs
Compare
A/B any two models on the same task
Three example task × usage-regime presets with real assistance text or deliverables, rubric spines, and judge rationales. Open Compare to customize models and regimes.
Go deeper
Task-Usage Scoreboard and qualitative browser
Open the scoreboard for per-task × usage-regime ranks and rubric meanings, then browse prompts, assistance text, outputs, and judge rationales for all runs in one place.
Engage with Our Research
CentaurBench
We welcome feedback, questions, and collaboration ideas.
Contact Us10-Run Result
Browse aggregate results across 10 runs (all candidates). Lower rank is better; error bars show standard error.
10×
Panel findings
Main Findings Across 10 Runs
These summaries aggregate rank-of-ranks across the full ten-run panel.
AUG
Usage regime
Augmentation · Mean Rank Across Runs
Ten-run panel average · lower rank is better · ± SE
AUTO
Usage regime
Automation · Mean Rank Across Runs
Ten-run panel average · lower rank is better · ± SE
Run-to-Run Stability
Model averages summarize rank-of-ranks across all seven tasks for each replicate. Click a row for per-run detail.
AUG
Stability
Augmentation Stability
Per-model mean rank across runs · click a row for detail
AUTO
Stability
Automation Stability
Per-model mean rank across runs · click a row for detail
Single-run Result
Single-run snapshot of model ranks across tasks (all candidates). Use the Run filter above; pick Average or an individual judge below. Lower cell = better rank. For paper-level aggregates see .
AUG
Usage regime
Augmentation
Assistant models guiding a fixed worker · lower rank is better
AUTO
Usage regime
Automation
Models solving the task end-to-end · lower rank is better
Compare models
Side-by-side panels on a shared task and run — each side picks its own model and usage mode (automation or augmentation), so same-model cross-regime compares are supported. Rubric spine, deliverables, and assistance text.
A
Model A
Model A
B
Model B
Model B
Head-to-head judge rationales Direct pairwise matchups for this pair, when available
Task-Usage Scoreboard
For readers who want a deeper read on what each rubric dimension means for each task, plus task × usage-regime rankings under Average or individual judges. Select a model on the left to load its dimension scores on the right. Automation vs augmentation use the same filters above; assistance text terminology is unchanged.
LB
Leaderboard
Task × usage-regime rankings
Select a model to update rubric scores
RD
Rubric dimensions
Dimension scores & meanings
Task-specific and general rubric attributes for the selected model
Qualitative Browser
Read the exact assistance text, worker output, rubric scores, and judge rationales for any task-model-regime cell.
TX
Evidence