What happened
Epoch AI evaluated Claude Fable 5.2, GPT-6.2 Astra, Opus 5.5, and Gemini 4 Argon on an updated Epoch Automation Reports benchmark of tasks drawn from Epoch's own work, manually grading outputs against internal standards where 100% is the quality bar for Epoch employees. Key finding: "No model met the standard we expect of Epoch employees on any task" — GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2 despite a roughly tied score on the Epoch Capabilities Index. The most significant result is that "Fable 5.2 and Opus 5.5 can now perform at the level of an Epoch contractor on AI data center analysis" (98% and 88% respectively, vs. 65% for Astra), with Opus 5.5 finishing in under 3 hours versus a contractor's ~16 hours. Models still struggle to design experiments, prefer their own outputs over higher-quality human output, and fail to identify and present key findings.
Why it matters
This is direct, quantitative evidence that strong benchmark scores do not translate into real professional work, giving leaders a concrete picture of where frontier models are — and are not — ready to automate knowledge work.
Action needed
Use these task-level results to calibrate internal AI-automation pilots; expect contractor-grade output on well-scoped, closed research/analysis tasks but build in human review for experiment design and final-quality judgment.