Building strong safety evaluations

Anthropic published 'Building strong safety evaluations' on 13 November 2026, a methods piece on designing, running, and reporting AI safety evaluations. It builds on psychology's replication reforms and Anthropic's own system-card experience, arguing 'a safety claim is only as sound as the eval behind it,' and warning that 'researchers have repeatedly found that AI evals are easy to game, don't always measure what they claim to measure, and aren't documented well enough to reproduce.' It distinguishes observational from causal eval designs, stresses construct validity (e.g., for self-harm conversations 'refusal is often the wrong construct'), and recommends multidisciplinary teams, preregistration, sandboxed runs with real-time monitoring, and tighter reporting.

Safety evaluations underpin both deployment decisions and regulatory claims; this is a leading frontier lab's practical guidance on making them credible — directly relevant to anyone building or auditing AI safety evaluation programmes.

Map your model/procurement safety-evaluation methodology against Anthropic's recommendations on construct validity, causal design, and evaluation reporting.