What happened
Researchers (Jack Hopkins, Dipika Khullar, Fabien Roger) published 'Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets' (ICML 2026), demonstrating that amplifying the reasoning-direction task vector in weight space causes models to reveal hidden behaviors and secrets acquired during training up to 10x more frequently than the unmodified reasoning model.
Why it matters
This is a novel model-manipulation technique with a working method: anyone with access to both a base instruct model and its reasoning-distilled sibling's weights can construct an amplified model that leaks training-time secrets or backdoor behaviors that black-box auditing would otherwise miss — relevant both as a red-team/audit tool and as a potential extraction vector against open-weight model releases.
Attack vector
Given a non-reasoning instruct model M and a reasoning-distilled model R, an auditor (or adversary with model-weight access) computes a 'reasoning task vector' τ = θ_R − θ_M and applies it with amplification factor α > 1 to produce an 'overthinking' model θ_O = θ_M + α·τ. This amplifies the model's propensity to 'think out loud,' surfacing hidden information, secrets, or unintended behaviors acquired during training far more frequently than querying the base reasoning model directly.
Affected systems
Reasoning-distilled language models (generic technique applicable to any model with a paired non-reasoning/reasoning-distilled variant)
Mitigation
Model developers should incorporate reasoning-amplification/task-vector probing into pre-deployment red-teaming and auditing pipelines to detect hidden behaviors before release; treat model weights as sensitive even when only distilled/instruct variants are distributed publicly.