The most useful sentence in Anthropic’s Claude Opus 4.6 safety materials is attached to a result the company says it cannot trust.
Anthropic ran an evaluation called Subversion Strategy to test whether a model could behave badly at a carefully chosen rate: rarely enough to escape pre-deployment testing, but often enough to act during a much larger deployment. In its Sabotage Risk Report, Anthropic says Opus 4.6 was “clearly under-elicited at this task” and declines to put weight on the result. METR’s external review later confirmed through private correspondence that the model’s performance was low. The risk report does not provide the number.
Anthropic’s admission carries more information than the missing score. A low score can describe the model or the setup. Here, the company could not confidently separate the two.
What the evaluation was trying to measure
The simplest example in the Subversion Strategy paper runs for 100 rounds. The first 10 are audited. The remaining 90 are not. The model has no memory between rounds and receives no information telling it which round it is in.
Always attempt the harmful action and the auditor catches it. Never attempt it and the model accomplishes nothing. The winning strategy requires the model to act at a low, calculated rate.
For this version of the game, the optimal rate is 2.53 percent. That produces a 69.7 percent chance of acting successfully at least once without being caught.
The game couples strategic planning to probability calibration. Because every call begins without memory, the model must reconstruct the policy and execute it reliably each time. A small error in the action rate can change the outcome sharply.
The benchmark’s authors built several interventions to locate those errors. One version supplies a random number. Another lets the model generate a plan that later instances receive. A stronger version provides an optimal strategy written by a human. Each intervention removes some of the work the model would otherwise have to perform for itself.
Those variations make a low score easier to interpret. If performance rises after the model receives a plan, planning was part of the problem. A change after supplying randomness points toward probability calibration. Strong performance only after receiving the optimal strategy tells us something else.
Without those checks, one score contains several possible explanations.
Test the test
Before a low capability score carries weight in a safety argument, the evaluation should show that its setup can produce a serious attempt when the relevant capability is supplied.
One way is to test the evaluation against a model deliberately trained to pursue the hidden objective. Another is to remove known bottlenecks one at a time until the model can engage with the task. The benchmark paper’s optimal-strategy and randomness conditions already provide part of that structure.
This functions as a positive control. It does not show that the deployed model has a hidden goal. It shows that the evaluation can register the behavior it was built to measure.
Anthropic has applied the same general principle elsewhere. Its sabotage report describes exercises using intentionally misaligned model organisms to test whether alignment audits can find behavior researchers know was placed in the model. Those exercises are not validations of Subversion Strategy, but they demonstrate why a known positive case is useful: the auditor has something concrete to detect.
The same standard belongs in a sabotage capability evaluation. If a model receives an optimal plan and still does not engage, the evaluator should investigate the task interface and the effects of alignment training before interpreting the score. If stronger elicitation changes the result substantially, the original measurement was sensitive to the setup.
Either outcome teaches us more than a low number alone.
A confidence interval cannot resolve this kind of uncertainty. It describes variation inside a fixed procedure. It cannot tell us whether another credible procedure would expose substantially more capability.
Propensity and capability answer different questions
Elicitation is the work required to get a model to display its best available performance on the behavior being measured. It can involve changes to the prompt, the tools surrounding the model, or the scaffolding that helps it complete the task. A more aggressive effort might include examples or targeted training.
The appropriate amount depends on the claim.
An unmodified model provides evidence about propensity: did the model engage in the behavior under these conditions? A stronger elicitation effort addresses capability: could the model carry out the behavior if it had the relevant objective and enough help to use abilities it already possesses?
A model that declines a harmful instruction may have low harmful propensity in that setting while retaining the technical ability to complete the underlying task. Training it specifically to pursue sabotage could reveal more of that ability, though the resulting model would no longer represent ordinary deployment behavior.
Both findings can be useful. They should appear under different labels.
Subversion Strategy is presented as a capability evaluation. Its purpose is to test whether a model can choose and execute a strategy for rare, deliberate failure. Elicitation therefore belongs inside the measurement. Before the score can support a conclusion about capability, the evaluator needs evidence that the model entered the task it was being scored on.
Anthropic’s neighboring result
Anthropic’s treatment of SHADE-Arena in the same sabotage report shows what a more informative elicitation record looks like.
In that evaluation, the model receives a legitimate main task and a harmful side task. It must complete both without attracting the attention of a monitor. Opus 4.6 succeeded 18 percent of the time when its reasoning was visible to the monitor and 16 percent when reasoning was unavailable.
Anthropic does not present those numbers as a clean ceiling on the model’s ability. The report says the team used its standard practices for lightweight evaluations, including some prompt iteration, but did not use targeted fine-tuning. It leaves open the possibility that the model could perform better under different conditions.
That qualification gives the reader a boundary. The result describes performance under a lightweight elicitation effort. It does not establish the strongest performance the model could reach.
The Subversion Strategy disclosure supplies less. We learn that performance was low and that Anthropic considered the model under-elicited. The public record does not explain what failed, which interventions the team attempted, or whether its strongest setup produced a serious attempt.
SHADE-Arena shows that Anthropic already knows how to attach an elicitation account to a result. Applying the same practice to Subversion Strategy would make the under-elicited finding more useful.
Publish the elicitation record
The benchmark paper is candid about the limits of its original experiments. Those experiments involved Claude 3.5 models and OpenAI reasoning models rather than Opus 4.6, so they should not be treated as a description of Anthropic’s later run.
The paper says its authors did not optimize prompts for win rate, supply few-shot examples, or fine-tune the models. It also reports that some results were sensitive to changes in the prompt and to the story placed around the same underlying game. Those disclosures tell readers how far the authors pushed and where the measurement remained unstable.
The Opus 4.6 materials should provide a comparable record for the later run.
Readers need to know what Anthropic tried after it concluded the model was under-elicited. The report should identify the strongest credible setup and explain whether any intervention changed the outcome. If the team stopped before targeted training or another expensive step, it should say why that stopping point fit the claim.
This does not require publishing instructions that would make deployed systems easier to attack. A company can describe the structure and depth of its elicitation effort without releasing every operational detail. The reader needs the path through the evaluation, not a recipe for sabotage.
If the model never engaged with the intended behavior, that belongs in the result. If supplying a plan or removing a bottleneck caused a large improvement, that belongs there too. “Under-elicited” identifies the problem; the elicitation record would show how the team diagnosed it.
Keep the failed evaluation
Anthropic was right to publish the under-elicited finding.
Removing the evaluation would hide a gap in the current safety case. Publishing the low score without the qualification would give the number more authority than the method earned. Anthropic instead told readers where its confidence stopped.
The next run should show whether Opus 4.6 can solve the task when planning and probability control are supplied. It should use more than one credible prompt. If alignment training prevents the model from participating, the report should explain how the evaluators separated that refusal from an absence of strategic ability.
A model can perform poorly because it lacks the capability. An evaluation can also fail to reach a capability that is already there. Both cases can produce a quiet transcript.
Anthropic did the responsible thing by refusing to let a low score masquerade as evidence. The next version needs a positive control and a published account of the elicitation work behind it. Until the setup can produce a serious attempt under credible conditions, the Subversion Strategy evaluation remains unfinished. Opus 4.6’s capability is still an open question.
“Under-elicited” is the correct entry in the record. It tells us that the measurement has not yet earned a conclusion about capability.