Where Evidence Ends and Judgment Begins in the GPT-5 System Card

The GPT-5 System Card is sixty pages of capability evaluations, safety benchmarks, red-team findings, safeguards, and deployment reasoning. It’s tempting to read it as a scientific report and ask whether the experiments are good enough. That misses the harder problem. The document is also making a safety case: these are the capabilities we measured, these are the risks we believe follow from them, these are the safeguards we deployed, and here is why we believe the residual risk is acceptable.

Those statements don’t share the same epistemic status. A benchmark result is an observation under specified conditions. A causal explanation is an inference about why that result occurred. A capability classification applies a decision rule to evidence. A statement about a control not yet deployed is a prediction about its future performance. A claim that risk has been “sufficiently minimized” combines evidence with a threat model, assumptions about adversary behavior, safeguard performance, and organizational risk tolerance. A system card has to hold all of these at once, and its integrity depends on keeping their provenance intact as they move through the document.

Let’s call that function a ledger. On one side: what the evaluations establish. On the other: what the organization concludes, assumes, predicts, or decides. The goal isn’t an empty second column; deployment can’t be reduced to a confidence interval. The goal is to make every transfer between columns inspectable.

The card frequently does this well. It reports regressions, flags limitations in its own confidence intervals, distinguishes model-level training from system-level protections, and warns that its preparedness evaluations establish only a lower bound on potential capability. It notes that its bootstrap procedure can produce intervals that are too tight when a pass rate sits near 0% or 100% with few attempts. It states plainly that the gap between its model-training scores and 1.0 is the fraction of adversarial cases where the other safeguard layers have to do the work. The interesting weaknesses aren’t omissions or marketing gloss. They show up where an initially precise distinction gets less precise as it travels.

Biology Is Not the Easy Criticism

The biological and chemical capability classification looks contradictory at first. OpenAI says it lacks definitive evidence that gpt-5-thinking can meaningfully help a novice cause severe biological harm (its stated threshold for High capability) while treating the launch as High and activating the corresponding safeguards anyway. Read as a scientific classification, the conclusion outruns the evidence. Read as a governance decision, the card is unusually explicit about what it’s doing.

OpenAI says the model remains “on the cusp” of the threshold, and that it’s treating the launch as High mainly to keep the organization ready for future updates that could push capability further. The preceding Preparedness Framework section warns its evaluations are lower bounds: different prompting, fine-tuning, longer rollouts, novel interactions, or scaffolding could elicit more than what was observed. The classification isn’t proof the threshold was crossed; it’s a precautionary call, made under uncertainty about a model believed to be close to it.

That’s defensible, but it creates an obligation. Once “demonstrated High capability” and “precautionary treatment as High” have been deliberately split apart, later references have to keep track of which one is meant. Otherwise a conservative governance decision quietly inherits, through repetition, the epistemic weight of the empirical finding the card explicitly says it doesn’t have. “We are treating the model as High” wants to become “the High-capability model” the moment enough people start writing downstream of the decision. That’s economical English and expensive epistemology.

Uncertainty Does Not Have One Job

The biological analysis gets more interesting when novice uplift reappears in the residual-risk section. Near the capability threshold, the absence of definitive evidence of novice uplift isn’t enough to avoid High treatment: uncertainty plus consequence severity argues for stronger safeguards instead. Later, OpenAI calls the risk of incremental leakage low, “as demonstrated” by a safeguard test in which novices failed to gain significant uplift.

Both conclusions can be true. A precautionary capability classification and a residual-risk assessment answer different conditional questions and don’t need identical burdens of proof. The first asks whether uncertainty around a potentially catastrophic capability justifies controls before the threshold is definitively crossed. The second asks whether post-safeguard evidence lowers the plausibility of one specific abuse pathway.

What’s missing is the rule connecting them, and in this case the test itself. The novice uplift safeguard test is named once, in the sentence that leans on it, and appears nowhere else in the card: no design, no sample size, no results, no comparison condition. It may well be a rigorous study. But “demonstrated” is a word that points at an artifact, and the artifact isn’t in the document. The reader is left to infer both why absence of demonstrated uplift is insufficient at one stage yet sufficient to “demonstrate” low risk at another, and what the demonstration consisted of. If different experimental conditions or different target claims reconcile the two uses, that reconciliation belongs in the text, not in the reader’s head.

The inference gap is exactly the kind of problem that survives ordinary fact-checking. Every number can be correct, every sentence locally defensible, and the finished document can still ask the reader to reconstruct why identical evidence carries different weight in different places. Consistency isn’t just making the numbers match: it’s preserving what each number is allowed to mean.

Component Evidence, System Product

The card opens by calling GPT-5 a “unified system”: a fast model, a deeper reasoning model, and a real-time router that picks which one handles a request based on conversation type, complexity, tool needs, and explicit intent. The router trains continuously on signals that include model switching, response preferences, and measured correctness. Once usage limits are hit, mini versions take the remaining queries. The card then says it focuses primarily on gpt-5-main and gpt-5-thinking, with other models pushed to the appendix.

That’s a unit-of-analysis problem. The object described to the reader is an orchestrated system; most of the safety evidence is reported for its components in isolation. That would matter less if the components behaved identically. They don’t. On the production safety benchmark, gpt-5-thinking generally performs on par with or above o3. gpt-5-main underperforms GPT-4o in several categories, with statistically significant regressions in hate/threatening and sexual/exploitative content alongside significant gains in illicit/nonviolent and illicit/violent content. The appendix shows yet another profile for the mini models. No single component result stands in for “GPT-5.”

The router doesn’t need to be built as a safety control to have safety consequences. By choosing the responding model, it decides which measured failure profile becomes the actual product for a given interaction. Usage-limit fallbacks add a second selection rule, routing identical requests to different components based on account state rather than task. The safety behavior a user meets is a function of component performance and the distribution orchestration produces over those components. That distribution is exactly what the card doesn’t report. No aggregate safety performance for the routed experience. No routing proportions by component. No safety outcomes conditional on which component answered.

A different asymmetry shows up in how the card explains its own numbers. It attributes gpt-5-main’s illicit/nonviolent and illicit/violent gains to safe-completions, and says elsewhere that the method improved safety and helpfulness across production comparisons and controlled experiments, so the claim has support beyond this one table. But the card doesn’t show enough of those experiments here for an outside reader to independently trace this specific result back to that mechanism. It’s more careful with the sexual/exploitative regression: acknowledged rather than explained away, with manual review finding the violations low severity. But there’s no severity rubric, no sample size, no reviewer agreement, so even the honest acknowledgment asks the reader to take more on faith than the artifact shows.

For biological risk, the gap is sharper than a distribution over failure profiles. The safeguard-design section states that system-level protections cover all production traffic for gpt-5-thinking, gpt-5-thinking-pro, and gpt-5-thinking-mini. gpt-5-main isn’t in that list. The two-tier monitor stack, a topical classifier followed by a reasoning monitor, is scoped to the thinking models. So component selection doesn’t only determine which measured failure rate applies; for the domain the card treats as High capability, it determines whether the monitor stack is in the path at all. There are sensible reasons this might be the right design: the High classification attaches to gpt-5-thinking, and gpt-5-main is a weaker model in a domain where capability is the trigger. But the card never states that reasoning, never reports how the router allocates biology-adjacent requests, and never says what happens to a biology-adjacent request that lands on gpt-5-main. A perimeter defined by model rather than by content, sitting behind a router that selects models by conversation type, is an architecture the reader can only guess at. The coverage statement is a measurement; what it implies for a routed product is a second column the card leaves blank.

A related transfer runs the other way. The card says it did not rerun safety evaluations for gpt-5-thinking-pro, having determined that gpt-5-thinking’s results are strong proxies under parallel test-time compute. That’s a defensible judgment about transferability, and a plausible reconciliation is available: the safety evaluations are near-saturated refusal measures, where extra compute has little room to move the score, while capability benchmarks are not. But the card doesn’t make that argument. It says in one section that the setting doesn’t move safety results, and in another that it may move capability results enough to warrant measuring gpt-5-thinking-pro separately on biological evaluations. Both can be true. The reader is left to supply the reason they are.

None of this proves the router is unsafe, or that OpenAI skipped testing it internally. It shows something narrower: the public document never builds the bridge from the components it evaluates to the system it names on page one. The result is a component safety case standing in for a product whose behavior is set by orchestration.

The biological safeguards section shows what a bridged argument looks like. Model safety training is reported separately from the system-level protections, and the card says outright which layer is expected to cover what the other misses. Same word, “system,” applied to a different layer. Here the seam between layers is visible. It’s the exception that proves the rule. Name which mechanism does which job, and “system” stays inspectable. Don’t, and “system” is just where the bridge should have been.

“Attributed to Noise” Is Not a Finding

Two phrases about noise earn their own space, and they aren’t equivalent.

In the standard disallowed content evaluation, gpt-5-thinking scores 0.881 on personal-data against o3’s 0.930. The card says the gap represents natural noise in the evaluation. That’s a flat assertion of cause. The very next sentence notes that gpt-5-thinking overperforms o3 on personal-data/restricted and that this too fails significance, so the framing is applied symmetrically, in both directions, which is to the card’s credit. But symmetry isn’t identification.

In the production benchmarks, several nonsignificant gpt-5-main regressions in non-violent hate, harassment/threatening, and sexual/minors “can be attributed to natural noise in the evaluation.” The modal verb does real work. “Can be attributed” concedes a live alternative; “represents” doesn’t. “Represents” is the phrasing worth flagging, and it’s the flatter of the two.

Statistical nonsignificance doesn’t identify a cause. It means the evaluation didn’t provide enough evidence, under the chosen analysis, to separate the difference from sampling variation. “Not distinguishable from noise” is an accurate description of the result. “Represents natural noise” and “attributed to natural noise” both supply an explanation the test never established; noise is one candidate among others the test wasn’t powered to rule out.

The gap is one verb wide and does a lot of work: a null result quietly becomes a causal story in a document otherwise careful enough that the swap is easy to miss.

A Jailbreak Is a Test of the Threat Model, Not a Refutation

The bioweaponization red-team results force the evidence/judgment distinction into the open. OpenAI tests model safety training, system-level monitors, three expert bioweaponization campaigns, third-party red teams, and government red teams. That’s substantial enough that reducing the section to “a jailbreak got through” would misrepresent the safety case. It’s worth saying plainly that an earlier pass at this argument came closer to that reduction than the evidence supports.

The short-horizon testing matters most. Nineteen biology PhD red teamers spent roughly 380 hours over ten days hunting for API jailbreaks and filed 46 potential reports. OpenAI says three contained specific, actionable information it considered practically useful for bioweapons development, and that all three would have been blocked by the generation monitor. FAR.AI spent eighty hours over a week and found partial vulnerabilities, plus a potential end-to-end attack that bypassed the monitoring system with substantial degradation in output quality, but no end-to-end jailbreak producing high-quality output while evading every layer. Gray Swan’s arena campaign yielded 277 high-quality reports across 28,367 attempts, an attack success rate under one percent, and OpenAI reviewed ten examples from each of six distinct jailbreak cohorts, judging 58 of those 60 blockable by the generation monitor and the remaining two artifacts of its own grading rubrics. In the same campaign, red teamers were blocked on average every four messages.

Then UK AISI found what the earlier campaigns hadn’t: a jailbreak that evaded every mitigation layer and was still being patched at publication. Two things are true about that finding, and the card states both. UK AISI held access no adversary would have, including prototype safeguards, the monitor system design, the biological content policy, and the monitor models’ chains of thought, and OpenAI says this let them identify vulnerabilities more easily than malicious users could. And the discovery still happened. That combination doesn’t erase the broader red-team evidence, and it doesn’t establish that an outside attacker could replicate the result. What it does is convert an assumption into a variable: the residual-risk argument rests on universal jailbreaks being hard to discover, and the one campaign with visibility into the safeguards found one. The card is entitled to say the privileged access explains the difference. It hasn’t shown how much of the difference the access explains.

There’s a second dependency running underneath all of this. Nearly every reassurance in the red-team sections is a counterfactual about the generation monitor: the three actionable reports would have been blocked; 58 of 60 Gray Swan examples would have been blocked; the UK AISI jailbreak would have thrown flags and led to a ban. The monitor’s measured performance on challenging prompts is a recall of 0.838 and a precision of 0.647, deliberately tuned toward recall, as the card explains, and reported with system reliability above 99.9%. Those are honest numbers, and a monitor that catches roughly five of every six adversarial cases in a hard test set is a real control. They are also the numbers from ChatGPT agent: the card states plainly that the same models power these protections and that additional testing wasn’t conducted for this deployment. Inheriting a validated control is reasonable engineering. But the safety case for GPT-5 rests substantially on a monitor whose performance was characterized against a different system. “Would have been blocked” reads as an observation and is actually a transfer: a measurement taken elsewhere, carried across, with 84% recall inside it every time it appears.

The residual-risk argument rests on a threat model. The card says the primary severe-harm pathway is persistent probing for biorisk content over weeks or months, and that this risk is sufficiently minimized because universal jailbreaks are hard to discover, abusive users can be banned or reported, and public jailbreaks can be found and patched. Under that model, account enforcement makes sense: it interrupts a process expected to take sustained interaction, rather than guaranteeing no dangerous output ever escapes. The blocked-every-four-messages figure is genuine evidence for that interruption story.

The open question isn’t whether a jailbreak can produce a bad answer: OpenAI already showed one can. It’s how much severe-harm uplift an adversary can bank before monitoring, enforcement, and remediation cut them off, and how much the deployment judgment leans on the assumption that the relevant attack needs weeks or months. The card supplies pieces of that argument, including strong short-horizon red-team results and a monitor with disclosed performance, but not the full calculation. The jailbreak doesn’t refute the safety case. It names the assumption holding it up.

“Sufficiently Minimized” Is a Governance Judgment

The Preparedness Framework requires safeguards that “sufficiently minimize” severe-harm risk for highly capable models. By the end of the card, OpenAI concludes the remaining biological risks clear that bar. That conclusion draws on empirical evidence, but it isn’t itself an experimental result: it synthesizes capability estimates, model-level safety performance, monitor performance, red-team findings, attacker effort, account enforcement, access restrictions, remediation capacity, policy conservatism, and assumptions about how long it takes to cause severe harm. No additional benchmark removes the need for that synthesis. Eventually someone decides whether the residual risk is acceptable under the framework.

So the right question for an external artifact isn’t whether the decision can be “proven.” It’s whether the assumptions and decision rules carrying it are visible enough to challenge. The biological section usually clears that bar: it tells the reader where the model failed, which downstream safeguard is expected to cover the gap, what outside evaluators found, and what risk remains after mitigation.

The concluding sufficiency subsection clears it less often, because several conclusions rest on stated beliefs and expectations rather than measurements. Unknown universal-jailbreak risk is called sufficiently minimized because discovery is expected to be hard, malicious users can be banned, and public jailbreaks are expected to be patched quickly. Incremental leakage risk is called low based on a novice-uplift test the card doesn’t present. Those may be reasonable judgments. But their role in the final decision belongs in the ledger as assumptions inside a threat model, not as measurements of residual risk. That itself is an inference about what the card’s disclosures leave out, not a claim about what OpenAI actually weighed internally. The card never says its own sufficiency conclusion is under-defended. That’s this piece’s read of the gap between what’s disclosed and what the conclusion needs, and it should be held to the same standard as everything above it: inference, not measurement.

A Planned Control Belongs in the Future Tense

The Trusted Access Program is the cleanest case for why tense matters. In the safeguard-design section, OpenAI says it’s “building” a Life Science Research Special Access Program for vetted customers, and describes the governance and safety signals it plans to weigh. That’s a prospective description: an intended architecture, not a deployed one.

Five pages later, the sufficiency section says OpenAI “plan[s] to introduce” a trusted access program, admits overall safety partly depends on its effectiveness, and calls the associated risk minimal because of strict access conditions and vetting. That’s not an absurd move: a well-specified control can support a prospective risk assessment before it’s live. But it can’t supply evidence of its own operational effectiveness before it operates. The card has shifted from control design to a prediction about control performance, not from observed performance to a conclusion. That distinction needs to stay visible, because the residual-risk argument explicitly leans its weight on the program working.

Compare a promise that costs nothing. The instruction-hierarchy table shows gpt-5-main regressing against GPT-4o on phrase protection against malicious users (0.619 versus 0.735) and against malicious developers (0.404 versus 0.449). The card names the regressions and says a fix will follow. That’s the right handling: a measurement disclosed, a remediation promised, and no conclusion resting on the promise. The Trusted Access Program is the same grammatical form doing structurally different work, a future control positioned underneath a present claim of sufficiency. Plans aren’t evidence. They can be assumptions. The fix isn’t pulling the Trusted Access Program out of the safety case — it’s labeling the dependency honestly. If the deployment judgment assumes a planned control performs as designed, the assumption stays in the ledger until production evidence can replace it.

A Narrow Result Next to an Open Question

One more boundary worth naming, briefly. OpenAI reports a real sycophancy improvement: an offline evaluation score of 0.145 for GPT-4o against 0.052 for gpt-5-main and 0.040 for gpt-5-thinking, and preliminary online A/B data showing sycophancy prevalence down 69% for free users and 75% for paid users versus the most recent GPT-4o. The card calls the online evidence preliminary and draws a correspondingly bounded conclusion. The evidence supports it.

One sentence in that section should be corrected in the next revision. The prose reads that gpt-5-main performed nearly three times better than the most recent GPT-4o model, scoring 0.145 and 0.052 respectively, which assigns the worse score to gpt-5-main. The table directly below gives GPT-4o 0.145 and gpt-5-main 0.052, and the multiplier in that same sentence is computed from the correct assignment, since 0.145 against 0.052 is a ratio of roughly 2.8. The arithmetic and the table agree; only the ordering clause dissents. It’s a transposition, not a claim, and it’s the only error in the card that a copyeditor would catch. That’s what makes it worth naming: everything else in this piece survives proofreading intact.

That result sits close, in the document, to an open question: OpenAI says it’s researching related concerns including emotional dependency and distress scenarios, where Microsoft’s red team separately flags room for improvement. The card never claims the sycophancy result speaks to those concerns, and treating it as if it did would mean writing the overclaim ourselves just to object to it. The real point is narrower: a mature, well-measured result sitting beside a construct with a much less mature measurement program can lend it borrowed confidence, even when the prose keeps the two properly apart. That’s a property of layout, not of evidence. It is a smaller version of the same provenance problem running through everything above it.

The Ledger Has to Survive the Whole Document

The card is strongest when it tells the reader exactly where its knowledge stops. The biological capability classification does this well: threshold not definitively demonstrated, model close to it, evaluations possibly underestimating capability, conservative treatment chosen anyway. A reader can disagree with the decision without first having to figure out what kind of decision was made.

The weaker moments are quieter than that. A precautionary classification loses its qualifier on repetition. Component evaluations get asked to support a routed product, with the monitored perimeter drawn around some components and not others and no account of how the router allocates between them. A test that never appears in the document is said to demonstrate a residual-risk conclusion. Monitor performance measured against a previous system carries the phrase “would have been blocked” through an entire red-team section. Statistical nonsignificance becomes a causal explanation in a single verb. A future control enters a residual-risk calculation as an assumption about its own eventual success. None of this requires bad faith, and none of it gets fixed by better copyediting alone. These are research-operations problems: every section can be locally accurate while the assembled document gets globally less precise. The people who wrote it know which findings are preliminary, which controls are inherited, which classifications are precautionary, which causal claims lean on experiments run elsewhere, and which deployment conclusions depend on threat-model assumptions. The reader knows only what survives onto the page.

A system card, then, is less a repository of safety results than a ledger of epistemic provenance. Measurements, interpretations, causal explanations, policy choices, predictions, and deployment judgments can all live in the same document. They can’t become interchangeable as the argument moves from evidence to action. The job of a system card isn’t to eliminate judgment. It’s to keep judgment from borrowing evidence’s clothes.