Uses the LHC black-hole safety assessments as the primary worked example for a general point: any stated probability-of-catastrophe estimate (e.g. LSAG’s or Giddings–Mangano’s) is implicitly conditional — “probability of disaster given that our argument for safety is sound.” If the probability that the argument itself is flawed (wrong model, computational error, unknown unknown) exceeds the stated catastrophe probability, the stated figure is not the right number to act on, and no amount of internal rigor within a single argument can rule this out. They formalize this as distinct from ordinary parameter/model uncertainty and argue safety assessments for rare high-stakes events systematically underweight it, proposing it be estimated from the historical track record of expert risk arguments (which have an appreciable error rate) rather than assumed away. relevance_note: The clearest generalizable articulation of “what would make even a technically correct safety case still not fully close the question” — directly answers the second half of the main question (“what does the conclusion hinge on”).
§2 Probability estimates
A-1 - A stated catastrophe probability is conditional on the argument's soundness and is dominated by argument-failure when very small
Reasoning:
- Let X = the catastrophe occurs, A = the report’s argument is sound. A report can at best deliver P(X|A): no argument can internally account for the possibility that it is itself flawed (that would need a further, higher-level argument; at most it can hedge specific sub-arguments).
- Law of total probability: P(X) = P(X|A)P(A) + P(X|¬A)P(¬A). Worked example: stated P(X|A) = 1e-9 with P(¬A) = 1e-3 and P(X|¬A) = 1e-3 gives P(X) ≈ 1e-6 - a thousandfold increase over the stated figure. If the catastrophe were to occur, it would far more likely be because the argument was flawed than because a 1-in-a-billion event happened.
- Asymmetry: the correction matters specifically for very small stated P(X|A) - practically any P(X|¬A) is then larger, so added argument-uncertainty pushes the estimate upward (for moderate P(X|A), e.g. 10%, the same term shifts 10% to ~10.001% and is negligible). The situation is mirror-symmetric for claims of near-certainty.
- Refinement: A can be decomposed into T (theories adequate), M (model adequate given theories), C (calculations correct), with P(T,M,C) = P(T)P(M|T)P(C) since calculational correctness is independent of the adequacy of what is being calculated. Each component has a historically nonzero failure rate, and T and M failures are not independent of each other in general.
- The decomposition is pure probability theory; the empirical content (how large P(¬A) actually is) enters through track-record data such as retraction rates - hence this argument turns on the measured error-rate observation.
Validity verdict (step 6)
Reconstruction: premise = a report can internally state only P(X|A); law of total probability; conclusion is explicitly conditional (“whenever stated P(X|A) is far smaller than P(X|N(not-A))P(not-A)”). Load-bearing step is pure probability algebra; traced it and the worked example (1e-9 + 1e-3*1e-3 ~ 1e-6). Undercutting probe: cases where P(X|not-A)P(not-A) is itself tiny do not undercut the claim, because the conclusion is stated as a conditional. The independence assumption in the T/M/C decomposition is a premise, priced downstream, not part of the validity call.
Link to original
O-12 - Measured error rates in published scientific arguments and calculations are of order 1e-3 or higher
Compiled empirical error-rate figures the paper assembles from the literature (not its own data collection):
quantity value cited source raw MEDLINE retraction rate 6.3e-5 Cokol, Iossifov et al. 2007 modeled retraction rate under top-tier scrutiny 1e-3 – 1e-2 Cokol, Iossifov et al. 2007 flawed statistical results in Nature/BMJ sample ~11% García-Berthou & Alcaraz 2004 hospital drug-dose error rate ~1–2% of administrations Prot 2005; Stubbs 2006; Walsh 2008 spreadsheet audits with errors ~88% Panko 1998 The paper notes retraction is only triggered by nontrivial, immediately noticeable flaws, so the retraction rate lower-bounds the serious-flaw rate; 62% of 1982-2002 retractions were unintentional error rather than misconduct (Nath et al. 2006).
Link to original
§3 Theories, models and calculations
O-13 - Castle Bravo (1954) yielded 15 Mt against a predicted 4-8 Mt because the model neglected lithium-7 reactions
Used by the paper as its flagship historical example of model (as opposed to theory or calculation) failure: the underlying theory of the relevant nuclear reactions was understood, but the model of which reactions mattered was too narrow. Fallout affected the Marshall Islands and killed a Japanese fisherman.
Link to original
§4 Application to particle-physics risks
O-14 - The RHIC strangelet safety analysis ran five years before its anthropic-selection flaw was found and corrected
The corrected analysis (anthropic bias removed, using Jaffe et al. 2000 inputs plus a planetary-formation-rate model) still bounded accelerator risk below 1e-12/yr, but the episode is a documented instance of a serious flaw persisting undetected in exactly the class of physics safety argument at issue for the LHC. The paper also notes Kent (2004) found mistakes in how risk probabilities were stated across versions of the Dar et al. paper and the Brookhaven report.
Link to original
A-2 - Independent sub-arguments multiplicatively shrink the probability that a safety case fails
Reasoning:
- For genuinely independent arguments, the joint failure probability is the product of individual failure probabilities (or at least strictly smaller than any single one): P(¬A1, ¬A2) < P(¬A1). Each added independent argument shrinks the grey area of eq. (1); a small residual grey area is acceptable when P(X|¬A)P(¬A) is small relative to the stakes.
- The paper reads Giddings-Mangano 2008 as exactly this structure, three sequential layers each covering the failure of the previous: (A1) rapid black-hole decay is a robust consequence of several distinct physical frameworks; (A2) even absent decay, a stable hole could not shed its charge, so cosmic-ray-produced stable holes would also stop in matter, making Earth’s and other bodies’ survival evidentially relevant; (A3) even for stable neutral holes, if multidimensional gravity’s scale is below ~20 nm Earth-accretion takes longer than the planet’s natural lifetime, while fast-accretion scenarios would destroy white dwarfs and neutron stars on timescales contradicting their observed lifetimes and cooling.
- Hence the total case (A1,A2,A3) is significantly stronger than any component - this argument cuts against treating a single-argument failure rate as the failure rate of the whole safety case, i.e. it moderates (without eliminating) the argument-fragility concern: what remains required is that the layers be genuinely independent and that each layer’s own modelling be sound.
- Corollary the paper endorses: independent replication of calculations, independent derivations of the same bound, and adversarial red-teaming can reduce the effective P(¬A) by orders of magnitude.
Validity verdict (step 6)
Reconstruction: premise = the sub-arguments are genuinely independent; conclusion = joint failure probability strictly below any single one. Elementary: P(notA1 and notA2) = P(notA1)P(notA2) < P(notA1) whenever P(notA2) < 1. Undercutting probe: correlated failures via shared modelling assumptions would break the product rule - but independence is an explicit premise here, and the body itself flags “layers be genuinely independent” as the required condition; whether GM’s layers actually are independent is premise-truth, priced in steps 7-8. Inference holds as stated.
Link to original
H-11 - The actionable LHC catastrophe probability is dominated by the chance the safety argument itself is flawed
The authors stress they can find no plausible combination of P(¬A), P(X|¬A) and an acceptable expected-death limit that clears their bound without substantive further argument; they explicitly remain open to additional independent arguments and verification lowering P(¬A). The figures used are illustrative, not calibrated.
Link to original