Appendix — known failure modes

Three independent shakedown runs of the 10-step pipeline, one per case study: black holes (N=5 curated sources), COVID origins (N=5), eggs (N=10). Every defect below was logged by an agent inside the run that hit it — searchers, cluster children and step-9 reviewers appended BLOCKER/MAJOR faults to a shared log as they encountered them, deduped and ranked by the orchestrator at the end. Across the three runs: 3 BLOCKERs and 33 MAJORs. Two of the three main reports disown their own headline number in the report text; that is intended behaviour, not a slip.

Defects that recur across independent runs

1. Write refuses prose .md deliverables in a subagent (3/3 runs). Steps 9-10 have prose artifacts; children get “Subagents should return findings as text, not write report files”. It is a harness-level refusal the pipeline cannot configure — it hit 4 of 4 prose-artifact children in eggs, and a child that accepts it produces no artifact, silently. Fixed: steps 9-10 now mandate a quoted heredoc plus wc -w verification.
2. external_consensus consumed but never written (3/3). Step 10 reads it as a per-cluster field from steps 7-8; steps 7-8 never mention it and step 9 says external consensus enters “no number and no frontmatter field”. All three step-10 children found nothing and all three substituted labelled own-characterisation rather than inventing a field. Open — a one-line spec fix, deferred to avoid touching step 9 pre-deadline.
3. create_node.py contract undocumented (2/3). The script needs the node body piped on stdin with an {{ID}} placeholder; mode-file examples show only the flag form, and every minting step copies that snippet. Two distinct failure modes were conflated in the logs; checked against the source: missing stdin is a hard exit (no content on stdin), whereas piping content that lacks the {{ID}} placeholder mints an id-less node with only a warning. Mirroring {{ID}} into --title separately corrupted filenames — 10 of 23 edge files renamed by hand in eggs. Fixed: every mode-file example now shows the pipe with an absolute path, and --title strips the placeholder.
4. used_for_prior never fired (eggs and COVID, independently). Marking edges used_for_prior is step 7’s prior/likelihood partition; in both runs every cluster child marked zero. Eggs children judged every edge case-specific, so consuming any would starve step 8; the COVID child was blocked structurally, since all 9 edges sat in correlation groups and rule 5 forces all-or-none marking. Both priors rest on the child’s outside view with zero graph input. Design question — see 19.
5. Runner’s BANNED-token scan matched English prose in comments (2/3). run.py scanned raw block source, so “an open July 2022 Author Correction” aborted composition with 'open' is not allowed in a block. Near-inevitable at scale, since steps 7-8 require an English reason comment on every number. Fixed: code_only() blanks COMMENT and STRING tokens before the scan; the 9-case self-check passes and the guard still rejects real import os / open(...) / __builtins__.

Epistemic defects the pipeline caught in its own output

6. Eggs: a top-ranked hypothesis nobody should read as top-ranked. H-9 finished at 0.349 against H-2’s 0.126, but HC-2’s own exclusivity field records the cluster as STRAINED — H-4/H-9/H-10 all assert a null effect and can hold simultaneously, so normalising over them is arithmetically fine and semantically undefined. H-9 leads only because the null branch was split three ways while the directional answers were not; branch totals read ~0.57 null / 0.13 protective / 0.10 harmful / 0.20 varies. The scope rider used to price H-9 appears in no node — it was invented in step-4b prose and propagated into every step-7/8 brief. The step-10 report ranks by branch, not member, and says why.
7. A defect claim the run then retracted. A child logged as BLOCKER that scope-restricted members drift upward by abstaining. Checked against the runner’s mixture (t·lik[i] + (1−t)·marg), H-9 gains where it fits the data (0.72 vs marg 0.58; 1.00 vs 0.88; 1.00 vs 0.71) and loses where it does not (0.40 vs 0.46) — ordinary updating, diagnosis withdrawn. What survives is narrower and real: no stated convention for a member silent on an observation, and two careful children used opposite conventions on the same member.
8. Residuals written as substantive claims. Step-4 rule 2 requires a residual to read “the answer is something not listed here”; 2 of 3 eggs residuals violate it (H-12 asserts a J-shape, H-13 homogeneity), and Check 3 inspects only the residual: true flag, never the statement. CG-4 then anchored on residual H-12 at 1.0, and genuinely unlisted answers (APOE genotype, baseline LDL, microbiome) are held nowhere — so the cluster is not a proper probability space.
9. Self-confirming hypothesis/evidence loops. Eggs H-5 was extracted from S-19 and its only evidence edges are S-19’s own observations. One child noticed and hand-capped the ratio at ~3.3:1; the identically-shaped H-8/S-8 case was never discounted, so HC-1’s 0.81 is partly circular. Nothing detects the shape — it depended on one child’s judgement.
10. COVID: a structurally unfalsifiable 42.4% residual, from a curation cut. Every curated market-side observation was sampled at Huanan and every research-side one concerns WIV/DEFUSE, so no likelihood can discriminate against either residual leg — a fact about the observation set, not the world. Upstream cause: curation ranks by score with no constraint that each evidence class survives, so N=5 removed case geolocation and epidemiology entirely, including a source at the pool’s top usefulness (4.5). The report states both and declines to allocate the residual rather than importing the prior’s leg ratio as a result.

What the shakedown confirmed works

11. Correlation-group construction matched the expected data-basis partition exactly in eggs — 7 CG nodes on the predicted bases, and the one expected pair that did not form was correctly suppressed because its partner observation orphaned.
12. Gate scripts held: check_missing_evidence_links.py clean, attach_argument_backlinks.py resolved both wikilink formats, orphan detection moved exactly the 2 non-discriminating observations and no others.
13. Step-6 argument corrections propagate into step-8 pricing — A-3’s correction cut H-4’s EPIC likelihood from a would-be 2–3x to ~1.15x.
14. Composition was byte-identical on re-run in all three cases; all 15 eggs evidence blocks carried correct vector lengths across cluster sizes 4/6/3, including batches spanning clusters.
15. Children surfaced structural problems unprompted — the abstention artifact, the H-6/H-7 non-discrimination, a stale-handoff duplicate-extraction risk and a fabricated author list were all found by children checking rather than assuming, none by the orchestrator.

Open design questions

16. Scope as a cluster dimension. Members giving the same answer under different scope riders are not mutually exclusive. The fix — forbid scope-restricted members inside a direction cluster, or make scope its own dimension — changes step 4’s core partition rule, too central to alter on three runs of evidence.
17. The 4a/5 re-merge gap. Step 5 twice proved members undiscriminated after 4a ruled them discriminable (HC-3’s H-6/H-7 had no separating observation at all), and 4b has no remit to merge. The fix is a feedback edge from 5 back into 4a — a backward loop in a forward pipeline, not a patch.
18. Correlation-group scoping. Groups are built strictly per-cluster, so an observation in two clusters never has its edges joined — 8 of 23 eggs edges carry no group despite being correlated elsewhere, which step 8 could misread as independence. Separately, defaulting data_basis to a self-link collapsed COVID’s groups 1:1 onto papers (4 CGs = 4 papers, zero lone edges), leaving the mechanism’s real case — two sources re-analysing one dataset — untested. Changing the default changes evidence-counting everywhere.
19. The prior/likelihood partition. Either zero marking is the intended design, and step 7 should stop implying base-rate edges are normally pulled, or step 3 must extract reference-class rates as separate observations. Both redesign what a prior means here; shipping the mechanism named and unfired is more honest than silently rewriting it three runs in.