EpiStack

problem-log

17 min read

Problem log — covid1 test run

Agents append problems here during the flf-epistack quick test run. Purpose: catch major errors or problems in the pipeline on a fast N=5 shakedown — before the real N=25/50 runs. This is NOT a place to record minor imperfections.

How to append (every agent)

  1. Only log BLOCKER or MAJOR problems. Skip minor/cosmetic imperfections — the run is deliberately quick and rough, and sub-optimal output is expected and fine.
  2. Append your entry at the bottom of ## Log, using a single write (e.g. a >> shell append) so parallel appends don’t clobber. Never edit or delete another agent’s entry.
  3. One block per problem, in this format:
### [SEVERITY] step <n> / <slice or agent> — <short title>
<1–3 lines: what happened, where (file / script / cross-step interface), and why it blocks or corrupts the run>

Severity scale:

  • BLOCKER — stops the step or run, or corrupts the graph; must be fixed before the real runs.
  • MAJOR — wrong/misleading output, a broken cross-step interface, or instructions ambiguous/contradictory enough that a competent agent could easily get them wrong.

The orchestrator appends a ## Ranked summary at the end of the run: deduped, all BLOCKERs first then MAJORs. That ranked list is the deliverable Simon reads.

Log

[MAJOR] step 1 / genomic-institutional-evidence — searcher budget undersized relative to brief’s own named-anchor list

The planner’s search-plan.md gave this slice a budget of “write 6 source notes, read ~8” but the same slice’s balance/anchors paragraph explicitly names ~10 distinct primary items across 4 required both-sides sub-areas (furin site x2, closest-relative genomes x2, lineage-A/B+restriction-site x2, WIV/institutional-and-govt x4). Fully honoring “represent both directions” on every named sub-area is not achievable within 6 written nodes without leaving at least one side of one sub-area uncovered. Resolved via judgment call (wrote 9, ~50% over budget) rather than dropping a required side of an axis; flagging so the orchestrator/planner can size write-budgets to ≥1 node per explicitly-named anchor in future runs.

[MAJOR] step 3 / orchestrator handoff — continuation note stale, would have caused duplicate extraction on resume

agent-notes/orchestrator-continuation.md was written before the step-3 extractor ran but states “NEXT: Step 3 — extract” and carries a ready-to-paste extractor brief, while the graph already contained the full step-3 output (O-1..O-12, H-1..H-5, A-1..A-5, extracted: lists on all 5 sources). An orchestrator resuming and following the note literally — as instructed (“continue from Step 3”) — would have re-spawned the extractor and minted a duplicate parallel node set (O-13.., H-6.., A-6..), silently corrupting the graph, since create_node.py mints next-free ids and never detects that a paper was already extracted. Caught only because I inventoried the directory before spawning. Fix: the resuming orchestrator must verify actual on-disk state before trusting a handoff note’s “NEXT” pointer, and/or the handoff note must be rewritten by the same agent that completes each step rather than written ahead of time.

  • MAJOR — step 4b (cluster), covid1: the ≥2-active-members rule leaves no valid home for a true-but-non-competing hypothesis. H-3 (natural evolution in the bat reservoir) is on-topic but non-exclusive with both H-1 and H-5, so it cannot join the route cluster; the only 2-member cluster it could form (with a constructed “engineered genome” complement) would be an alternative slicing of the same sub-question H-5 already answers, double-counting O-10/O-11/O-12 in violation of rule 3. Forced to drop an on-topic hypothesis. Two consequences: (i) drop_reason is specified as “discriminates nothing relevant”, which is false here — the schema has no state for “background claim, not a competing member”; (ii) dropping H-3 breaks check 1 for H-4, whose merged_into now resolves to a dropped rather than active H — 4a/4b own overlapping bookkeeping and nothing in step 4 tells 4b to re-point or re-drop a survivor’s absorbed nodes when it drops that survivor, so step 5’s one-hop resolution will land on a dropped node.

[MAJOR] step 4 / 4b — dropping a 4a merge survivor strands its absorbed nodes, violating check 1

4b dropped H-3 (a 4a merge survivor) as a straggler that clusters nowhere — explicitly licensed by the spec’s division of labour. But H-4 had been absorbed into H-3 at 4a, so its merged_into then pointed at a dropped node, breaking check 1’s “merged ⇒ merged_into resolves in one hop to an active H” and leaving step 5’s one-hop resolution to land on a dropped node. Step 4 gives 4b no instruction for re-pointing or re-dispositioning absorbed nodes when it drops a survivor, and 4a cannot anticipate it (4a runs first and does no clustering). Orchestrator fixed by re-dispositioning H-4 to dropped with its 4a merge preserved in former_merged_into + merge_note. Fix for the spec: state that dropping a survivor cascades to its merged_from set, and add a check that no merged_into/merged_from edge crosses a state boundary.

[MAJOR] step 5 / consolidator — correlation groups collapsed 1:1 onto sources; the mechanism’s real case was never exercised

build_correlation_groups.py produced exactly 4 CGs, each containing precisely the edges from one source: CG-1={E-1,E-2}/D-2 (S-2), CG-2={E-3,E-4}/S-4, CG-3={E-5,E-6,E-7}/S-12, CG-4={E-8,E-9}/S-15. This is a direct consequence of step 2 defaulting a source’s data_basis to its own self-link [[S-N]]: every observation from one paper then trivially shares a basis, so “correlation group” degenerates into “group by paper”. The mechanism’s actual purpose — catching two different sources that re-analyse one shared dataset, which is how Bayesian aggregation manufactures false confidence — was never tested here, because the one shared external basis (D-2, Huanan swabs) is used by only one curated source; S-3 (Crits-Christoph/Débarre, a reanalysis of D-2’s raw reads) was scored but not curated at N=5. Two knock-on gaps: (i) zero lone edges, so step 8’s “batch of ≤3 independent edges” path gets no coverage in this run at all, and step 8 will write only joint CG blocks; (ii) grouping by paper is not harmless — CG-3 bundles E-7 (no furin site in BANAL, favours H-5) with E-6 (BANAL binds human ACE2, cuts against H-5), so one joint block must price opposite-direction evidence together. Before the real N=25/50 runs, deliberately curate ≥2 sources resting on one shared D node to exercise cross-source correlation, and confirm a lone edge exists somewhere.

[MAJOR] step 7 / A-5 — affects_hypotheses is silently dead on any argument that also has affects_observations

no_observation_arguments.py --cluster HC-1 returns “no no-observation arguments”, even though A-5 carries affects_hypotheses: [[H-5]] and H-5 is a member of HC-1. Cause: the script skips any argument with a non-empty affects_observations (scripts/no_observation_arguments.py:138), by design — step 7 defines a “no-observation argument” as one with empty affects_observations. But step 8 reads only the edge’s arguments list, which attach_argument_backlinks.py builds from affects_observations alone; a grep shows no script or step file anywhere reads affects_hypotheses for scoring (only for orphan-retention and the missing-link sweep). Net: for an argument carrying both attachments, the affects_hypotheses link is read by nothing and its direct bearing on the hypothesis is silently dropped. This is not a rare shape — step 3 explicitly invites setting both fields, and A-5 is the run’s single most load-bearing lab-leak argument: its claim that documented institutional intent+capability raises P(lab origin) is a prior-level consideration about H-5, but it can now only enter as a modifier on O-11’s likelihood. Either step 3 should forbid dual attachment, or step 7 should also consult arguments whose affects_hypotheses hits the cluster regardless of affects_observations (with an explicit anti-double-counting rule), or the dual case needs defined semantics. Consequence here: HC-1’s prior is built with zero argument input.

[MAJOR] step 7 / HC-1 — the prior/likelihood partition mechanism was structurally unusable; prior rests on zero graph evidence

Step 7’s central design is that the analyst pulls outside-view/base-rate observations into the prior and marks their edges used_for_prior: true, leaving case-specific ones for step 8 — “the marking is the partition between the two steps”. Here the child could mark nothing, because (a) all 9 edges sit in correlation groups (see the step-5 CG entry) and (b) rule 5 requires marking all-or-none across a group. The one genuine base-rate candidate, O-4 (47,381 live animals incl. susceptible raccoon dogs — a precondition fact), is grouped with O-5 (no bats/pangolins on sale), which is explicitly discriminating, so admitting the base rate would have deleted live evidence from step 8. Result: HC-1’s prior rests entirely on the child’s own reasoned base rates with zero input from the graph’s evidence layer, and — combined with the affects_hypotheses dead-field bug above — zero input from its argument layer too. The all-or-none rule is right in itself (partial marking would double-count), but it composes badly with basis-defaults that put every edge in a group: the two rules together can silently disable the partition entirely. Suggested fix: allow per-edge marking within a group when the group’s joint likelihood explicitly conditions on the marked member, or let step 2 default a source’s data_basis to something other than a self-link so single-paper observations aren’t auto-correlated.

[BLOCKER] step 8→9 gate / runner — banned-token scan reads comments, so ordinary prose kills the run

runner/run.py ran BANNED.search(src) over the raw block source, comments included (run.py:127, pattern at :25 = \b(import|open|eval|exec|globals|locals|__\w+__)\b). CG-3’s likelihood comment contained the phrase “an open July 2022 Author Correction”, so the composition gate aborted with ValueError: CG-3: 'open' is not allowed in a block — on a block that is entirely well-formed and passed all four of step 8’s own Checks. This is a hard stop: no posterior can be composed, so steps 9 and 10 cannot run at all. It is also near-inevitable at scale rather than a fluke, because step 8 rule 2 and step 7 rule 1 require every number to carry a reason comment in English, and “open”, “import”, “globals” and “locals” are all common words in exactly the register those comments are written in (“an open question”, “we could not import the raw reads”, “locals to the region”). Any single such word in any comment in any block fails the whole run, and the error message points at a banned token rather than at prose, so the natural reaction is to suspect the note is malformed. Fixed in place (runner/run.py): added code_only(src), which uses tokenize to blank COMMENT and STRING tokens (preserving line/column offsets so genuine hits still report a usable position) before the banned-token scan; the scan now reads code only. Verified: the skill’s own test_run.py self-check still passes all 9 cases including its determinism-guard test, and the guard still rejects a real import os, a real open(...) and __builtins__ while allowing the same words in prose. Recommend keeping this fix.

  • MAJOR — step 9, HC-1 slice: the Write tool refused to create the required cluster-review artifact (“Subagents should return findings as text, not write report files”), because a step-9 review is .md prose. Worked around with a shell heredoc. Any step whose deliverable is a prose .md file will hit this in a subagent.

[MAJOR] step 10 / part 4 — external_consensus is spec’d as a field that no step ever creates

Step 10 part 4 instructs the report to use “external_consensus — the labelled field recording what the field thinks”, and its Interface lists external_consensus among what it consumes from steps 7-8. But no such field exists anywhere: steps 7 and 8 never mention it, and step 9 — the only step that discusses external consensus at all — makes it an optional prose section and explicitly says to “let it enter no number and no frontmatter field”. So step 10 is told to read a field that the spec elsewhere forbids creating. The step-10 child correctly found nothing, wrote the comparison from its own knowledge labelled as external, and flagged the gap rather than inventing a field. Consequence is mild here (the comparison still got written) but the interface is contradictory and will keep producing this confusion. Fix: either drop the field references in step 10 and describe it as “the optional external-consensus section in each step-9 review”, or add the field to step 9’s spec and make it MUST-when-written.

Ranked summary

Deduped across all entries above, BLOCKERs first then MAJORs, each roughly by how much it would cost a real N=25/50 run. 10 distinct problems: 1 BLOCKER (fixed), 9 MAJOR. The pipeline ran end to end and produced a coherent, gated graph — every failure below was caught by a gate or by a child that checked rather than assumed, and none silently corrupted state except where noted.

BLOCKER

  1. Runner’s banned-token scan reads comments, so ordinary English prose kills the whole run (step 8→9 gate). BANNED.search(src) ran over raw block source including comments; the phrase “an open July 2022 Author Correction” in a well-formed CG-3 comment aborted the composition gate with ValueError: CG-3: 'open' is not allowed in a block. No posterior composes ⇒ steps 9-10 cannot run. Near-inevitable at scale, because steps 7/8 require every number to carry an English reason comment and “open / import / globals / locals” are common in that register. FIXED in runner/run.py — added code_only() (tokenize-based blanking of COMMENT and STRING tokens, offsets preserved) before the scan; test_run.py still passes all 9 self-checks and the guard still rejects real import os / open(...) / __builtins__. Recommend keeping this fix.

MAJOR

  1. Step 4’s ≥2-members rule has no home for a true-but-non-competing hypothesis, and forces a false drop_reason. H-3 (natural bat-reservoir evolution) is on-topic but non-exclusive with both H-1 and H-5, so it can join no cluster; the only 2-member cluster it could form would re-slice the sub-question H-5 already answers, double-counting O-10/O-11/O-12 against rule 3. It had to be dropped with drop_reason semantics (“discriminates nothing relevant”) that are simply false. The schema needs a state for background / enabling claim, distinct from dropped. At larger N this will silently discard real structure.
  2. Dropping a 4a merge survivor strands its absorbed nodes, breaking check 1. When 4b dropped H-3, H-4’s merged_into pointed at a dropped node, violating “merged ⇒ resolves in one hop to an active H” and aiming step 5’s resolution at a tombstone. 4a cannot anticipate this (it runs first, does no clustering) and step 4 gives 4b no cascade instruction. Fixed manually here (H-4 re-dispositioned to dropped, 4a merge preserved in former_merged_into + merge_note). Spec fix: state that dropping a survivor cascades to its merged_from set, and add a check that no merged_into/merged_from edge crosses a state boundary.
  3. affects_hypotheses is silently dead on any argument that also carries affects_observations. no_observation_arguments.py skips such arguments by design; step 8 reads only the edge’s arguments list (built from affects_observations); no script anywhere reads affects_hypotheses for scoring. So A-5 — the run’s single most load-bearing lab-leak argument, whose claim is prior-level — reached the model only as a likelihood modifier. Step 3 explicitly invites dual attachment, so this is not an exotic shape. Needs defined semantics, or a prohibition on dual attachment.
  4. The prior/likelihood partition mechanism was structurally unusable; HC-1’s prior rests on zero graph evidence. Step 7’s design is that marking edges used_for_prior is the partition between steps 7 and 8. All 9 edges sat in correlation groups and rule 5 requires all-or-none marking, so the only base-rate candidate (O-4) could not be admitted without deleting discriminating evidence (O-5) from step 8. Combined with #4, the prior received zero evidence input and zero argument input — it is entirely the child’s own reasoned base rates. The two rules are each right alone but compose to disable the partition.
  5. Correlation groups collapsed 1:1 onto sources, and the mechanism’s real case was never exercised. Because step 2 defaults a source’s data_basis to its own self-link, every observation from one paper trivially shares a basis and “correlation group” degenerates into “group by paper” (4 CGs = 4 papers, zero lone edges). What build_correlation_groups.py exists to catch — two different sources re-analysing one shared dataset — got no coverage, because the one shared external basis (D-2) has just one curated source; S-3, the reanalysis of D-2’s own raw reads, was cut at N=5. Step 8’s lone-edge path also got zero coverage. Before the real runs, deliberately curate ≥2 sources on one shared D node. Root fix candidate: stop defaulting data_basis to a self-link.
  6. Curation cuts by score, with no constraint that each evidence class survives — so N=5 removed an entire class. Case-geolocation/epidemiology is wholly absent from the curated set (S-1, S-7, S-13 sat at the pool’s top usefulness 4.5), as is S-5 (Bloom, market-linked ascertainment bias) — precisely the counterargument no curated source had an incentive to make, which step 9’s own process 3.4 warns to hunt for. This is the largest driver of the analysis’s weakness: the 42.4% residual is unfalsifiable because of what was cut. Step 2 should enforce class coverage, not just rank.
  7. Stale handoff note nearly caused duplicate extraction. orchestrator-continuation.md said “NEXT: Step 3” with a paste-ready extractor brief while step 3 was already complete on disk; following it literally would have minted a duplicate node set (O-13.., H-6.., A-6..), since create_node.py takes next-free ids and cannot detect that a paper was already extracted. Caught only by inventorying first. Fix: completion is recorded by the agent that finishes a step, never predicted ahead; a resuming orchestrator must verify on-disk state before trusting any “NEXT” pointer. (Note rewritten with a verify-first warning.)
  8. The Write tool refuses prose .md deliverables inside a subagent (“Subagents should return findings as text, not write report files”). Steps 9 and 10 both have prose .md artifacts, so both hit it; both worked around with a shell heredoc. Any future prose-artifact step will hit it too — either grant subagents write access for these paths or make the heredoc route explicit in the step files.
  9. external_consensus is spec’d as a field no step ever creates. Step 10 part 4 and its Interface treat it as a field consumed from steps 7-8, but steps 7/8 never mention it and step 9 explicitly says it enters “no number and no frontmatter field”. Step 10 is told to read something the spec elsewhere forbids creating.
  10. Step-1 searcher write-budgets were undersized against the brief’s own named anchors — a slice budgeted 6 written nodes while its balance paragraph named ~10 primary anchors across 4 required both-sides sub-areas; the searcher wrote 9 rather than drop a side. Size write-budgets at ≥1 node per explicitly-named anchor.

Analysis-quality findings (not pipeline defects, but worth carrying forward)

  1. Step 5 wrongly orphaned O-3 (market closed/disinfected 1 Jan, animal sampling only from 18 Jan). It does discriminate, but only through a path the graph cannot represent: it reaches the model as A-1’s comment-level correction of the CG-1 likelihood rather than as an edge. The step-5 child left it off deliberately as a test of that interface, and step 9 judged the result wrong. Evidence whose force is “this other observation is weak” has no first-class representation.
  2. Joint CG pricing trades away sensitivity granularity. CG-3 bundles O-9 (cuts against H-5) with O-10 (favours H-5) because both come from S-12. Judging them as one composite is correct — splitting would count S-12 as three witnesses — but it buries O-10’s isolable pro-H-5 signal, whose strength turns entirely on reservoir sampling density, inside a comment instead of exposing it as an overridable variable for step 10’s --set lever.
  3. H-5 silently bundles two hypotheses — escape of a collected, unmodified natural virus vs. escape of an engineered one — which fit O-11/O-12 very differently. With H-3 dropped (see #2), that distinction now exists nowhere in the graph, and lik_defuse_H5 = 1.0 averages over both.
  4. Sensitivity is concentrated in the prior, not the evidence. --set HC-1:uplift_coincident_lab_city=1.0 moves H-5 from 0.0809 to 0.0179 — a larger swing than any likelihood override tried. The lab-leak posterior is mostly one step-7 Fermi factor surviving weak evidence, not a finding the evidence produced.

Graph View