Refactor: analysis review (stable run ids, stats sources, bias assumptions) #25

Open
opened 2026-11-19 03:29:36 +00:00 by kathleen · 0 comments
Owner

This review is still open. The core, data-collection and site reviews are done. Only the shadow's reporting bias is left, and it waits on data.

  • Stable run ids (#46). Form and register runs are now sheet-<hash> / register-<hash>. The hash covers the row's non-empty answers and its timestamp, so deleting or re-sorting rows moves no id. Exact copies get -2, -3.

  • Statistics use every source.

    • The statistics now count the form, "Report a run", Reddit comments and the bot (#40), without impossible runs (#42).
    • Deliberate deaths are never counted as failed bonds (#40).
    • Checked how the statistics read for bot and Reddit runs. Questions those sources never ask (wing, section, squad) are left out of their factor tables; nothing shows as an "Unknown" group. Every bot run has malek_intent = Yes, which is correct: the bot never bonds a dragon it wasn't sent for and dies on purpose instead. In the all-runs factor tables, those runs fall in the "would seek Malek" group, as a form rider's would. The Reddit-only view has too few runs for the logistic models, which say so.
  • Reporting-bias assumptions (planner/policy.yaml reporting_bias). The shadow borrows the meadow-Blue bias, because no complete log has a first-attempt shadow yet:

    • bulk: 29 first attempts at the tree, 0 shadows;
    • bot: 515 tree runs, all later attempts.

    Waiting on Ken's fresh-account bot runs. Revisit once the bot logs first attempts at the tree.

  • Presenting the Black (#45). It's shown as a first-attempt range marked low evidence, and the valley map solves the Black's routes for a first attempt. Fixed along the way: black.json had Infinity, so the page's Black panels didn't load.

  • Black evidence (#47). Reddit comments get a column in the attempt table (3 gated shadow runs of 17), with a note on why they show what shadow runs look like, not how often the shadow comes. Reported runs get a column once there are any.

  • Tests for site_stats.py (#46): 9 tests on a sample of the real runs.

This review is still open. The core, data-collection and site reviews are done. Only the shadow's reporting bias is left, and it waits on data. - [x] **Stable run ids** (#46). Form and register runs are now `sheet-<hash>` / `register-<hash>`. The hash covers the row's non-empty answers and its timestamp, so deleting or re-sorting rows moves no id. Exact copies get `-2`, `-3`. - [x] **Statistics use every source.** - [x] The statistics now count the form, "Report a run", Reddit comments and the bot (#40), without impossible runs (#42). - [x] Deliberate deaths are never counted as failed bonds (#40). - [x] Checked how the statistics read for bot and Reddit runs. Questions those sources never ask (wing, section, squad) are left out of their factor tables; nothing shows as an "Unknown" group. Every bot run has `malek_intent = Yes`, which is correct: the bot never bonds a dragon it wasn't sent for and dies on purpose instead. In the all-runs factor tables, those runs fall in the "would seek Malek" group, as a form rider's would. The Reddit-only view has too few runs for the logistic models, which say so. - [ ] **Reporting-bias assumptions** (`planner/policy.yaml` `reporting_bias`). The shadow borrows the meadow-Blue bias, because no complete log has a first-attempt shadow yet: - bulk: 29 first attempts at the tree, 0 shadows; - bot: 515 tree runs, all later attempts. **Waiting on Ken's fresh-account bot runs.** Revisit once the bot logs first attempts at the tree. - [x] **Presenting the Black** (#45). It's shown as a first-attempt range marked low evidence, and the valley map solves the Black's routes for a first attempt. Fixed along the way: `black.json` had `Infinity`, so the page's Black panels didn't load. - [x] **Black evidence** (#47). Reddit comments get a column in the attempt table (3 gated shadow runs of 17), with a note on why they show what shadow runs look like, not how often the shadow comes. Reported runs get a column once there are any. - [x] **Tests** for `site_stats.py` (#46): 9 tests on a sample of the real runs.
kathleen added the refactor label 2026-11-19 03:29:36 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#25