Use every possible run for the statistics and the route model; exclude only impossible runs #21

Closed
opened 2026-11-19 03:29:18 +00:00 by kathleen · 0 comments
Owner

Goal (Kathleen, 2026-11-18): the statistics and the route predictions use every run we have from every source that records a route. A run is left out only if it is impossible: it contradicts how the game works.

What is left out today

  • Statistics (analysis/stats/measures.yaml sources): only reddit spreadsheet. Reported runs, Reddit comment runs and the bot are left out.
  • Route model (planner/policy.yaml sources): reddit spreadsheet, reported, bot. Reddit comment runs (#18) are left out.
  • Flags (data_flags): they never exclude a run (except the "Rows without data-quality flags" preset). So runs that break game rules are in both today.

The change

  1. Sources. Both files take every route-recording source: reddit spreadsheet, reported, reddit comment, bot, and discord once it exists.
    • Ideally there's one list in one place, so a new source can't be forgotten in one of them.
    • Sources that record only the dragon (reddit thread, register) stay out of the route tables, but could count toward "which dragon bonded" totals.
  2. "Impossible" is defined in data, not code. Each data_flags reason is marked either impossible (the run is excluded) or doubtful (the run is kept and flagged). This list could live in core/spec/form.yaml. A first proposal:
    • impossible:
      • Green recorded on a valley run (there are no Greens in the valley);
      • died in combat on a path without combat;
      • more than one dragon marked Bonded;
      • a rare-dragon answer that doesn't match the path.
    • doubtful, kept:
      • first sighting outside the documented pool (the new PDF lists "possibly very rare encounters");
      • direction taken from the older column;
      • more than one dragon marked Incinerated (the burn order resolves it);
      • multiple wings;
      • a tree run with Orange/Brown but no Green.
  3. Bias stays handled by weighting, not exclusion. The reporting-bias correction (bias.py) and the low-bias sample already handle selective reporting. The bot's strategic choices are handled by its propensities and bot_mode. Neither needs runs dropped.
  4. The page says what it uses. It shows how many runs were excluded as impossible, and why.

Checks

  • Every source in runs_clean.csv that records a route is in both lists (a test).
  • The number of runs excluded as impossible is reported in stats.json and in the daily pull request.

Depends on #(plausibility checks), which sets the flags for the self-reported sources.

**Goal (Kathleen, 2026-11-18):** the statistics and the route predictions use every run we have from every source that records a route. A run is left out only if it is *impossible*: it contradicts how the game works. ## What is left out today - **Statistics** (`analysis/stats/measures.yaml` `sources`): only `reddit spreadsheet`. Reported runs, Reddit comment runs and the bot are left out. - **Route model** (`planner/policy.yaml` `sources`): `reddit spreadsheet`, `reported`, `bot`. Reddit comment runs (#18) are left out. - **Flags** (`data_flags`): they never exclude a run (except the "Rows without data-quality flags" preset). So runs that break game rules are *in* both today. ## The change 1. **Sources.** Both files take every route-recording source: `reddit spreadsheet`, `reported`, `reddit comment`, `bot`, and `discord` once it exists. - Ideally there's one list in one place, so a new source can't be forgotten in one of them. - Sources that record only the dragon (`reddit thread`, `register`) stay out of the route tables, but could count toward "which dragon bonded" totals. 2. **"Impossible" is defined in data, not code.** Each `data_flags` reason is marked either `impossible` (the run is excluded) or `doubtful` (the run is kept and flagged). This list could live in `core/spec/form.yaml`. A first proposal: - **impossible:** - Green recorded on a valley run (there are no Greens in the valley); - died in combat on a path without combat; - more than one dragon marked Bonded; - a rare-dragon answer that doesn't match the path. - **doubtful, kept:** - first sighting outside the documented pool (the new PDF lists "possibly very rare encounters"); - direction taken from the older column; - more than one dragon marked Incinerated (the burn order resolves it); - multiple wings; - a tree run with Orange/Brown but no Green. 3. **Bias stays handled by weighting, not exclusion.** The reporting-bias correction (`bias.py`) and the low-bias sample already handle selective reporting. The bot's strategic choices are handled by its propensities and `bot_mode`. Neither needs runs dropped. 4. **The page says what it uses.** It shows how many runs were excluded as impossible, and why. ## Checks - Every source in `runs_clean.csv` that records a route is in both lists (a test). - The number of runs excluded as impossible is reported in `stats.json` and in the daily pull request. Depends on #(plausibility checks), which sets the flags for the self-reported sources.
kathleen added the refactordata quality labels 2026-11-19 03:29:18 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#21