Stable run ids (fingerprints) for the form and the register; tests for the site statistics #46

Merged
kathleen merged 1 commits from atlas/fingerprint-ids into main 2026-11-19 20:23:21 +00:00
Owner

Part of #25.

Stable run ids

  • collection/etl/runs.py fingerprints(): a form or register run's id is now sheet-<hash> or register-<hash>. The hash covers the row's non-empty answers, timestamp included. Before, the id was the row number.
    • Deleting or re-sorting rows in the sheet moves no other run's id.
    • A column added later, empty in old rows, changes none.
    • The 32 rows that are exact copies of another row get -2 and -3. They're identical, so which copy is which doesn't matter.
    • A row edited in the sheet gets a new id, because it's a different record.
  • Every form and register id in runs_clean.csv and overlap.json changes once; that's why the diff touches about 9,800 lines. Nothing else changes, since no statistic or model reads the ids.
  • New test: re-sorting, deleting, an empty new column, exact copies and an edited row.

Tests for the site statistics

tests/core/test_site_stats.py has 9 tests, run on a fixed sample of the published runs:

  • the conditions in measures.yaml;
  • source groups and presets: the bot and Reddit are never counted as bulk logs;
  • bond denominators without deliberate deaths, on the cards, the encounters and the route map;
  • the riders' paths layers;
  • a JSON-safe stats.json, with no NaN or Infinity, which is the bug #45 fixed in black.json.

Checks

  • test:py passes: 69 core tests and 31 collection tests.
  • All --checks pass, and the 21 JS tests pass.

🤖 Generated with Claude Code

Part of #25. ## Stable run ids - `collection/etl/runs.py` `fingerprints()`: a form or register run's id is now `sheet-<hash>` or `register-<hash>`. The hash covers the row's non-empty answers, timestamp included. Before, the id was the row number. - Deleting or re-sorting rows in the sheet moves no other run's id. - A column added later, empty in old rows, changes none. - The 32 rows that are exact copies of another row get `-2` and `-3`. They're identical, so which copy is which doesn't matter. - A row edited in the sheet gets a new id, because it's a different record. - Every form and register id in `runs_clean.csv` and `overlap.json` changes once; that's why the diff touches about 9,800 lines. Nothing else changes, since no statistic or model reads the ids. - New test: re-sorting, deleting, an empty new column, exact copies and an edited row. ## Tests for the site statistics `tests/core/test_site_stats.py` has 9 tests, run on a fixed sample of the published runs: - the conditions in `measures.yaml`; - source groups and presets: the bot and Reddit are never counted as bulk logs; - bond denominators without deliberate deaths, on the cards, the encounters and the route map; - the riders' paths layers; - a JSON-safe `stats.json`, with no NaN or Infinity, which is the bug #45 fixed in `black.json`. ## Checks - `test:py` passes: 69 core tests and 31 collection tests. - All `--check`s pass, and the 21 JS tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kathleen added 1 commit 2026-11-19 20:22:28 +00:00
- collection/etl/runs.py fingerprints(): a form or register run's id is now <prefix>-<hash of the row's non-empty
  answers, timestamp included> instead of its row number, so deleting or re-sorting rows in the sheet moves no
  other run's id, and a column added later changes none. The 32 rows that are exact copies of another row get
  -2, -3 (identical, so which is which does not matter). A row edited in the sheet gets a new id. Every id in
  runs_clean.csv and overlap.json changes once; nothing else does (no statistic or model reads the ids).
- tests/core/test_site_stats.py (9 tests): measures.yaml conditions, source groups and presets, bond
  denominators without deliberate deaths (cards, encounters, route map), the riders' paths, and a JSON-safe
  stats.json, on a fixed sample of the published runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
kathleen merged commit fbf42508a0 into main 2026-11-19 20:23:21 +00:00
kathleen deleted branch atlas/fingerprint-ids 2026-11-19 20:23:22 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#46