Stable run ids (fingerprints) for the form and the register; tests for the site statistics (#25)
CI / test (pull_request) Successful in 43s

- collection/etl/runs.py fingerprints(): a form or register run's id is now <prefix>-<hash of the row's non-empty
  answers, timestamp included> instead of its row number, so deleting or re-sorting rows in the sheet moves no
  other run's id, and a column added later changes none. The 32 rows that are exact copies of another row get
  -2, -3 (identical, so which is which does not matter). A row edited in the sheet gets a new id. Every id in
  runs_clean.csv and overlap.json changes once; nothing else does (no statistic or model reads the ids).
- tests/core/test_site_stats.py (9 tests): measures.yaml conditions, source groups and presets, bond
  denominators without deliberate deaths (cards, encounters, route map), the riders' paths, and a JSON-safe
  stats.json, on a fixed sample of the published runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-19 20:16:16 +00:00
co-authored by Claude Opus 5.5
parent ca411e8315
commit 1dfa8fb4c3
7 changed files with 9979 additions and 9818 deletions
+13
View File
@@ -359,6 +359,19 @@ class RedditComments(unittest.TestCase):
self.assertIn("THRESH-X1", text)
self.assertTrue(text.startswith(review_reddit.MARKER))
def test_fingerprint_ids_are_stable(self):
a = {"Timestamp": "2026-10-04 11:49:07", "Q1": "Alone", "Q2": None}
b = {"Timestamp": "2026-10-04 11:50:00", "Q1": "West"}
dup = {"Timestamp": "Data Import", "Q1": "Alone"}
ids = runs.fingerprints([a, b, dup, dict(dup)], "sheet")
self.assertEqual(len(set(ids)), 4)
self.assertTrue(all(i.startswith("sheet-") for i in ids))
self.assertTrue(ids[3].endswith("-2") and ids[3].startswith(ids[2])) # an exact copy gets -2
self.assertEqual(runs.fingerprints([b, a], "sheet"), [ids[1], ids[0]]) # re-sorting moves no id
self.assertEqual(runs.fingerprints([b], "sheet"), [ids[1]]) # deleting a row moves no other id
self.assertEqual(runs.fingerprints([{**a, "Q9": ""}], "sheet"), [ids[0]]) # a new, empty column changes none
self.assertNotEqual(runs.fingerprints([{**a, "Q1": "West"}], "sheet"), [ids[0]]) # an edited row is new
def test_delivered_not_extracted(self):
spec = read_yaml("collection/pipeline.yaml")["sources"]["reddit_comments"]
with self.assertRaises(extract.Skip):