Stable run ids (fingerprints) for the form and the register; tests for the site statistics (#25)
CI / test (pull_request) Successful in 43s
CI / test (pull_request) Successful in 43s
- collection/etl/runs.py fingerprints(): a form or register run's id is now <prefix>-<hash of the row's non-empty answers, timestamp included> instead of its row number, so deleting or re-sorting rows in the sheet moves no other run's id, and a column added later changes none. The 32 rows that are exact copies of another row get -2, -3 (identical, so which is which does not matter). A row edited in the sheet gets a new id. Every id in runs_clean.csv and overlap.json changes once; nothing else does (no statistic or model reads the ids). - tests/core/test_site_stats.py (9 tests): measures.yaml conditions, source groups and presets, bond denominators without deliberate deaths (cards, encounters, route map), the riders' paths, and a JSON-safe stats.json, on a fixed sample of the published runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
ca411e8315
commit
1dfa8fb4c3
@@ -359,6 +359,19 @@ class RedditComments(unittest.TestCase):
|
||||
self.assertIn("THRESH-X1", text)
|
||||
self.assertTrue(text.startswith(review_reddit.MARKER))
|
||||
|
||||
def test_fingerprint_ids_are_stable(self):
|
||||
a = {"Timestamp": "2026-10-04 11:49:07", "Q1": "Alone", "Q2": None}
|
||||
b = {"Timestamp": "2026-10-04 11:50:00", "Q1": "West"}
|
||||
dup = {"Timestamp": "Data Import", "Q1": "Alone"}
|
||||
ids = runs.fingerprints([a, b, dup, dict(dup)], "sheet")
|
||||
self.assertEqual(len(set(ids)), 4)
|
||||
self.assertTrue(all(i.startswith("sheet-") for i in ids))
|
||||
self.assertTrue(ids[3].endswith("-2") and ids[3].startswith(ids[2])) # an exact copy gets -2
|
||||
self.assertEqual(runs.fingerprints([b, a], "sheet"), [ids[1], ids[0]]) # re-sorting moves no id
|
||||
self.assertEqual(runs.fingerprints([b], "sheet"), [ids[1]]) # deleting a row moves no other id
|
||||
self.assertEqual(runs.fingerprints([{**a, "Q9": ""}], "sheet"), [ids[0]]) # a new, empty column changes none
|
||||
self.assertNotEqual(runs.fingerprints([{**a, "Q1": "West"}], "sheet"), [ids[0]]) # an edited row is new
|
||||
|
||||
def test_delivered_not_extracted(self):
|
||||
spec = read_yaml("collection/pipeline.yaml")["sources"]["reddit_comments"]
|
||||
with self.assertRaises(extract.Skip):
|
||||
|
||||
Reference in New Issue
Block a user