Review summary on Reddit scraper PRs; measure the form/Reddit overlap #44

Merged
kathleen merged 1 commits from atlas/reddit-review into main 2026-11-19 16:50:49 +00:00
Owner

Closes #23. Closes #24.

#23: reviewing the scraper's PRs

  • New command and workflow: collection/etl/review_reddit.py (python -m collection review-reddit) and .gitea/workflows/reddit-review.yml. On every scraper PR they post one comment, refreshed on each push. It lists:
    • runs added, changed and gone;
    • runs that break a game rule, linked to their comments (left out of the model and stats until corrected);
    • other flags;
    • runs that may also be in the form;
    • result pages already counted;
    • how to reject, fix or add a run.
  • forge.upsert_comment keeps it to one comment. On today's 82 runs it finds the same 5 rule-breakers as the manual review.
  • collection/README.md gets a "Reviewing Reddit runs" section.

#24: the same run reported twice

  • The only proven copy is the one #18 already applies (a comment linked to its result page). Neither the form nor the register records a result-page id.
  • The overlap is measured instead: analysis/overlap/overlap.py writes data/public/overlap.json.
    • A Reddit run with at least 4 route answers matches a form row that gives the same answers within a day.
    • Chance is the same test with the dates shifted a few weeks.
    • 9 of the 36 comparable runs match, against about 7.3 by chance. So about 2 of 82 look reported twice: too few to matter. Nothing is removed.
  • Where it appears: the Data & methods tab, the dataset metadata, test:py (--check) and rebuild.

Also

  • data/public/stats.json: the statistics from #42 (impossible runs left out), which didn't make it into that merge.

🤖 Generated with Claude Code

Closes #23. Closes #24. ## #23: reviewing the scraper's PRs - **New command and workflow:** `collection/etl/review_reddit.py` (`python -m collection review-reddit`) and `.gitea/workflows/reddit-review.yml`. On every scraper PR they post one comment, refreshed on each push. It lists: - runs added, changed and gone; - runs that break a game rule, linked to their comments (left out of the model and stats until corrected); - other flags; - runs that may also be in the form; - result pages already counted; - how to `reject`, `fix` or `add` a run. - `forge.upsert_comment` keeps it to one comment. On today's 82 runs it finds the same 5 rule-breakers as the manual review. - `collection/README.md` gets a "Reviewing Reddit runs" section. ## #24: the same run reported twice - **The only proven copy is the one #18 already applies** (a comment linked to its result page). Neither the form nor the register records a result-page id. - **The overlap is measured instead:** `analysis/overlap/overlap.py` writes `data/public/overlap.json`. - A Reddit run with at least 4 route answers *matches* a form row that gives the same answers within a day. - Chance is the same test with the dates shifted a few weeks. - 9 of the 36 comparable runs match, against about 7.3 by chance. So about 2 of 82 look reported twice: too few to matter. Nothing is removed. - Where it appears: the Data & methods tab, the dataset metadata, `test:py` (`--check`) and `rebuild`. ## Also - **`data/public/stats.json`:** the statistics from #42 (impossible runs left out), which didn't make it into that merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kathleen added 1 commit 2026-11-19 16:50:01 +00:00
- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml:
  on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the
  runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also
  be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it
  to one comment. collection/README.md: "Reviewing Reddit runs".
- analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a
  form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs
  match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the
  page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`.
- Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page)
  stays the one #18 applies.
- data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
kathleen merged commit d7befc8e18 into main 2026-11-19 16:50:49 +00:00
kathleen deleted branch atlas/reddit-review 2026-11-19 16:50:49 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#44