Reddit scraper PRs get a review summary; the form/Reddit overlap is measured (#23, #24)
CI / test (pull_request) Successful in 40s

- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml:
  on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the
  runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also
  be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it
  to one comment. collection/README.md: "Reviewing Reddit runs".
- analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a
  form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs
  match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the
  page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`.
- Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page)
  stays the one #18 applies.
- data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-19 16:44:52 +00:00
co-authored by Claude Opus 5.5
parent 0df6b8d473
commit 98be27fc1f
15 changed files with 497 additions and 6 deletions
+19
View File
@@ -41,6 +41,25 @@ The route PDF is a flowchart of the choices, not a log of runs, so it adds no ro
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
led there, so it adds no rows either.
## Reviewing Reddit runs
Ken's scraper opens (or updates) a pull request from `data/reddit-comments` that changes
`data/public/reddit_comments.json`. The **Review Reddit runs** workflow (`.gitea/workflows/reddit-review.yml`) posts a
summary on it, refreshed on every push (`python -m collection review-reddit --base <main's file>` locally):
- runs added, changed and gone since main;
- **runs that break a game rule** (`core/spec/form.yaml` `impossible_flags`): left out of the route model and the
statistics until corrected. Open the comment and decide;
- other flags on the new runs (kept);
- runs that may also be in the community form: the same answers within a day, with how many chance alone gives
(`analysis/overlap/overlap.py`, `data/public/overlap.json`). Measured only, never removed;
- runs whose result page is already a `reddit thread` row (counted once automatically).
Corrections go in `manual/reddit_comments.yaml`, applied at build time, so they never wait for the threads:
`reject: [id]` (not a run), `fix: {id: {question: answer}}` (misread; `null` removes an answer) and
`add: [{id, thread, posted, answers}]` (missed). A corrected run loses the "not checked by hand" note. No usernames:
the comment id is enough to find the comment again. Then merge, once CI passes.
## Raw files and privacy
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
+12
View File
@@ -131,6 +131,18 @@ def open_issue(title, body, env=None, f=None):
return _call("POST", f"{api_base(f)}/issues", {"title": title, "body": body}, tok, f)["number"]
def upsert_comment(number, marker, body, env=None, f=None):
"""Post `body` on issue or pull request `number`, replacing an earlier comment that starts with `marker`
(so a summary refreshed on every push stays one comment)."""
f = f or config()
tok = token(env)
comments = _call("GET", f"{api_base(f)}/issues/{number}/comments?limit=50&per_page=100", tok=tok, f=f) or []
mine = [c for c in comments if (c.get("body") or "").startswith(marker)]
if mine:
return _call("PATCH", f"{api_base(f)}/issues/comments/{mine[-1]['id']}", {"body": body}, tok, f)["id"]
return _call("POST", f"{api_base(f)}/issues/{number}/comments", {"body": body}, tok, f)["id"]
def ensure_label(name, color, env=None, f=None):
f = f or config()
tok = token(env)
+1
View File
@@ -88,6 +88,7 @@ def rebuild():
_step("what the runs show about the Black", [py, "analysis/black/evidence.py"])
_step("screen tree", [py, "analysis/tree/build_tree.py"])
_step("known answers for the bot (guide + screen tree)", [py, "analysis/known/build_known.py"])
_step("runs reported twice (form + Reddit)", [py, "analysis/overlap/overlap.py"])
try:
import scipy # noqa: F401
_step("statistics", [py, "analysis/stats/site_stats.py"])
+133
View File
@@ -0,0 +1,133 @@
"""A review summary for a Reddit scraper pull request (issue #23).
python -m collection review-reddit --base BASE.json [--out summary.md] [--pr N]
Compares the pull request's data/public/reddit_comments.json with `BASE.json` (the same file on main) and reads the
runs as built from it (data/public/runs_clean.csv, so run `python -m collection build` first). The summary lists
what a reviewer needs to decide on, before merging:
- runs added, changed (answers differ) and gone;
- runs that break a game rule: left out of the route model and the statistics until corrected;
- other flags on the new or changed runs (kept, for information);
- runs that may also be in the community form (analysis/overlap/overlap.py: same answers within a day);
- runs whose result page is already a "reddit thread" row (counted once, automatically).
Corrections go in collection/manual/reddit_comments.yaml (reject / fix / add; see collection/README.md). With
`--pr N` the summary is posted on the pull request, replacing the previous one.
"""
import importlib.util
import json
import sys
from .common import ROOT, path, pipeline, read_yaml
MARKER = "<!-- atlas:reddit-review -->"
PROVENANCE = "read from a Reddit comment"
def _overlap():
spec = importlib.util.spec_from_file_location("overlap", ROOT / "analysis/overlap/overlap.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
def link(run_id, thread):
cid = str(run_id).split("-")[0]
return f"[{run_id}](https://www.reddit.com/r/fourthwing/comments/{thread}/_/{cid}/)"
def diff(base, head):
"""(added, changed, gone) comment-run ids between two reddit_comments.json documents."""
b = {str(r["id"]): r for r in base.get("runs", [])}
h = {str(r["id"]): r for r in head.get("runs", [])}
added = sorted(set(h) - set(b))
gone = sorted(set(b) - set(h))
changed = sorted(k for k in set(h) & set(b) if h[k].get("answers") != b[k].get("answers"))
return added, changed, gone
def summary(base, head, runs, known_results=(), overlap=None):
"""The Markdown summary. `runs`: typed records of runs_clean.csv (with run_id)."""
sys.path.insert(0, str(ROOT / "core"))
from atlas import runs as runs_io
added, changed, gone = diff(base, head)
focus = set(added) | set(changed)
thread = {str(r["id"]): r.get("thread") for r in head.get("runs", [])}
result = {str(r["id"]): r.get("result") for r in head.get("runs", [])}
rows = {r["run_id"][len("comment-"):]: r for r in runs if r["source"] == "reddit comment"}
flags = {k: [f for f in (r.get("data_flags") or "").split("; ") if f and not f.startswith(PROVENANCE)]
for k, r in rows.items()}
impossible = sorted(k for k in rows if runs_io.impossible(rows[k]))
L = [MARKER, "## Reddit comment runs: review summary", ""]
L.append(f"**{len(head.get('runs', []))} runs** in this file: **{len(added)} new**, {len(changed)} changed, "
f"{len(gone)} gone since main. {head.get('unread', 0)} comments mention a bond or a burn but could not "
"be read as a run.")
L.append("")
L += ["### Runs that break a game rule", "",
"Left out of the route model and the statistics until corrected. Check the comment: `fix` a misread "
"answer, or `reject` a run that is not one.", ""]
bad = [k for k in impossible if k in focus] + [k for k in impossible if k not in focus]
if bad:
L += ["| Run | New? | Rule |", "|---|---|---|"]
L += [f"| {link(k, thread.get(k) or '')} | {'yes' if k in focus else ''} | "
f"{'; '.join(f for f in flags[k] if runs_io.impossible({'data_flags': f}))} |" for k in bad]
else:
L.append("None.")
L.append("")
other = [k for k in sorted(focus) if k in flags and flags[k] and k not in impossible]
L += ["### Other flags on new or changed runs (kept)", ""]
L += [f"- {link(k, thread.get(k) or '')}: {'; '.join(flags[k])}" for k in other] or ["None."]
L.append("")
if overlap is not None:
twice = {m["comment"][len("comment-"):]: m["form"] for m in overlap.get("matches", [])}
hits = [k for k in sorted(focus) if k in twice]
L += ["### May also be in the community form", "",
f"Same answers as a form row submitted within {overlap['rule']['window_days']} day(s). Over all Reddit "
f"runs: {overlap['matched']} of {overlap['comparable']} comparable runs match, about "
f"{overlap['expected_by_chance']} by chance, so about {overlap['likely_double']} look reported twice. "
"Nothing is removed: identical routes are expected.", ""]
L += [f"- {link(k, thread.get(k) or '')}: {', '.join(twice[k][:5])}" for k in hits] or ["None among the new runs."]
L.append("")
counted = [k for k in sorted(focus) if result.get(k) and str(result[k]).upper() in known_results]
L += ["### Result pages already counted", "",
"Their result page is also a \"reddit thread\" row; the run is counted once, with its route.", ""]
L += [f"- {link(k, thread.get(k) or '')}: {result[k]}" for k in counted] or ["None."]
L.append("")
L += ["### How to correct", "",
"In `collection/manual/reddit_comments.yaml` (applied at build time, on this branch or later):",
"- `reject: [comment id]`: not a run (a joke, a friend's run, two runs told as one);",
"- `fix: {comment id: {question: answer}}`: answers read wrong (`null` removes one);",
"- `add: [{id, thread, posted, answers}]`: a run the reader missed.",
"", "A corrected run loses the \"not checked by hand\" note. Then merge."]
return "\n".join(L) + "\n"
def main(base_file, out=None, pr=None):
sys.path.insert(0, str(ROOT / "core"))
from atlas import runs as runs_io
spec = pipeline()["sources"]["reddit_comments"]
base = json.loads(open(base_file, encoding="utf-8").read()) if base_file else {"runs": []}
head = json.loads(path(spec["load"]).read_text(encoding="utf-8"))
runs = runs_io.read(drop=())
thread_spec = pipeline()["sources"].get("reddit_thread") or {}
manual = (thread_spec.get("transform") or {}).get("manual")
known = {str(r["id"]).upper() for r in (read_yaml(manual)["results"] if manual else [])}
text = summary(base, head, runs, known, _overlap().measure(runs))
if out:
open(out, "w", encoding="utf-8").write(text)
else:
print(text)
if pr:
from . import forge
forge.upsert_comment(int(pr), MARKER, text)
print(f"summary posted on #{pr}")
return 0
+7
View File
@@ -132,6 +132,10 @@ def main(argv=None):
k.add_argument("--out", default="exports")
sub.add_parser("rebuild", help="rebuild everything made from the runs: graph, route model, plan, reports, statistics, build")
sub.add_parser("send-pdf", help="open a pull request on the bot's repo when the route PDF changed (private copy only)")
rv = sub.add_parser("review-reddit", help="a review summary for a Reddit scraper pull request (run `build` first)")
rv.add_argument("--base", help="data/public/reddit_comments.json as it is on main")
rv.add_argument("--out", help="write the summary here (default: print it)")
rv.add_argument("--pr", help="also post it on this pull request, replacing the previous summary")
r = sub.add_parser("refresh", help="update every source, then rebuild everything (the daily workflow, locally)")
r.add_argument("--bot", action="append", default=[], help="a bot folder or a .zip of its data (repeatable)")
c = sub.add_parser("cycle", help="the loop with the bot: import its runs, refit, re-plan, deliver (safe on a timer)")
@@ -155,6 +159,9 @@ def main(argv=None):
if a.cmd == "rebuild":
from .etl.handoff import rebuild
return rebuild()
if a.cmd == "review-reddit":
from .etl.review_reddit import main as review
return review(a.base, a.out, a.pr)
if a.cmd == "send-pdf":
from .etl.send_pdf import send
return send()