Reddit scraper PRs get a review summary; the form/Reddit overlap is measured (#23, #24)
CI / test (pull_request) Successful in 40s
CI / test (pull_request) Successful in 40s
- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml: on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it to one comment. collection/README.md: "Reviewing Reddit runs". - analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`. - Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page) stays the one #18 applies. - data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
0df6b8d473
commit
98be27fc1f
@@ -41,6 +41,25 @@ The route PDF is a flowchart of the choices, not a log of runs, so it adds no ro
|
||||
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
|
||||
led there, so it adds no rows either.
|
||||
|
||||
## Reviewing Reddit runs
|
||||
|
||||
Ken's scraper opens (or updates) a pull request from `data/reddit-comments` that changes
|
||||
`data/public/reddit_comments.json`. The **Review Reddit runs** workflow (`.gitea/workflows/reddit-review.yml`) posts a
|
||||
summary on it, refreshed on every push (`python -m collection review-reddit --base <main's file>` locally):
|
||||
|
||||
- runs added, changed and gone since main;
|
||||
- **runs that break a game rule** (`core/spec/form.yaml` `impossible_flags`): left out of the route model and the
|
||||
statistics until corrected. Open the comment and decide;
|
||||
- other flags on the new runs (kept);
|
||||
- runs that may also be in the community form: the same answers within a day, with how many chance alone gives
|
||||
(`analysis/overlap/overlap.py`, `data/public/overlap.json`). Measured only, never removed;
|
||||
- runs whose result page is already a `reddit thread` row (counted once automatically).
|
||||
|
||||
Corrections go in `manual/reddit_comments.yaml`, applied at build time, so they never wait for the threads:
|
||||
`reject: [id]` (not a run), `fix: {id: {question: answer}}` (misread; `null` removes an answer) and
|
||||
`add: [{id, thread, posted, answers}]` (missed). A corrected run loses the "not checked by hand" note. No usernames:
|
||||
the comment id is enough to find the comment again. Then merge, once CI passes.
|
||||
|
||||
## Raw files and privacy
|
||||
|
||||
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
|
||||
|
||||
@@ -131,6 +131,18 @@ def open_issue(title, body, env=None, f=None):
|
||||
return _call("POST", f"{api_base(f)}/issues", {"title": title, "body": body}, tok, f)["number"]
|
||||
|
||||
|
||||
def upsert_comment(number, marker, body, env=None, f=None):
|
||||
"""Post `body` on issue or pull request `number`, replacing an earlier comment that starts with `marker`
|
||||
(so a summary refreshed on every push stays one comment)."""
|
||||
f = f or config()
|
||||
tok = token(env)
|
||||
comments = _call("GET", f"{api_base(f)}/issues/{number}/comments?limit=50&per_page=100", tok=tok, f=f) or []
|
||||
mine = [c for c in comments if (c.get("body") or "").startswith(marker)]
|
||||
if mine:
|
||||
return _call("PATCH", f"{api_base(f)}/issues/comments/{mine[-1]['id']}", {"body": body}, tok, f)["id"]
|
||||
return _call("POST", f"{api_base(f)}/issues/{number}/comments", {"body": body}, tok, f)["id"]
|
||||
|
||||
|
||||
def ensure_label(name, color, env=None, f=None):
|
||||
f = f or config()
|
||||
tok = token(env)
|
||||
|
||||
@@ -88,6 +88,7 @@ def rebuild():
|
||||
_step("what the runs show about the Black", [py, "analysis/black/evidence.py"])
|
||||
_step("screen tree", [py, "analysis/tree/build_tree.py"])
|
||||
_step("known answers for the bot (guide + screen tree)", [py, "analysis/known/build_known.py"])
|
||||
_step("runs reported twice (form + Reddit)", [py, "analysis/overlap/overlap.py"])
|
||||
try:
|
||||
import scipy # noqa: F401
|
||||
_step("statistics", [py, "analysis/stats/site_stats.py"])
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
"""A review summary for a Reddit scraper pull request (issue #23).
|
||||
|
||||
python -m collection review-reddit --base BASE.json [--out summary.md] [--pr N]
|
||||
|
||||
Compares the pull request's data/public/reddit_comments.json with `BASE.json` (the same file on main) and reads the
|
||||
runs as built from it (data/public/runs_clean.csv, so run `python -m collection build` first). The summary lists
|
||||
what a reviewer needs to decide on, before merging:
|
||||
|
||||
- runs added, changed (answers differ) and gone;
|
||||
- runs that break a game rule: left out of the route model and the statistics until corrected;
|
||||
- other flags on the new or changed runs (kept, for information);
|
||||
- runs that may also be in the community form (analysis/overlap/overlap.py: same answers within a day);
|
||||
- runs whose result page is already a "reddit thread" row (counted once, automatically).
|
||||
|
||||
Corrections go in collection/manual/reddit_comments.yaml (reject / fix / add; see collection/README.md). With
|
||||
`--pr N` the summary is posted on the pull request, replacing the previous one.
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
|
||||
from .common import ROOT, path, pipeline, read_yaml
|
||||
|
||||
MARKER = "<!-- atlas:reddit-review -->"
|
||||
PROVENANCE = "read from a Reddit comment"
|
||||
|
||||
|
||||
def _overlap():
|
||||
spec = importlib.util.spec_from_file_location("overlap", ROOT / "analysis/overlap/overlap.py")
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def link(run_id, thread):
|
||||
cid = str(run_id).split("-")[0]
|
||||
return f"[{run_id}](https://www.reddit.com/r/fourthwing/comments/{thread}/_/{cid}/)"
|
||||
|
||||
|
||||
def diff(base, head):
|
||||
"""(added, changed, gone) comment-run ids between two reddit_comments.json documents."""
|
||||
b = {str(r["id"]): r for r in base.get("runs", [])}
|
||||
h = {str(r["id"]): r for r in head.get("runs", [])}
|
||||
added = sorted(set(h) - set(b))
|
||||
gone = sorted(set(b) - set(h))
|
||||
changed = sorted(k for k in set(h) & set(b) if h[k].get("answers") != b[k].get("answers"))
|
||||
return added, changed, gone
|
||||
|
||||
|
||||
def summary(base, head, runs, known_results=(), overlap=None):
|
||||
"""The Markdown summary. `runs`: typed records of runs_clean.csv (with run_id)."""
|
||||
sys.path.insert(0, str(ROOT / "core"))
|
||||
from atlas import runs as runs_io
|
||||
|
||||
added, changed, gone = diff(base, head)
|
||||
focus = set(added) | set(changed)
|
||||
thread = {str(r["id"]): r.get("thread") for r in head.get("runs", [])}
|
||||
result = {str(r["id"]): r.get("result") for r in head.get("runs", [])}
|
||||
rows = {r["run_id"][len("comment-"):]: r for r in runs if r["source"] == "reddit comment"}
|
||||
flags = {k: [f for f in (r.get("data_flags") or "").split("; ") if f and not f.startswith(PROVENANCE)]
|
||||
for k, r in rows.items()}
|
||||
impossible = sorted(k for k in rows if runs_io.impossible(rows[k]))
|
||||
|
||||
L = [MARKER, "## Reddit comment runs: review summary", ""]
|
||||
L.append(f"**{len(head.get('runs', []))} runs** in this file: **{len(added)} new**, {len(changed)} changed, "
|
||||
f"{len(gone)} gone since main. {head.get('unread', 0)} comments mention a bond or a burn but could not "
|
||||
"be read as a run.")
|
||||
L.append("")
|
||||
|
||||
L += ["### Runs that break a game rule", "",
|
||||
"Left out of the route model and the statistics until corrected. Check the comment: `fix` a misread "
|
||||
"answer, or `reject` a run that is not one.", ""]
|
||||
bad = [k for k in impossible if k in focus] + [k for k in impossible if k not in focus]
|
||||
if bad:
|
||||
L += ["| Run | New? | Rule |", "|---|---|---|"]
|
||||
L += [f"| {link(k, thread.get(k) or '')} | {'yes' if k in focus else ''} | "
|
||||
f"{'; '.join(f for f in flags[k] if runs_io.impossible({'data_flags': f}))} |" for k in bad]
|
||||
else:
|
||||
L.append("None.")
|
||||
L.append("")
|
||||
|
||||
other = [k for k in sorted(focus) if k in flags and flags[k] and k not in impossible]
|
||||
L += ["### Other flags on new or changed runs (kept)", ""]
|
||||
L += [f"- {link(k, thread.get(k) or '')}: {'; '.join(flags[k])}" for k in other] or ["None."]
|
||||
L.append("")
|
||||
|
||||
if overlap is not None:
|
||||
twice = {m["comment"][len("comment-"):]: m["form"] for m in overlap.get("matches", [])}
|
||||
hits = [k for k in sorted(focus) if k in twice]
|
||||
L += ["### May also be in the community form", "",
|
||||
f"Same answers as a form row submitted within {overlap['rule']['window_days']} day(s). Over all Reddit "
|
||||
f"runs: {overlap['matched']} of {overlap['comparable']} comparable runs match, about "
|
||||
f"{overlap['expected_by_chance']} by chance, so about {overlap['likely_double']} look reported twice. "
|
||||
"Nothing is removed: identical routes are expected.", ""]
|
||||
L += [f"- {link(k, thread.get(k) or '')}: {', '.join(twice[k][:5])}" for k in hits] or ["None among the new runs."]
|
||||
L.append("")
|
||||
|
||||
counted = [k for k in sorted(focus) if result.get(k) and str(result[k]).upper() in known_results]
|
||||
L += ["### Result pages already counted", "",
|
||||
"Their result page is also a \"reddit thread\" row; the run is counted once, with its route.", ""]
|
||||
L += [f"- {link(k, thread.get(k) or '')}: {result[k]}" for k in counted] or ["None."]
|
||||
L.append("")
|
||||
|
||||
L += ["### How to correct", "",
|
||||
"In `collection/manual/reddit_comments.yaml` (applied at build time, on this branch or later):",
|
||||
"- `reject: [comment id]`: not a run (a joke, a friend's run, two runs told as one);",
|
||||
"- `fix: {comment id: {question: answer}}`: answers read wrong (`null` removes one);",
|
||||
"- `add: [{id, thread, posted, answers}]`: a run the reader missed.",
|
||||
"", "A corrected run loses the \"not checked by hand\" note. Then merge."]
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
|
||||
def main(base_file, out=None, pr=None):
|
||||
sys.path.insert(0, str(ROOT / "core"))
|
||||
from atlas import runs as runs_io
|
||||
|
||||
spec = pipeline()["sources"]["reddit_comments"]
|
||||
base = json.loads(open(base_file, encoding="utf-8").read()) if base_file else {"runs": []}
|
||||
head = json.loads(path(spec["load"]).read_text(encoding="utf-8"))
|
||||
runs = runs_io.read(drop=())
|
||||
thread_spec = pipeline()["sources"].get("reddit_thread") or {}
|
||||
manual = (thread_spec.get("transform") or {}).get("manual")
|
||||
known = {str(r["id"]).upper() for r in (read_yaml(manual)["results"] if manual else [])}
|
||||
text = summary(base, head, runs, known, _overlap().measure(runs))
|
||||
if out:
|
||||
open(out, "w", encoding="utf-8").write(text)
|
||||
else:
|
||||
print(text)
|
||||
if pr:
|
||||
from . import forge
|
||||
forge.upsert_comment(int(pr), MARKER, text)
|
||||
print(f"summary posted on #{pr}")
|
||||
return 0
|
||||
@@ -132,6 +132,10 @@ def main(argv=None):
|
||||
k.add_argument("--out", default="exports")
|
||||
sub.add_parser("rebuild", help="rebuild everything made from the runs: graph, route model, plan, reports, statistics, build")
|
||||
sub.add_parser("send-pdf", help="open a pull request on the bot's repo when the route PDF changed (private copy only)")
|
||||
rv = sub.add_parser("review-reddit", help="a review summary for a Reddit scraper pull request (run `build` first)")
|
||||
rv.add_argument("--base", help="data/public/reddit_comments.json as it is on main")
|
||||
rv.add_argument("--out", help="write the summary here (default: print it)")
|
||||
rv.add_argument("--pr", help="also post it on this pull request, replacing the previous summary")
|
||||
r = sub.add_parser("refresh", help="update every source, then rebuild everything (the daily workflow, locally)")
|
||||
r.add_argument("--bot", action="append", default=[], help="a bot folder or a .zip of its data (repeatable)")
|
||||
c = sub.add_parser("cycle", help="the loop with the bot: import its runs, refit, re-plan, deliver (safe on a timer)")
|
||||
@@ -155,6 +159,9 @@ def main(argv=None):
|
||||
if a.cmd == "rebuild":
|
||||
from .etl.handoff import rebuild
|
||||
return rebuild()
|
||||
if a.cmd == "review-reddit":
|
||||
from .etl.review_reddit import main as review
|
||||
return review(a.base, a.out, a.pr)
|
||||
if a.cmd == "send-pdf":
|
||||
from .etl.send_pdf import send
|
||||
return send()
|
||||
|
||||
Reference in New Issue
Block a user