Receive runs from the Reddit megathread comments (source "reddit comment")
CI / test (pull_request) Successful in 46s
CI / test (pull_request) Successful in 46s
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs riders describe in the r/fourthwing megathreads and delivers them here by pull request as data/public/reddit_comments.json: each run is the "Report a run" answers (option labels of collection/specs/report_form.yaml), with the comment id and day only; no usernames, no comment text. - pipeline.yaml: a reddit_comments source of extract kind "delivered" (the atlas fetches nothing; `update` skips it with a note) - runs.py: the coder, sharing one answers_row() with the reported coder; every row is flagged as read from a Reddit comment (and "not checked by hand" until corrected); a result page a comment already carries is not counted twice as a "reddit thread" row - collection/manual/reddit_comments.yaml: reject / fix / add, applied at build time - dictionary, SOURCES, README, croissant: the new source The route model and statistics do not use the source until it is named in planner/policy.yaml / analysis/stats/measures.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,7 @@ it exactly.
|
||||
|---|---|---|---|
|
||||
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
|
||||
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
|
||||
| reddit comment | `reddit_comments`: runs riders describe in the megathread comments | not extracted here: the [threshing-reddit](https://git.kenschiano.net/DragonKind/threshing-reddit) scraper reads the threads every 6 hours and opens (or updates) a pull request that changes the public file | `data/public/reddit_comments.json`: each comment read into the "Report a run" answers, no usernames or text; corrections by hand in `manual/reddit_comments.yaml` |
|
||||
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
|
||||
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
|
||||
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
|
||||
|
||||
@@ -64,6 +64,11 @@ def reddit(spec, env):
|
||||
return json.dumps(threads, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
def delivered(spec, env):
|
||||
"""A source another repository delivers by pull request (it changes the public file itself): nothing to fetch."""
|
||||
raise Skip(f"delivered by pull request from {spec.get('by') or 'another repository'}")
|
||||
|
||||
|
||||
def forge_issues(spec, env):
|
||||
"""Every issue with the label (open and closed), from the repository's host (forge.yaml: Gitea or GitHub)."""
|
||||
from . import forge
|
||||
@@ -107,7 +112,7 @@ def pack_local(spec, src):
|
||||
return json.dumps(files, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
EXTRACTORS = {"http": http, "reddit": reddit, "forge_issues": forge_issues, "local_dir": local_dir}
|
||||
EXTRACTORS = {"http": http, "reddit": reddit, "forge_issues": forge_issues, "local_dir": local_dir, "delivered": delivered}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the raw store
|
||||
|
||||
+82
-35
@@ -4,6 +4,7 @@ One coder per source (pipeline.yaml `runs:`). Every coder returns rows keyed by
|
||||
missing answers get the same "not known" values the community form's coding uses, so every source can be filtered
|
||||
the same way. The `source` column says where a run came from; `entry` keeps the spreadsheet's own row label.
|
||||
"""
|
||||
import functools
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
@@ -67,8 +68,13 @@ def spreadsheet(name, spec):
|
||||
|
||||
def reddit_results(name, spec):
|
||||
"""Dragons read from result pages linked in the megathreads: bonded, colour and tail known, route unknown."""
|
||||
from .common import pipeline
|
||||
comments = pipeline()["sources"].get("reddit_comments")
|
||||
told = {r.get("result") for r in _comment_runs(comments)} if comments else set()
|
||||
rows = []
|
||||
for r in read_yaml(spec["transform"]["manual"])["results"]:
|
||||
if r["id"] in told:
|
||||
continue # the same run, with its route, is a "reddit comment" row
|
||||
row = blank_row(run_id=f"thread-{r['id']}", source=spec["run_source"], outcome="Bonded",
|
||||
outcome_color=r["colour"], tail=r["tail"])
|
||||
encounter_columns(row)
|
||||
@@ -110,46 +116,86 @@ def register(name, spec):
|
||||
|
||||
def reports(name, spec):
|
||||
"""Runs reported through the "Report a run" issue form (collection/specs/report_form.yaml)."""
|
||||
form = report_form.spec()
|
||||
fields = {f["id"]: f for f in form["fields"]}
|
||||
p = path(spec["load"])
|
||||
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"reports": []}
|
||||
rows = []
|
||||
for rep in data["reports"]:
|
||||
vals, flags = {}, []
|
||||
for fid, answer in rep["answers"].items():
|
||||
f = fields.get(fid)
|
||||
if not f:
|
||||
continue
|
||||
if f.get("type") == "integer":
|
||||
m = re.search(r"\d+", answer)
|
||||
vals[f["column"]] = int(m.group()) if m else None
|
||||
elif answer in f["options"]:
|
||||
vals[f["column"]] = f["options"][answer]
|
||||
return [answers_row(f"reported-{rep['number']}", spec["run_source"], rep.get("opened"), rep["answers"])
|
||||
for rep in data["reports"]]
|
||||
|
||||
|
||||
def _comment_runs(spec):
|
||||
"""data/public/reddit_comments.json with the hand corrections of collection/manual/reddit_comments.yaml:
|
||||
`reject` (comment ids), `fix` (id -> answers that replace the read ones; null removes one) and `add` (runs the
|
||||
reader missed: {id, thread, posted, answers})."""
|
||||
p = path(spec["load"])
|
||||
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"runs": []}
|
||||
manual = read_yaml(spec["manual"]) if spec.get("manual") and path(spec["manual"]).exists() else {}
|
||||
reject = {str(x) for x in manual.get("reject") or []}
|
||||
fix = {str(k): v for k, v in (manual.get("fix") or {}).items()}
|
||||
out = []
|
||||
for run in list(data["runs"]) + list(manual.get("add") or []):
|
||||
if str(run["id"]) in reject or str(run["id"]).split("-")[0] in reject:
|
||||
continue
|
||||
answers = dict(run["answers"])
|
||||
for k, v in (fix.get(str(run["id"])) or {}).items():
|
||||
if v is None:
|
||||
answers.pop(k, None)
|
||||
else:
|
||||
flags.append(f"unrecognised answer to {fid}")
|
||||
row = blank_row(run_id=f"reported-{rep['number']}", source=spec["run_source"], date=rep.get("opened"), **vals)
|
||||
row["attempt_group"] = attempt_group(row["attempt"])
|
||||
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
|
||||
row["hour"] = "DIED before the hour"
|
||||
branch = row["branch"]
|
||||
if branch:
|
||||
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
|
||||
if row["hour"] == "Unknown":
|
||||
row["hour"] = "Tree" if tree else "Valley"
|
||||
elif row["hour"] != ("Tree" if tree else "Valley"):
|
||||
flags.append("encounter does not match the tree/valley answer")
|
||||
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
|
||||
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
|
||||
row["saw_blue_meadow"] = True
|
||||
if row["outcome"] == "Bonded" and not row["outcome_color"]:
|
||||
flags.append("bonded, colour not given")
|
||||
encounter_columns(row)
|
||||
row["data_flags"] = "; ".join(flags)
|
||||
rows.append(row)
|
||||
answers[k] = str(v)
|
||||
out.append({**run, "answers": answers, "checked": str(run["id"]) in fix or run in (manual.get("add") or [])})
|
||||
return out
|
||||
|
||||
|
||||
def reddit_comments(name, spec):
|
||||
"""Runs riders described in the megathread comments, read into the "Report a run" answers (automatic: every
|
||||
row is flagged, and those not checked by hand say so)."""
|
||||
rows = []
|
||||
for run in _comment_runs(spec):
|
||||
rows.append(answers_row(f"comment-{run['id']}", spec["run_source"], run.get("posted"), run["answers"],
|
||||
["read from a Reddit comment" + ("" if run["checked"] else " automatically, not checked by hand")]))
|
||||
return rows
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def _form_fields():
|
||||
return {f["id"]: f for f in report_form.spec()["fields"]}
|
||||
|
||||
|
||||
def answers_row(run_id, source, date, answers, flags=None):
|
||||
"""One run from the "Report a run" answers (option labels) -> a coded row."""
|
||||
fields = _form_fields()
|
||||
vals, flags = {}, list(flags or [])
|
||||
for fid, answer in answers.items():
|
||||
f = fields.get(fid)
|
||||
if not f:
|
||||
continue
|
||||
if f.get("type") == "integer":
|
||||
m = re.search(r"\d+", answer)
|
||||
vals[f["column"]] = int(m.group()) if m else None
|
||||
elif answer in f["options"]:
|
||||
vals[f["column"]] = f["options"][answer]
|
||||
else:
|
||||
flags.append(f"unrecognised answer to {fid}")
|
||||
row = blank_row(run_id=run_id, source=source, date=date, **vals)
|
||||
row["attempt_group"] = attempt_group(row["attempt"])
|
||||
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
|
||||
row["hour"] = "DIED before the hour"
|
||||
branch = row["branch"]
|
||||
if branch:
|
||||
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
|
||||
if row["hour"] == "Unknown":
|
||||
row["hour"] = "Tree" if tree else "Valley"
|
||||
elif row["hour"] != ("Tree" if tree else "Valley"):
|
||||
flags.append("encounter does not match the tree/valley answer")
|
||||
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
|
||||
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
|
||||
row["saw_blue_meadow"] = True
|
||||
if row["outcome"] == "Bonded" and not row["outcome_color"]:
|
||||
flags.append("bonded, colour not given")
|
||||
encounter_columns(row)
|
||||
row["data_flags"] = "; ".join(flags)
|
||||
return row
|
||||
|
||||
|
||||
def _norm(text):
|
||||
from .transform.bot import normalize
|
||||
return normalize(text)
|
||||
@@ -262,7 +308,8 @@ def bot(name, spec):
|
||||
return rows
|
||||
|
||||
|
||||
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reports": reports, "bot": bot, "register": register}
|
||||
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reddit_comments": reddit_comments,
|
||||
"reports": reports, "bot": bot, "register": register}
|
||||
|
||||
|
||||
def build_runs(pipe):
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# Hand corrections to the runs read from the r/fourthwing megathread comments (data/public/reddit_comments.json).
|
||||
# The threshing-reddit scraper reads them automatically; every new run it finds is listed, with a link to its
|
||||
# comment, in the pull request that brings it in. Check them there and correct them here. Applied when the
|
||||
# runs are built (`python -m collection build`), so a correction never waits for the threads to change.
|
||||
#
|
||||
# reject: comment ids whose "run" is not a run (a joke, a friend's run, two runs told as one). An id without
|
||||
# its "-n" suffix rejects every run read from that comment.
|
||||
# fix: comment id -> answers that replace the ones read (form option labels from
|
||||
# collection/specs/report_form.yaml; null removes an answer). A fixed run counts as checked by hand.
|
||||
# add: runs the reader missed: { id, thread, posted: YYYY-MM-DD, answers: {...} }
|
||||
# No usernames: the comment id is enough to find the comment again.
|
||||
version: 1
|
||||
reject: []
|
||||
fix: {}
|
||||
add: []
|
||||
@@ -18,7 +18,7 @@ manifest: data/manifest.json # for each source: the sha256 of the current
|
||||
runs:
|
||||
path: data/public/runs_clean.csv
|
||||
# sources in this order; within a source, the order the coder returns
|
||||
order: [reddit_spreadsheet, reddit_thread, register, reported, bot, reddit_pdf]
|
||||
order: [reddit_spreadsheet, reddit_thread, reddit_comments, register, reported, bot, reddit_pdf]
|
||||
|
||||
sources:
|
||||
reddit_spreadsheet:
|
||||
@@ -69,6 +69,24 @@ sources:
|
||||
load: null # the public source file is the hand-checked collection/manual/reddit_results.yaml
|
||||
runs: reddit_results
|
||||
|
||||
reddit_comments:
|
||||
label: Runs riders describe in the r/fourthwing Dragonkind megathread comments
|
||||
run_source: reddit comment
|
||||
links:
|
||||
part3: https://www.reddit.com/r/fourthwing/comments/1wvwfp9/
|
||||
part4: https://www.reddit.com/r/fourthwing/comments/1wz5cnj/
|
||||
scraper: https://git.kenschiano.net/DragonKind/threshing-reddit
|
||||
# delivered, not extracted: the threshing-reddit scraper (its own repository) reads the threads every 6 hours and
|
||||
# opens a pull request here that changes data/public/reddit_comments.json. Each comment is read into the
|
||||
# "Report a run" answers (option labels of specs/report_form.yaml); only the comment id, the day and the
|
||||
# answers are kept: no usernames, no comment text. `update` leaves this source alone.
|
||||
extract: { kind: delivered, by: threshing-reddit }
|
||||
transform: null
|
||||
load: data/public/reddit_comments.json
|
||||
# hand corrections (reject / fix / add a run), applied when the runs are built
|
||||
manual: collection/manual/reddit_comments.yaml
|
||||
runs: reddit_comments
|
||||
|
||||
reported:
|
||||
label: Runs reported on the Atlas ("Report a run", an issue form on the Atlas's repository)
|
||||
run_source: reported
|
||||
|
||||
Reference in New Issue
Block a user