Author SHA1 Message Date
kschianoandClaude Opus 5.5 e457f89b40 Game-rule checks for runs answered through "Report a run"
Reported and Reddit-comment runs now get the checks the community form
gets in core/atlas/coding.py, as data_flags (the run is kept):
- first dragon outside the documented pool for the company/direction
- a bond or burn by a dragon the encounter does not have (new
  encounter_colours table in core/spec/form.yaml; Orange/Brown after the
  two Greens allowed, as a Green's rescue)
- a Green on a valley run
- a cadet outcome off its scream path, fight answers or a post-fight path
  on a run that did not help, a second fight move without assessing
- died before the hour, yet reached the tree/valley or an encounter

planner/policy.yaml drop_flagged: flagged runs of sources read from free
text (reddit comment) stay out of the route model; provenance notes do not
count as flags. No effect today: the source is not in `sources` yet.

On the first 82 scraped runs: 6 flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 08:46:13 -05:00
kschianoandClaude Opus 5.5 197c16dac0 Receive runs from the Reddit megathread comments (source "reddit comment")
CI / test (pull_request) Successful in 46s
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs
riders describe in the r/fourthwing megathreads and delivers them here by
pull request as data/public/reddit_comments.json: each run is the "Report a
run" answers (option labels of collection/specs/report_form.yaml), with the
comment id and day only; no usernames, no comment text.

- pipeline.yaml: a reddit_comments source of extract kind "delivered" (the
  atlas fetches nothing; `update` skips it with a note)
- runs.py: the coder, sharing one answers_row() with the reported coder;
  every row is flagged as read from a Reddit comment (and "not checked by
  hand" until corrected); a result page a comment already carries is not
  counted twice as a "reddit thread" row
- collection/manual/reddit_comments.yaml: reject / fix / add, applied at
  build time
- dictionary, SOURCES, README, croissant: the new source

The route model and statistics do not use the source until it is named in
planner/policy.yaml / analysis/stats/measures.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:06:14 -05:00
kathleen f3bbde24b7 New data from the community sources (#17)
Send the route model to the bot / send (push) Successful in 7s
CI / test (push) Successful in 39s
2026-11-19 02:55:25 +00:00
kathleen e61134ecc8 Merge pull request 'Send a changed route PDF to the bot by pull request' (#16) from send-pdf-to-bot into main
CI / test (push) Successful in 40s
2026-11-19 02:50:30 +00:00
kathleenandClaude Opus 5.5 66181b8489 Send a changed route PDF to the bot by pull request
CI / test (pull_request) Successful in 38s
The daily check keeps the community route PDF only in its private raw store (never published here). When that
copy differs from "Threshing Options.pdf" in the bot's private repo, `python -m collection send-pdf` opens (or
updates) a pull request there with the new PDF and the lines its text gained and lost, so Ken can update the
hand-written routes in src/pdfRoutes.js (the bot does not read the PDF itself).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:49:25 +00:00
24 changed files with 2821 additions and 2422 deletions
+6
View File
@@ -86,6 +86,12 @@ jobs:
with:
path: ${{ runner.temp }}/raw-archive
key: raw-archive-${{ hashFiles('data/manifest.json') }}
- name: Send a changed route PDF to the bot (its private repo; the PDF is never published here)
if: always() && steps.check.outcome == 'success'
continue-on-error: true
env:
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
run: python -m collection send-pdf
- name: Open a pull request with the new data (it merges itself once CI passes)
if: steps.check.outputs.changed == 'true' && !contains(steps.check.outputs.failed, 'build')
env:
+1 -1
View File
@@ -11,7 +11,7 @@ Written by `analysis/black/evidence.py` from `data/public/runs_clean.csv`. Count
| 3 | 0 / 193 (0.0%) | 0 / 14 (0.0%) | 0 / 5 (0.0%) |
| 4-10 | 0 / 538 (0.0%) | 0 / 102 (0.0%) | 0 / 36 (0.0%) |
| 11-30 | 0 / 626 (0.0%) | 0 / 57 (0.0%) | 0 / 137 (0.0%) |
| 31 or more | 0 / 693 (0.0%) | 0 / 0 | 0 / 333 (0.0%) |
| 31 or more | 0 / 694 (0.0%) | 0 / 0 | 0 / 333 (0.0%) |
| not given | 0 / 1581 (0.0%) | 0 / 149 (0.0%) | 0 / 0 |
Complete logs (bot and bulk) have 29 first attempts at the tree and 709 later ones, with 0 shadows. Zero in 709 later attempts rules out a rate above 0.4% for them; zero in 29 first attempts only rules out more than 10%.
+1
View File
@@ -31,6 +31,7 @@ it exactly.
|---|---|---|---|
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
| reddit comment | `reddit_comments`: runs riders describe in the megathread comments | not extracted here: the [threshing-reddit](https://git.kenschiano.net/DragonKind/threshing-reddit) scraper reads the threads every 6 hours and opens (or updates) a pull request that changes the public file | `data/public/reddit_comments.json`: each comment read into the "Report a run" answers, no usernames or text; corrections by hand in `manual/reddit_comments.yaml` |
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
+6 -1
View File
@@ -64,6 +64,11 @@ def reddit(spec, env):
return json.dumps(threads, ensure_ascii=False, sort_keys=True).encode("utf-8")
def delivered(spec, env):
"""A source another repository delivers by pull request (it changes the public file itself): nothing to fetch."""
raise Skip(f"delivered by pull request from {spec.get('by') or 'another repository'}")
def forge_issues(spec, env):
"""Every issue with the label (open and closed), from the repository's host (forge.yaml: Gitea or GitHub)."""
from . import forge
@@ -107,7 +112,7 @@ def pack_local(spec, src):
return json.dumps(files, ensure_ascii=False, sort_keys=True).encode("utf-8")
EXTRACTORS = {"http": http, "reddit": reddit, "forge_issues": forge_issues, "local_dir": local_dir}
EXTRACTORS = {"http": http, "reddit": reddit, "forge_issues": forge_issues, "local_dir": local_dir, "delivered": delivered}
# ---------------------------------------------------------------- the raw store
+134 -35
View File
@@ -4,6 +4,7 @@ One coder per source (pipeline.yaml `runs:`). Every coder returns rows keyed by
missing answers get the same "not known" values the community form's coding uses, so every source can be filtered
the same way. The `source` column says where a run came from; `entry` keeps the spreadsheet's own row label.
"""
import functools
import json
import re
from pathlib import Path
@@ -67,8 +68,13 @@ def spreadsheet(name, spec):
def reddit_results(name, spec):
"""Dragons read from result pages linked in the megathreads: bonded, colour and tail known, route unknown."""
from .common import pipeline
comments = pipeline()["sources"].get("reddit_comments")
told = {r.get("result") for r in _comment_runs(comments)} if comments else set()
rows = []
for r in read_yaml(spec["transform"]["manual"])["results"]:
if r["id"] in told:
continue # the same run, with its route, is a "reddit comment" row
row = blank_row(run_id=f"thread-{r['id']}", source=spec["run_source"], outcome="Bonded",
outcome_color=r["colour"], tail=r["tail"])
encounter_columns(row)
@@ -110,46 +116,138 @@ def register(name, spec):
def reports(name, spec):
"""Runs reported through the "Report a run" issue form (collection/specs/report_form.yaml)."""
form = report_form.spec()
fields = {f["id"]: f for f in form["fields"]}
p = path(spec["load"])
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"reports": []}
rows = []
for rep in data["reports"]:
vals, flags = {}, []
for fid, answer in rep["answers"].items():
f = fields.get(fid)
if not f:
continue
if f.get("type") == "integer":
m = re.search(r"\d+", answer)
vals[f["column"]] = int(m.group()) if m else None
elif answer in f["options"]:
vals[f["column"]] = f["options"][answer]
return [answers_row(f"reported-{rep['number']}", spec["run_source"], rep.get("opened"), rep["answers"])
for rep in data["reports"]]
def _comment_runs(spec):
"""data/public/reddit_comments.json with the hand corrections of collection/manual/reddit_comments.yaml:
`reject` (comment ids), `fix` (id -> answers that replace the read ones; null removes one) and `add` (runs the
reader missed: {id, thread, posted, answers})."""
p = path(spec["load"])
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"runs": []}
manual = read_yaml(spec["manual"]) if spec.get("manual") and path(spec["manual"]).exists() else {}
reject = {str(x) for x in manual.get("reject") or []}
fix = {str(k): v for k, v in (manual.get("fix") or {}).items()}
out = []
for run in list(data["runs"]) + list(manual.get("add") or []):
if str(run["id"]) in reject or str(run["id"]).split("-")[0] in reject:
continue
answers = dict(run["answers"])
for k, v in (fix.get(str(run["id"])) or {}).items():
if v is None:
answers.pop(k, None)
else:
flags.append(f"unrecognised answer to {fid}")
row = blank_row(run_id=f"reported-{rep['number']}", source=spec["run_source"], date=rep.get("opened"), **vals)
row["attempt_group"] = attempt_group(row["attempt"])
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
row["hour"] = "DIED before the hour"
branch = row["branch"]
if branch:
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
if row["hour"] == "Unknown":
row["hour"] = "Tree" if tree else "Valley"
elif row["hour"] != ("Tree" if tree else "Valley"):
flags.append("encounter does not match the tree/valley answer")
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
row["saw_blue_meadow"] = True
if row["outcome"] == "Bonded" and not row["outcome_color"]:
flags.append("bonded, colour not given")
encounter_columns(row)
row["data_flags"] = "; ".join(flags)
rows.append(row)
answers[k] = str(v)
out.append({**run, "answers": answers, "checked": str(run["id"]) in fix or run in (manual.get("add") or [])})
return out
def reddit_comments(name, spec):
"""Runs riders described in the megathread comments, read into the "Report a run" answers (automatic: every
row is flagged, and those not checked by hand say so)."""
rows = []
for run in _comment_runs(spec):
rows.append(answers_row(f"comment-{run['id']}", spec["run_source"], run.get("posted"), run["answers"],
["read from a Reddit comment" + ("" if run["checked"] else " automatically, not checked by hand")]))
return rows
@functools.lru_cache(maxsize=1)
def _form_fields():
return {f["id"]: f for f in report_form.spec()["fields"]}
def answers_row(run_id, source, date, answers, flags=None):
"""One run from the "Report a run" answers (option labels) -> a coded row."""
fields = _form_fields()
vals, flags = {}, list(flags or [])
for fid, answer in answers.items():
f = fields.get(fid)
if not f:
continue
if f.get("type") == "integer":
m = re.search(r"\d+", answer)
vals[f["column"]] = int(m.group()) if m else None
elif answer in f["options"]:
vals[f["column"]] = f["options"][answer]
else:
flags.append(f"unrecognised answer to {fid}")
row = blank_row(run_id=run_id, source=source, date=date, **vals)
row["attempt_group"] = attempt_group(row["attempt"])
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
row["hour"] = "DIED before the hour"
branch = row["branch"]
if branch:
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
if row["hour"] == "Unknown":
row["hour"] = "Tree" if tree else "Valley"
elif row["hour"] != ("Tree" if tree else "Valley"):
flags.append("encounter does not match the tree/valley answer")
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
row["saw_blue_meadow"] = True
if row["outcome"] == "Bonded" and not row["outcome_color"]:
flags.append("bonded, colour not given")
flags += rule_flags(row)
encounter_columns(row)
row["data_flags"] = "; ".join(flags)
return row
# the scream answer each cadet outcome follows (a run cannot end the encounter on another path)
CADET_PATH = {
"Killed the bullies yourself": "Investigate & help", "Squadmate's revenge": "Investigate & help",
"Left bullies to the dragons": "Investigate & help", "DIED in combat": "Investigate & help",
"Tourniquet": "Ignore the scream", "Left injured cadet": "Ignore the scream", "Took their weapons": "Ignore the scream",
"Left them to it": "Investigate & leave",
}
REACHED = ("Tree", "Valley")
@functools.lru_cache(maxsize=1)
def _form_rules():
from atlas import spec as atlas_spec
f = atlas_spec.form()
return f["sighting_pool"], {k: set(v) for k, v in f["encounter_colours"].items()}
def rule_flags(row):
"""The game's own rules, for runs answered through "Report a run": the checks core/atlas/coding.py makes on
the community form (first-dragon pools, Greens in the valley) plus the encounter's dragons and the cadet path.
Each broken rule is a flag; the run is kept."""
pools, enc_colours = _form_rules()
flags = []
company, direction, first = row["company"], row["direction"], row["first_sighting"]
if company in pools and direction in pools[company] and first in COLOURS:
pool = list(pools[company][direction]) + (list(pools["Alone"][direction]) if row["left_squadmates"] is True else [])
if first not in pool:
flags.append(f"first sighting {first} not in documented pool for {company}/{direction}")
colour, branch = row["outcome_color"], row["branch"]
if row["outcome"] in ("Bonded", "Incinerated") and colour:
if branch in enc_colours and colour not in enc_colours[branch]:
flags.append(f"{colour} at the {branch} encounter, which has only {'/'.join(sorted(enc_colours[branch]))}")
if row["hour"] == "Valley" and colour == "Green":
flags.append("Green recorded on a Valley run (no Greens in the valley)")
scream, res = row["scream"], row["resolution"]
if scream in ("Investigate & help", "Ignore the scream", "Investigate & leave"):
if res in CADET_PATH and CADET_PATH[res] != scream:
flags.append(f"cadet outcome {res!r} does not follow {scream!r}")
if scream != "Investigate & help" and (row["combat_first"] not in (None, "(not recorded)")
or row["combat_second"] not in (None, "(not recorded)")):
flags.append("fight answers on a run that did not help")
if scream != "Investigate & help" and row["post_path"] in ("Forest", "West", "East"):
flags.append("a path after the fight on a run without a fight")
if row["combat_second"] not in (None, "(not recorded)") and row["combat_first"] != "Assess the situation":
flags.append("a second fight move without pausing to assess")
died_early = (scream or "").startswith("DIED") or (res or "").startswith("DIED")
if died_early and (row["hour"] in REACHED or row["outcome"] in ("Bonded", "Incinerated")):
flags.append("died before the hour, yet reached the tree, the valley or a bonding encounter")
return flags
def _norm(text):
from .transform.bot import normalize
return normalize(text)
@@ -262,7 +360,8 @@ def bot(name, spec):
return rows
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reports": reports, "bot": bot, "register": register}
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reddit_comments": reddit_comments,
"reports": reports, "bot": bot, "register": register}
def build_runs(pipe):
+116
View File
@@ -0,0 +1,116 @@
"""Send the community route PDF to the bot's private repo when it changed (python -m collection send-pdf).
The PDF is never published by the Atlas: the daily check keeps it in the private raw store only
(data/raw/reddit_pdf/current.pdf). The bot's repo (private, same host) keeps its own copy, "Threshing Options.pdf",
which its hand-written routes (src/pdfRoutes.js) were transcribed from. When the two differ, this opens (or updates)
one pull request there, on branch atlas/route-pdf, with the new file and the lines its text gained and lost.
The token is ATLAS_BOT_TOKEN (write access to the bot's repo); git gets it through GIT_CONFIG_* variables, never argv.
"""
import hashlib
import io
import os
import shutil
import subprocess
import tempfile
from pathlib import Path
from . import forge
from .common import path
BOT_REPO = "DragonKind/DragonKind"
BOT_PDF = "Threshing Options.pdf"
BRANCH = "atlas/route-pdf"
RAW_PDF = "data/raw/reddit_pdf/current.pdf"
MAX_LINES = 60
def pdf_lines(data):
"""The PDF's text, one entry per non-empty line, whitespace collapsed."""
from pypdf import PdfReader
out = []
for page in PdfReader(io.BytesIO(data)).pages:
for line in (page.extract_text() or "").splitlines():
line = " ".join(line.split())
if line:
out.append(line)
return out
def text_changes(old, new):
"""(lines only in the new PDF, lines only in the old one), each in the order they appear."""
o, n = pdf_lines(old), pdf_lines(new)
so, sn = set(o), set(n)
gained = list(dict.fromkeys(l for l in n if l not in so))
lost = list(dict.fromkeys(l for l in o if l not in sn))
return gained, lost
def _bullets(lines):
if not lines:
return "- (none)"
shown = [f"- {l}" for l in lines[:MAX_LINES]]
if len(lines) > MAX_LINES:
shown.append(f"- ... and {len(lines) - MAX_LINES} more")
return "\n".join(shown)
def body(new, old, gained, lost):
h = lambda b: hashlib.sha256(b).hexdigest()[:12]
return (
"The community route PDF (u/ClumsyBadger's Threshing Options, the Google Drive file linked from the "
"megathreads) changed. This replaces `Threshing Options.pdf` with the current version.\n\n"
f"| | bytes | sha256 |\n|---|---|---|\n| this repo's copy | {len(old):,} | `{h(old)}` |\n"
f"| current PDF | {len(new):,} | `{h(new)}` |\n\n"
"**The bot does not read the PDF itself**: its routes are written out by hand in `src/pdfRoutes.js`. "
"Check them against the changes below.\n\n"
f"### Text that is new in this version\n{_bullets(gained)}\n\n"
f"### Text no longer in it\n{_bullets(lost)}\n\n"
"(Text extracted line by line from a flowchart, so some lines are fragments of boxes.) "
"Opened automatically by the Threshing Atlas's daily source check; it is updated in place when the PDF "
"changes again.\n"
)
def _git(args, cwd=None, env=None):
subprocess.run(["git", *args], cwd=cwd, env=env, check=True, capture_output=True, text=True)
def send(env=None):
env = dict(os.environ if env is None else env)
src = path(RAW_PDF)
if not src.exists():
print("no route PDF in the raw store: nothing to send")
return 0
tok = forge.token(env)
if not tok:
print("ATLAS_BOT_TOKEN is not set: the PDF is not sent")
return 0
f = forge.config()
host = f["url"].rstrip("/")
genv = {**env, "GIT_TERMINAL_PROMPT": "0", "GIT_CONFIG_COUNT": "1",
"GIT_CONFIG_KEY_0": f"http.{host}/.extraheader", "GIT_CONFIG_VALUE_0": f"AUTHORIZATION: token {tok}"}
new = src.read_bytes()
tmp = Path(tempfile.mkdtemp(prefix="dragonkind-"))
try:
repo = tmp / "repo"
_git(["clone", "--quiet", "--depth", "1", f"{host}/{BOT_REPO}.git", str(repo)], env=genv)
dst = repo / BOT_PDF
old = dst.read_bytes() if dst.exists() else b""
if old == new:
print("the bot already has the current route PDF")
return 0
gained, lost = text_changes(old, new) if old else (pdf_lines(new), [])
dst.write_bytes(new)
_git(["config", "user.name", "Atlas data bot"], cwd=repo)
_git(["config", "user.email", "atlas-bot@noreply.localhost"], cwd=repo)
_git(["checkout", "-q", "-b", BRANCH], cwd=repo)
_git(["add", BOT_PDF], cwd=repo)
_git(["commit", "-q", "-m", f"Community route PDF: current version ({len(new):,} bytes)"], cwd=repo)
_git(["push", "-q", "--force", "origin", BRANCH], cwd=repo, env=genv)
n = forge.open_pr(BRANCH, "Community route PDF changed", body(new, old, gained, lost), env=env,
repo=BOT_REPO, base="main")
print(f"route PDF sent: pull request #{n} on {BOT_REPO} ({len(gained)} lines new, {len(lost)} gone)")
return 0
finally:
shutil.rmtree(tmp, ignore_errors=True)
+15
View File
@@ -0,0 +1,15 @@
# Hand corrections to the runs read from the r/fourthwing megathread comments (data/public/reddit_comments.json).
# The threshing-reddit scraper reads them automatically; every new run it finds is listed, with a link to its
# comment, in the pull request that brings it in. Check them there and correct them here. Applied when the
# runs are built (`python -m collection build`), so a correction never waits for the threads to change.
#
# reject: comment ids whose "run" is not a run (a joke, a friend's run, two runs told as one). An id without
# its "-n" suffix rejects every run read from that comment.
# fix: comment id -> answers that replace the ones read (form option labels from
# collection/specs/report_form.yaml; null removes an answer). A fixed run counts as checked by hand.
# add: runs the reader missed: { id, thread, posted: YYYY-MM-DD, answers: {...} }
# No usernames: the comment id is enough to find the comment again.
version: 1
reject: []
fix: {}
add: []
+19 -1
View File
@@ -18,7 +18,7 @@ manifest: data/manifest.json # for each source: the sha256 of the current
runs:
path: data/public/runs_clean.csv
# sources in this order; within a source, the order the coder returns
order: [reddit_spreadsheet, reddit_thread, register, reported, bot, reddit_pdf]
order: [reddit_spreadsheet, reddit_thread, reddit_comments, register, reported, bot, reddit_pdf]
sources:
reddit_spreadsheet:
@@ -69,6 +69,24 @@ sources:
load: null # the public source file is the hand-checked collection/manual/reddit_results.yaml
runs: reddit_results
reddit_comments:
label: Runs riders describe in the r/fourthwing Dragonkind megathread comments
run_source: reddit comment
links:
part3: https://www.reddit.com/r/fourthwing/comments/1wvwfp9/
part4: https://www.reddit.com/r/fourthwing/comments/1wz5cnj/
scraper: https://git.kenschiano.net/DragonKind/threshing-reddit
# delivered, not extracted: the threshing-reddit scraper (its own repository) reads the threads every 6 hours and
# opens a pull request here that changes data/public/reddit_comments.json. Each comment is read into the
# "Report a run" answers (option labels of specs/report_form.yaml); only the comment id, the day and the
# answers are kept: no usernames, no comment text. `update` leaves this source alone.
extract: { kind: delivered, by: threshing-reddit }
transform: null
load: data/public/reddit_comments.json
# hand corrections (reject / fix / add a run), applied when the runs are built
manual: collection/manual/reddit_comments.yaml
runs: reddit_comments
reported:
label: Runs reported on the Atlas ("Report a run", an issue form on the Atlas's repository)
run_source: reported
+4
View File
@@ -131,6 +131,7 @@ def main(argv=None):
k = sub.add_parser("export-bot-kit", help="zip policy.json, experiments.json, solver.js and graph.js for the bot")
k.add_argument("--out", default="exports")
sub.add_parser("rebuild", help="rebuild everything made from the runs: graph, route model, plan, reports, statistics, build")
sub.add_parser("send-pdf", help="open a pull request on the bot's repo when the route PDF changed (private copy only)")
r = sub.add_parser("refresh", help="update every source, then rebuild everything (the daily workflow, locally)")
r.add_argument("--bot", action="append", default=[], help="a bot folder or a .zip of its data (repeatable)")
c = sub.add_parser("cycle", help="the loop with the bot: import its runs, refit, re-plan, deliver (safe on a timer)")
@@ -154,6 +155,9 @@ def main(argv=None):
if a.cmd == "rebuild":
from .etl.handoff import rebuild
return rebuild()
if a.cmd == "send-pdf":
from .etl.send_pdf import send
return send()
if a.cmd == "refresh":
from .etl.handoff import refresh
return refresh(main, a.bot)
+11
View File
@@ -153,6 +153,17 @@ sighting_pool:
1 Squadmate: { East: [Green, Orange, Red], West: [Red, Brown, Green], Center: [Red, Brown] }
2 Squadmates: { East: [Red, Orange], West: [Green, Brown, Red], Center: [Red, Brown] }
# The dragons each bonding encounter can end with (bond or burn), from the community PDF. Only used to FLAG runs
# answered through "Report a run" (reported, reddit comment) that end with a dragon the encounter does not have.
# Orange and Brown after the two Greens: a Green's rescue, then the Orange or Brown.
encounter_colours:
"Shadow (rare)": [Blue, Black]
"Descend quickly: Red": [Red]
"Descend quickly: let the Red pass": [Green]
"Descend slowly: two Greens": [Green, Orange, Brown]
"Valley Red": [Red]
"Passed the Red: meadow": [Orange, Brown, Blue]
# First-sighting answers that are not a single colour: coded as this colour, and the row is flagged with the
# original answer (data_flags), so nothing is lost.
first_sighting_alias:
+1
View File
@@ -7,6 +7,7 @@ Every source is declared in `collection/pipeline.yaml` and collected by `python
|---|---|
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
| Runs described in the r/fourthwing megathread comments (`reddit comment`) | `public/reddit_comments.json` (comment id, day and answers; no usernames or comment text) |
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
+27 -12
View File
@@ -78,6 +78,11 @@
"@type": "sc:CreativeWork",
"name": "Runs posted in the r/fourthwing Dragonkind megathreads",
"url": "https://www.reddit.com/r/fourthwing/comments/1wvwfp9/"
},
{
"@type": "sc:CreativeWork",
"name": "Runs riders describe in the r/fourthwing Dragonkind megathread comments",
"url": "https://www.reddit.com/r/fourthwing/comments/1wvwfp9/"
}
],
"distribution": [
@@ -88,8 +93,8 @@
"description": "Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's rules are core/spec/form.yaml). The `source` column says where each run came from.",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/runs_clean.csv",
"encodingFormat": "text/csv",
"sha256": "5d67698cd1083c268891eabe8f94f0e7d59c33a5866973b4738cdd0bb929a74c",
"contentSize": "3679922 B"
"sha256": "c5b7302bee16c063a400548e8bdf95256df50acbc00f2c6e9813eed7f50c42d4",
"contentSize": "3680593 B"
},
{
"@type": "cr:FileObject",
@@ -98,8 +103,18 @@
"description": "The community \"Threshing Routes\" form export as published (headers and rows), with email addresses in free-text answers redacted. The source of runs_clean.",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/form_responses.json",
"encodingFormat": "application/json",
"sha256": "3a07ae1755536a2a48bca63ed3d2874488bfe9bb772f9b54d704f50569eecb27",
"contentSize": "12620781 B"
"sha256": "319f6217e0a431454114a67276f429379ec383cd6ed2666443b48b8f48c66490",
"contentSize": "12623695 B"
},
{
"@type": "cr:FileObject",
"@id": "reddit_comments",
"name": "reddit_comments",
"description": "Runs described in the r/fourthwing megathread comments, read into the \"Report a run\" answers: comment id, day and answers only, no usernames or comment text.",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/reddit_comments.json",
"encodingFormat": "application/json",
"sha256": "c180f8b82ed1119d4ff28697fcf45d0c7f22859597cd10f5e82a2e50ca1dc51a",
"contentSize": "105 B"
},
{
"@type": "cr:FileObject",
@@ -128,8 +143,8 @@
"description": "Every number the Atlas page shows, for six filter presets (analysis/stats/site_stats.py; what is measured is analysis/stats/measures.yaml).",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/stats.json",
"encodingFormat": "application/json",
"sha256": "a2958e4a3b2dd4dafbd6376f22536848b94e664d34b74ceee8fdc8ac9d95a260",
"contentSize": "5462331 B"
"sha256": "75ed4be2199b2192e904ce7697892eed01abfefc4e897cd3fbeaf0c7ed4c637c",
"contentSize": "5461481 B"
},
{
"@type": "cr:FileObject",
@@ -138,8 +153,8 @@
"description": "The route model: the game (core/spec/game.yaml), the count tables behind every roll, and the best decision tree for every dragon colour under two goals, with posterior intervals (planner/build_policy.py).",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/policy.json",
"encodingFormat": "application/json",
"sha256": "1ba447faaa01f8898829fa5319a7c82795c647460b7f7edb7b5ac70c0cf92b8a",
"contentSize": "986068 B"
"sha256": "8360beabea06d77117ddd10741c4e004cf361f8bd3d6377286bb053d2ba6e1b0",
"contentSize": "986194 B"
},
{
"@type": "cr:FileObject",
@@ -148,8 +163,8 @@
"description": "The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).",
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/graph.json",
"encodingFormat": "application/json",
"sha256": "a1d7c767e672298ab5b076f586fb5692466240b2f7791fb04e4fd826ca76e35c",
"contentSize": "75567 B"
"sha256": "f4d4d02d5ce0d57de2c706a84874133aab9b5c915c59aeef9760b3141b3044e0",
"contentSize": "75584 B"
}
],
"recordSet": [
@@ -166,7 +181,7 @@
"@type": "cr:Field",
"@id": "runs/run_id",
"name": "run_id",
"description": "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>.",
"description": "Unique id of the run: sheet-<row of the form export>, thread-<result id>, comment-<Reddit comment id>, reported-<issue number> or bot-<hash>.",
"dataType": "sc:Text",
"source": {
"fileObject": {
@@ -181,7 +196,7 @@
"@type": "cr:Field",
"@id": "runs/source",
"name": "source",
"description": "Where the run came from. \"reddit spreadsheet\": the community Threshing Routes form and spreadsheet linked from the r/fourthwing megathreads. \"reddit thread\": a dragonkind.com result page posted in a megathread (bonded, route not recorded). \"reported\": the Atlas's own \"Report a run\" form. \"bot\": an attempt logged by the DragonKind bot. \"register\": a bonded dragon entered in the community's Unofficial Dragon Register (outcome only, no route; the same run may also be in the spreadsheet). \"reddit pdf\": the community route PDF (a flowchart of the routes, not a log of runs, so it adds no rows today). Values: reddit spreadsheet; reddit thread; register; reported; bot; reddit pdf.",
"description": "Where the run came from. \"reddit spreadsheet\": the community Threshing Routes form and spreadsheet linked from the r/fourthwing megathreads. \"reddit thread\": a dragonkind.com result page posted in a megathread (bonded, route not recorded). \"reddit comment\": a run a rider described in a megathread comment, read automatically into the \"Report a run\" answers by the threshing-reddit scraper, which delivers them by pull request (hand corrections in collection/manual/reddit_comments.yaml). \"reported\": the Atlas's own \"Report a run\" form. \"bot\": an attempt logged by the DragonKind bot. \"register\": a bonded dragon entered in the community's Unofficial Dragon Register (outcome only, no route; the same run may also be in the spreadsheet). \"reddit pdf\": the community route PDF (a flowchart of the routes, not a log of runs, so it adds no rows today). Values: reddit spreadsheet; reddit thread; reddit comment; register; reported; bot; reddit pdf.",
"dataType": "sc:Text",
"source": {
"fileObject": {
+11 -3
View File
@@ -35,6 +35,12 @@ files:
description: >-
The community "Threshing Routes" form export as published (headers and rows), with email addresses in
free-text answers redacted. The source of runs_clean.
reddit_comments:
path: data/public/reddit_comments.json
format: application/json
description: >-
Runs described in the r/fourthwing megathread comments, read into the "Report a run" answers: comment id, day
and answers only, no usernames or comment text.
reports:
path: data/public/reports.json
format: application/json
@@ -63,18 +69,20 @@ files:
# The columns of runs_clean. type: text, integer, boolean or date. values: every value the column may take
# (empty cells are always allowed and mean "does not apply" or "not recorded").
columns:
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>." }
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, comment-<Reddit comment id>, reported-<issue number> or bot-<hash>." }
source:
type: text
required: true
description: >-
Where the run came from. "reddit spreadsheet": the community Threshing Routes form and spreadsheet linked from
the r/fourthwing megathreads. "reddit thread": a dragonkind.com result page posted in a megathread (bonded,
route not recorded). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
route not recorded). "reddit comment": a run a rider described in a megathread comment, read automatically into
the "Report a run" answers by the threshing-reddit scraper, which delivers them by pull request (hand
corrections in collection/manual/reddit_comments.yaml). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
DragonKind bot. "register": a bonded dragon entered in the community's Unofficial Dragon Register (outcome
only, no route; the same run may also be in the spreadsheet). "reddit pdf": the community route PDF (a
flowchart of the routes, not a log of runs, so it adds no rows today).
values: [reddit spreadsheet, reddit thread, register, reported, bot, reddit pdf]
values: [reddit spreadsheet, reddit thread, reddit comment, register, reported, bot, reddit pdf]
entry:
type: text
description: >-
+3 -3
View File
@@ -10,12 +10,12 @@
"changed": "2026-11-16"
},
"reddit_spreadsheet": {
"sha256": "0f54f7695accdeeb1be4bf792f8f30da7ab1ffbe6c1f7bbdaaea8b68f0af1f00",
"bytes": 5946448,
"sha256": "e3cccdbdf8ce67d7ea8e85156ccd5b5f5e404ab6cd195ee8883ea140249d16a6",
"bytes": 5949959,
"changed": "2026-11-19"
},
"register": {
"sha256": "9091f67ecce0dd62210443cf98a0d004027b21839182b7fb280ede0e0dd9e8c6",
"sha256": "ea9807f6d2391ecce7425888b913d4b618ac85463a459e3908abf1233c888f19",
"bytes": 51331,
"changed": "2026-11-19"
},
+2 -2
View File
@@ -24,7 +24,7 @@
],
"31 or more": [
0,
693
694
]
},
"bulk logs": {
@@ -93,7 +93,7 @@
],
"form_later": [
0,
2346
2347
],
"best_route": {
"route": [
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
{"source":"Runs riders describe in the r/fourthwing Dragonkind megathread comments","runs":[],"unread":0}
+2361 -2359
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+10 -1
View File
@@ -36,6 +36,15 @@ def low_bias(r, lb=CONFIG["low_bias"]):
return r["goal"] not in lb["rare_goals"]
def drop_flagged(runs, cfg=CONFIG.get("drop_flagged") or {}):
"""Without the flagged runs of the sources read automatically from free text (planner/policy.yaml drop_flagged)."""
sources, keep = set(cfg.get("sources") or ()), tuple(cfg.get("except") or ())
def problems(r):
return [f for f in (r.get("data_flags") or "").split("; ") if f and not f.startswith(keep)] if keep else [f for f in (r.get("data_flags") or "").split("; ") if f]
return [r for r in runs if r["source"] not in sources or not problems(r)]
def clean(o):
"""JSON-safe: numpy numbers to floats, infinities and NaN to null (as JavaScript's JSON.stringify does)."""
if isinstance(o, dict):
@@ -56,7 +65,7 @@ def main(argv=None):
ap.add_argument("--out", default=str(ROOT / "data/public/policy.json"))
a = ap.parse_args(argv)
runs = runs_io.read(a.data, sources=SOURCES)
runs = drop_flagged(runs_io.read(a.data, sources=SOURCES))
rows = len(runs)
samples = {
"all": runs,
+8
View File
@@ -8,6 +8,14 @@ version: 1
# the `source` values of data/public/runs_clean.csv the model learns from
sources: [reddit spreadsheet, reported, bot]
# Sources read automatically from free text: a run with any data flag (a broken game rule, an unrecognised answer)
# is left out of the model, since the flag may mean the text was misread or misremembered. Flags that only say where
# the run came from (`except`, matched at the start of a flag) do not count. A hand fix that clears the flags
# brings the run back. Applies once the source is also listed in `sources` above.
drop_flagged:
sources: [reddit comment]
except: ["read from a Reddit comment"]
# Two samples. "all" is every run above. "lowBias" leaves out runs where a rare sighting is more likely to have
# been reported than a miss: form submissions and reports from riders chasing Blue or Black. The bot logs every
# run, and bulk-pasted logs record every run of the players who kept them, so both stay in.
+80
View File
@@ -259,6 +259,86 @@ class Reddit(unittest.TestCase):
self.assertNotIn("someone", json.dumps(out))
class RedditComments(unittest.TestCase):
"""Runs the threshing-reddit scraper delivers (data/public/reddit_comments.json), with hand corrections."""
def source(self, public, manual):
spec = read_yaml("collection/pipeline.yaml")["sources"]["reddit_comments"]
return {**spec, "load": str(public), "manual": str(manual)}
def test_coded_with_hand_corrections(self):
public = TMP / "reddit_comments.json"
public.write_text(json.dumps({"runs": [
{"id": "c1", "thread": "abc", "posted": "2026-11-17", "result": None,
"answers": {"company": "Alone", "direction": "East (toward the roar)", "outcome": "Bonded",
"outcome_color": "Green", "branch": "Tree: climbed down slowly to the two Greens"}},
{"id": "c2-1", "thread": "abc", "posted": "2026-11-17", "result": None,
"answers": {"company": "Alone", "outcome": "Bonded", "outcome_color": "Red"}},
{"id": "c3", "thread": "abc", "posted": None, "result": None,
"answers": {"company": "Alone", "outcome": "Incinerated at a bonding encounter", "outcome_color": "Red"}}]}),
encoding="utf-8")
manual = TMP / "reddit_comments_manual.yaml"
manual.write_text("reject: [c2]\nfix: {c1: {outcome_color: Blue, branch: 'Valley: passed the Red, the meadow', direction: null}}\n"
"add: [{id: c9, thread: abc, posted: '2026-11-16', "
"answers: {company: Two squadmates, direction: West (toward the scent), outcome: Died before the hour}}]\n",
encoding="utf-8")
rows = runs.reddit_comments("reddit_comments", self.source(public, manual))
self.assertEqual([r["run_id"] for r in rows], ["comment-c1", "comment-c3", "comment-c9"])
c1, c3, c9 = rows
self.assertEqual((c1["source"], c1["outcome_color"], c1["direction"], c1["hour"], c1["enc_Blue"]),
("reddit comment", "Blue", "Unknown", "Valley", "Bonded!"))
self.assertEqual(c1["data_flags"], "read from a Reddit comment")
self.assertEqual(c3["data_flags"], "read from a Reddit comment automatically, not checked by hand")
self.assertEqual((c9["company"], c9["direction"], c9["hour"]), ("2 Squadmates", "West", "DIED before the hour"))
self.assertEqual(validate.problems(rows, runs.DICT), [])
def test_game_rules_flag_impossible_runs(self):
def flags(**answers):
return runs.answers_row("x", "reddit comment", None, answers)["data_flags"]
ok = flags(company="Alone", direction="East (toward the roar)", first_sighting="Orange",
scream="Kept walking (ignored it)", resolution="Wounded cadet: tourniquet",
branch="Tree: climbed down slowly to the two Greens", outcome="Bonded", outcome_color="Brown")
self.assertEqual(ok, "") # Brown after the Greens: a Green's rescue
cases = {
"first sighting Orange not in documented pool for Alone/Center":
dict(company="Alone", direction="Toward the forest centre", first_sighting="Orange", outcome="Died before the hour"),
"Brown at the Shadow (rare) encounter, which has only Black/Blue":
dict(branch="Tree: the shadow (Blue or Black)", outcome="Bonded", outcome_color="Brown"),
"Red at the Passed the Red: meadow encounter":
dict(branch="Valley: passed the Red, the meadow", outcome="Incinerated at a bonding encounter", outcome_color="Red"),
"Green recorded on a Valley run":
dict(hour="Went to the valley centre", outcome="Bonded", outcome_color="Green"),
"does not follow 'Ignore the scream'":
dict(scream="Kept walking (ignored it)", resolution="Bullies: my squadmate's revenge", outcome="Died before the hour"),
"fight answers on a run that did not help":
dict(scream="Kept walking (ignored it)", combat_first="Pause to assess", outcome="Died before the hour"),
"a path after the fight on a run without a fight":
dict(scream="Investigated, and walked away", post_path="West", outcome="Died before the hour"),
"a second fight move without pausing to assess":
dict(scream="Investigated, and helped my squadmate", combat_first="Charge the smaller cadet",
combat_second="Engage the smaller cadet", outcome="Died before the hour"),
"died before the hour, yet reached":
dict(scream="Investigated, and helped my squadmate", resolution="I died in the fight", outcome="Bonded", outcome_color="Red"),
}
for expected, answers in cases.items():
self.assertIn(expected, flags(**answers), expected)
def test_flagged_comment_runs_stay_out_of_the_route_model(self):
sys.path.insert(0, str(ROOT / "planner"))
import build_policy
rows = [{"source": "reddit comment", "data_flags": "read from a Reddit comment automatically, not checked by hand"},
{"source": "reddit comment", "data_flags": "read from a Reddit comment; Green recorded on a Valley run (no Greens in the valley)"},
{"source": "reddit spreadsheet", "data_flags": "multiple wings"},
{"source": "reported", "data_flags": "Green recorded on a Valley run (no Greens in the valley)"}]
kept = build_policy.drop_flagged(rows)
self.assertEqual([r["source"] for r in kept], ["reddit comment", "reddit spreadsheet", "reported"])
def test_delivered_not_extracted(self):
spec = read_yaml("collection/pipeline.yaml")["sources"]["reddit_comments"]
with self.assertRaises(extract.Skip):
extract.EXTRACTORS[spec["extract"]["kind"]](spec["extract"], {})
class Reports(unittest.TestCase):
def issue(self, number, answers, **kw):
form = report_form.spec()