- core/spec/form.yaml impossible_flags: the flags that mean a run cannot have happened as recorded (a dragon an encounter does not have, a Green or the Black in the valley, two bonds, two burns, a cadet path that does not follow, an early death that still reached the hour...). atlas.runs.impossible() reads it. - The route model and the site statistics (and the riders' paths) leave those runs out, from every source; every other run is used, however partial. The runs stay in runs_clean.csv, flagged. 80 runs: 75 form, 5 Reddit. Route-model odds move by at most 0.2 points. - collection/etl/runs.py: the community form gets the same game-rule checks as "Report a run" and the Reddit comments (10 form rows newly flagged); new rule: the Black in the valley. - The first-sighting pool stays a flag only: it is what riders have seen, not a rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
collection
Getting data in, as a strict extract → transform → load pipeline. Every source is declared in
pipeline.yaml; the code only carries those declarations out.
python -m collection update # every source: extract, then transform + load if it changed; then build
python -m collection update --source reported --source register
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
python -m collection build # code every public source file into data/public/runs_clean.csv
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
The stages
| Stage | Does | Never does |
|---|---|---|
extract (etl/extract.py) |
Fetches the source as it is now into data/raw/<source>/current.<ext>. The version it replaces becomes previous.<ext>; nothing older is kept. data/manifest.json records the current raw file's sha256 and the day it last changed. |
Change the data. |
transform (etl/transform/) |
Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
validate (etl/validate.py) |
Checks the combined runs against data/dictionary.yaml (types, every value listed, unique ids). A failure stops the load and is reported. |
Fix anything silently. |
load (etl/load.py) |
The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
build then codes every public source file into one run per row (etl/runs.py, one coder per source) and writes
data/public/runs_clean.csv, plus the files generated from the specs: the "Report a run" issue form, the bot
crosswalk inside core/graph.js, and data/croissant.json. Every input to build is committed, so CI rebuilds
it exactly.
The sources
source in runs_clean.csv |
Pipeline source | Extract | Public source file |
|---|---|---|---|
| reddit spreadsheet | reddit_spreadsheet: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads |
the sheet's .xlsx export | data/public/form_responses.json (emails redacted), coded by core/spec/form.yaml |
| reddit thread | reddit_thread: dragonkind.com result pages posted in the r/fourthwing megathreads |
the threads, through the official Reddit API (needs REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET) |
manual/reddit_results.yaml, checked by hand; the extract lists result links not in it yet |
| reddit comment | reddit_comments: runs riders describe in the megathread comments |
not extracted here: the threshing-reddit scraper reads the threads every 6 hours and opens (or updates) a pull request that changes the public file | data/public/reddit_comments.json: each comment read into the "Report a run" answers, no usernames or text; corrections by hand in manual/reddit_comments.yaml |
| reported | reported: the Atlas's "Report a run" form (a GitHub issue form, specs/report_form.yaml) |
issues labelled run report, through the GitHub API |
data/public/reports.json (answers only, no usernames) |
| bot | bot: Ken's DragonKind bot |
a local copy of the bot's private repo (DRAGONKIND_BOT_DIR) |
data/bot/scenes.json, data/bot/trajectories-<date>.jsonl (no prose, accounts hashed with DK_ACCOUNT_SALT; specs/bot.yaml) |
| reddit pdf | reddit_pdf: the community route PDF |
the PDF | data/derived/pdf_fingerprint.json only: hashes, never the text |
| — | register: the Unofficial Dragon Register |
the sheet's .xlsx export | data/public/register.json (rider names and photos dropped) |
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to runs_clean.csv; its
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
led there, so it adds no rows either.
Raw files and privacy
data/raw/ is git-ignored and never published. It holds the current raw file of each source and the one before
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
with the RAW_ARCHIVE_KEY secret (see docs/REPO_SETTINGS.md). Privacy rules sit with each source in
pipeline.yaml: emails redacted, register names and photos dropped, the bot's fields refused if they look like
prose or account ids, GitHub and Reddit usernames never kept.
Adding a source
- Add a block to
pipeline.yaml(extract kind, transform, privacy, load target, andrun_source/runsif it has runs). - If it needs a new transform or coder, add one module to
etl/transform/and one function toetl/runs.py. - Add its
sourcevalue todata/dictionary.yaml, thenpython -m collection buildandnpm test.