Files
threshing-atlas/collection/README.md
T
ekcSchianoandClaude Opus 5.5 98be27fc1f
CI / test (pull_request) Successful in 40s
Reddit scraper PRs get a review summary; the form/Reddit overlap is measured (#23, #24)
- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml:
  on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the
  runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also
  be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it
  to one comment. collection/README.md: "Reviewing Reddit runs".
- analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a
  form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs
  match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the
  page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`.
- Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page)
  stays the one #18 applies.
- data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 16:44:52 +00:00

6.2 KiB

collection

Getting data in, as a strict extract → transform → load pipeline. Every source is declared in pipeline.yaml; the code only carries those declarations out.

python -m collection update          # every source: extract, then transform + load if it changed; then build
python -m collection update --source reported --source register
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx   # a local file instead of the download
python -m collection build           # code every public source file into data/public/runs_clean.csv
python -m collection build --check   # nothing written; exit 1 if anything is out of date (CI runs this)

The stages

Stage Does Never does
extract (etl/extract.py) Fetches the source as it is now into data/raw/<source>/current.<ext>. The version it replaces becomes previous.<ext>; nothing older is kept. data/manifest.json records the current raw file's sha256 and the day it last changed. Change the data.
transform (etl/transform/) Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. Write files.
validate (etl/validate.py) Checks the combined runs against data/dictionary.yaml (types, every value listed, unique ids). A failure stops the load and is reported. Fix anything silently.
load (etl/load.py) The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. Publish anything raw.

build then codes every public source file into one run per row (etl/runs.py, one coder per source) and writes data/public/runs_clean.csv, plus the files generated from the specs: the "Report a run" issue form, the bot crosswalk inside core/graph.js, and data/croissant.json. Every input to build is committed, so CI rebuilds it exactly.

The sources

source in runs_clean.csv Pipeline source Extract Public source file
reddit spreadsheet reddit_spreadsheet: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads the sheet's .xlsx export data/public/form_responses.json (emails redacted), coded by core/spec/form.yaml
reddit thread reddit_thread: dragonkind.com result pages posted in the r/fourthwing megathreads the threads, through the official Reddit API (needs REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET) manual/reddit_results.yaml, checked by hand; the extract lists result links not in it yet
reddit comment reddit_comments: runs riders describe in the megathread comments not extracted here: the threshing-reddit scraper reads the threads every 6 hours and opens (or updates) a pull request that changes the public file data/public/reddit_comments.json: each comment read into the "Report a run" answers, no usernames or text; corrections by hand in manual/reddit_comments.yaml
reported reported: the Atlas's "Report a run" form (a GitHub issue form, specs/report_form.yaml) issues labelled run report, through the GitHub API data/public/reports.json (answers only, no usernames)
bot bot: Ken's DragonKind bot a local copy of the bot's private repo (DRAGONKIND_BOT_DIR) data/bot/scenes.json, data/bot/trajectories-<date>.jsonl (no prose, accounts hashed with DK_ACCOUNT_SALT; specs/bot.yaml)
reddit pdf reddit_pdf: the community route PDF the PDF data/derived/pdf_fingerprint.json only: hashes, never the text
— register: the Unofficial Dragon Register the sheet's .xlsx export data/public/register.json (rider names and photos dropped)

The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to runs_clean.csv; its fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that led there, so it adds no rows either.

Reviewing Reddit runs

Ken's scraper opens (or updates) a pull request from data/reddit-comments that changes data/public/reddit_comments.json. The Review Reddit runs workflow (.gitea/workflows/reddit-review.yml) posts a summary on it, refreshed on every push (python -m collection review-reddit --base <main's file> locally):

  • runs added, changed and gone since main;
  • runs that break a game rule (core/spec/form.yaml impossible_flags): left out of the route model and the statistics until corrected. Open the comment and decide;
  • other flags on the new runs (kept);
  • runs that may also be in the community form: the same answers within a day, with how many chance alone gives (analysis/overlap/overlap.py, data/public/overlap.json). Measured only, never removed;
  • runs whose result page is already a reddit thread row (counted once automatically).

Corrections go in manual/reddit_comments.yaml, applied at build time, so they never wait for the threads: reject: [id] (not a run), fix: {id: {question: answer}} (misread; null removes an answer) and add: [{id, thread, posted, answers}] (missed). A corrected run loses the "not checked by hand" note. No usernames: the comment id is enough to find the comment again. Then merge, once CI passes.

Raw files and privacy

data/raw/ is git-ignored and never published. It holds the current raw file of each source and the one before it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted with the RAW_ARCHIVE_KEY secret (see docs/REPO_SETTINGS.md). Privacy rules sit with each source in pipeline.yaml: emails redacted, register names and photos dropped, the bot's fields refused if they look like prose or account ids, GitHub and Reddit usernames never kept.

Adding a source

  1. Add a block to pipeline.yaml (extract kind, transform, privacy, load target, and run_source / runs if it has runs).
  2. If it needs a new transform or coder, add one module to etl/transform/ and one function to etl/runs.py.
  3. Add its source value to data/dictionary.yaml, then python -m collection build and npm test.