Files
ekcSchianoandClaude Opus 5.5 98be27fc1f
CI / test (pull_request) Successful in 40s
Reddit scraper PRs get a review summary; the form/Reddit overlap is measured (#23, #24)
- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml:
  on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the
  runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also
  be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it
  to one comment. collection/README.md: "Reviewing Reddit runs".
- analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a
  form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs
  match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the
  page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`.
- Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page)
  stays the one #18 applies.
- data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 16:44:52 +00:00

77 lines
6.2 KiB
Markdown

# collection
Getting data in, as a strict **extract → transform → load** pipeline. Every source is declared in
[`pipeline.yaml`](pipeline.yaml); the code only carries those declarations out.
```
python -m collection update # every source: extract, then transform + load if it changed; then build
python -m collection update --source reported --source register
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
python -m collection build # code every public source file into data/public/runs_clean.csv
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
```
## The stages
| Stage | Does | Never does |
|---|---|---|
| **extract** (`etl/extract.py`) | Fetches the source as it is now into `data/raw/<source>/current.<ext>`. The version it replaces becomes `previous.<ext>`; nothing older is kept. `data/manifest.json` records the current raw file's sha256 and the day it last changed. | Change the data. |
| **transform** (`etl/transform/`) | Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
| **validate** (`etl/validate.py`) | Checks the combined runs against `data/dictionary.yaml` (types, every value listed, unique ids). A failure stops the load and is reported. | Fix anything silently. |
| **load** (`etl/load.py`) | The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
`build` then codes every public source file into one run per row (`etl/runs.py`, one coder per source) and writes
`data/public/runs_clean.csv`, plus the files generated from the specs: the "Report a run" issue form, the bot
crosswalk inside `core/graph.js`, and `data/croissant.json`. Every input to `build` is committed, so CI rebuilds
it exactly.
## The sources
| `source` in runs_clean.csv | Pipeline source | Extract | Public source file |
|---|---|---|---|
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
| reddit comment | `reddit_comments`: runs riders describe in the megathread comments | not extracted here: the [threshing-reddit](https://git.kenschiano.net/DragonKind/threshing-reddit) scraper reads the threads every 6 hours and opens (or updates) a pull request that changes the public file | `data/public/reddit_comments.json`: each comment read into the "Report a run" answers, no usernames or text; corrections by hand in `manual/reddit_comments.yaml` |
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
| — | `register`: the Unofficial Dragon Register | the sheet's .xlsx export | `data/public/register.json` (rider names and photos dropped) |
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to `runs_clean.csv`; its
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
led there, so it adds no rows either.
## Reviewing Reddit runs
Ken's scraper opens (or updates) a pull request from `data/reddit-comments` that changes
`data/public/reddit_comments.json`. The **Review Reddit runs** workflow (`.gitea/workflows/reddit-review.yml`) posts a
summary on it, refreshed on every push (`python -m collection review-reddit --base <main's file>` locally):
- runs added, changed and gone since main;
- **runs that break a game rule** (`core/spec/form.yaml` `impossible_flags`): left out of the route model and the
statistics until corrected. Open the comment and decide;
- other flags on the new runs (kept);
- runs that may also be in the community form: the same answers within a day, with how many chance alone gives
(`analysis/overlap/overlap.py`, `data/public/overlap.json`). Measured only, never removed;
- runs whose result page is already a `reddit thread` row (counted once automatically).
Corrections go in `manual/reddit_comments.yaml`, applied at build time, so they never wait for the threads:
`reject: [id]` (not a run), `fix: {id: {question: answer}}` (misread; `null` removes an answer) and
`add: [{id, thread, posted, answers}]` (missed). A corrected run loses the "not checked by hand" note. No usernames:
the comment id is enough to find the comment again. Then merge, once CI passes.
## Raw files and privacy
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
with the `RAW_ARCHIVE_KEY` secret (see `docs/REPO_SETTINGS.md`). Privacy rules sit with each source in
`pipeline.yaml`: emails redacted, register names and photos dropped, the bot's fields refused if they look like
prose or account ids, GitHub and Reddit usernames never kept.
## Adding a source
1. Add a block to `pipeline.yaml` (extract kind, transform, privacy, load target, and `run_source` / `runs` if it
has runs).
2. If it needs a new transform or coder, add one module to `etl/transform/` and one function to `etl/runs.py`.
3. Add its `source` value to `data/dictionary.yaml`, then `python -m collection build` and `npm test`.