Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+50 -13
View File
@@ -1,19 +1,56 @@
# collection
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
under `data/`.
Getting data in, as a strict **extract → transform → load** pipeline. Every source is declared in
[`pipeline.yaml`](pipeline.yaml); the code only carries those declarations out.
| Source | Code | Output |
```
python -m collection update # every source: extract, then transform + load if it changed; then build
python -m collection update --source reported --source register
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
python -m collection build # code every public source file into data/public/runs_clean.csv
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
```
## The stages
| Stage | Does | Never does |
|---|---|---|
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/atlas/coding.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
| Reddit megathreads | not collected yet | — |
| **extract** (`etl/extract.py`) | Fetches the source as it is now into `data/raw/<source>/current.<ext>`. The version it replaces becomes `previous.<ext>`; nothing older is kept. `data/manifest.json` records the current raw file's sha256 and the day it last changed. | Change the data. |
| **transform** (`etl/transform/`) | Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
| **validate** (`etl/validate.py`) | Checks the combined runs against `data/dictionary.yaml` (types, every value listed, unique ids). A failure stops the load and is reported. | Fix anything silently. |
| **load** (`etl/load.py`) | The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
merges itself when CI passes. A broken link opens an issue.
`build` then codes every public source file into one run per row (`etl/runs.py`, one coder per source) and writes
`data/public/runs_clean.csv`, plus the files generated from the specs: the "Report a run" issue form, the bot
crosswalk inside `core/graph.js`, and `data/croissant.json`. Every input to `build` is committed, so CI rebuilds
it exactly.
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.
## The sources
| `source` in runs_clean.csv | Pipeline source | Extract | Public source file |
|---|---|---|---|
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
| — | `register`: the Unofficial Dragon Register | the sheet's .xlsx export | `data/public/register.json` (rider names and photos dropped) |
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to `runs_clean.csv`; its
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
led there, so it adds no rows either.
## Raw files and privacy
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
with the `RAW_ARCHIVE_KEY` secret (see `docs/REPO_SETTINGS.md`). Privacy rules sit with each source in
`pipeline.yaml`: emails redacted, register names and photos dropped, the bot's fields refused if they look like
prose or account ids, GitHub and Reddit usernames never kept.
## Adding a source
1. Add a block to `pipeline.yaml` (extract kind, transform, privacy, load target, and `run_source` / `runs` if it
has runs).
2. If it needs a new transform or coder, add one module to `etl/transform/` and one function to `etl/runs.py`.
3. Add its `source` value to `data/dictionary.yaml`, then `python -m collection build` and `npm test`.