Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+50
-13
@@ -1,19 +1,56 @@
|
||||
# collection
|
||||
|
||||
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
|
||||
under `data/`.
|
||||
Getting data in, as a strict **extract → transform → load** pipeline. Every source is declared in
|
||||
[`pipeline.yaml`](pipeline.yaml); the code only carries those declarations out.
|
||||
|
||||
| Source | Code | Output |
|
||||
```
|
||||
python -m collection update # every source: extract, then transform + load if it changed; then build
|
||||
python -m collection update --source reported --source register
|
||||
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
|
||||
python -m collection build # code every public source file into data/public/runs_clean.csv
|
||||
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
|
||||
```
|
||||
|
||||
## The stages
|
||||
|
||||
| Stage | Does | Never does |
|
||||
|---|---|---|
|
||||
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/atlas/coding.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
|
||||
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
|
||||
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
|
||||
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
|
||||
| Reddit megathreads | not collected yet | — |
|
||||
| **extract** (`etl/extract.py`) | Fetches the source as it is now into `data/raw/<source>/current.<ext>`. The version it replaces becomes `previous.<ext>`; nothing older is kept. `data/manifest.json` records the current raw file's sha256 and the day it last changed. | Change the data. |
|
||||
| **transform** (`etl/transform/`) | Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
|
||||
| **validate** (`etl/validate.py`) | Checks the combined runs against `data/dictionary.yaml` (types, every value listed, unique ids). A failure stops the load and is reported. | Fix anything silently. |
|
||||
| **load** (`etl/load.py`) | The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
|
||||
|
||||
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
|
||||
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
|
||||
merges itself when CI passes. A broken link opens an issue.
|
||||
`build` then codes every public source file into one run per row (`etl/runs.py`, one coder per source) and writes
|
||||
`data/public/runs_clean.csv`, plus the files generated from the specs: the "Report a run" issue form, the bot
|
||||
crosswalk inside `core/graph.js`, and `data/croissant.json`. Every input to `build` is committed, so CI rebuilds
|
||||
it exactly.
|
||||
|
||||
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
|
||||
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.
|
||||
## The sources
|
||||
|
||||
| `source` in runs_clean.csv | Pipeline source | Extract | Public source file |
|
||||
|---|---|---|---|
|
||||
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
|
||||
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
|
||||
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
|
||||
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
|
||||
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
|
||||
| — | `register`: the Unofficial Dragon Register | the sheet's .xlsx export | `data/public/register.json` (rider names and photos dropped) |
|
||||
|
||||
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to `runs_clean.csv`; its
|
||||
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
|
||||
led there, so it adds no rows either.
|
||||
|
||||
## Raw files and privacy
|
||||
|
||||
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
|
||||
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
|
||||
with the `RAW_ARCHIVE_KEY` secret (see `docs/REPO_SETTINGS.md`). Privacy rules sit with each source in
|
||||
`pipeline.yaml`: emails redacted, register names and photos dropped, the bot's fields refused if they look like
|
||||
prose or account ids, GitHub and Reddit usernames never kept.
|
||||
|
||||
## Adding a source
|
||||
|
||||
1. Add a block to `pipeline.yaml` (extract kind, transform, privacy, load target, and `run_source` / `runs` if it
|
||||
has runs).
|
||||
2. If it needs a new transform or coder, add one module to `etl/transform/` and one function to `etl/runs.py`.
|
||||
3. Add its `source` value to `data/dictionary.yaml`, then `python -m collection build` and `npm test`.
|
||||
|
||||
Reference in New Issue
Block a user