Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+18 -11
View File
@@ -1,18 +1,25 @@
# Data sources
| Source | Where it lives here | How to update |
|---|---|---|
| Community "Threshing Routes" Google Form (u/ClumsyBadger) | `raw/form/*.xlsx` → `public/form_responses.json` | Download the sheet as .xlsx, then `python collection/sources/export_raw.py` |
| Community route PDF (u/ClumsyBadger) | not redistributed; linked from the Reddit megathreads | A scene crosswalk will be kept in `derived/` |
| Unofficial Dragon Register | `raw/register/` (git-ignored, has rider names) → `public/register.json` | `python collection/sources/export_register.py` |
| Reddit megathreads (r/fourthwing, Dragonkind bonding Parts 3 and 4) | qualitative; routes cited in the valley map | manual |
| DragonKind bot exports | `bot/runs-YYYY-MM-DD.jsonl` | by pull request, see `docs/BOT_DATA_PROTOCOL.md` |
| Discord | not yet | planned, with admin consent and anonymised usernames |
Every source is declared in `collection/pipeline.yaml` and collected by `python -m collection update`
(see `collection/README.md`). Every run from every source is in `public/runs_clean.csv`, with a `source` column.
| Source (`source` column) | Where it lives here |
|---|---|
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
| Unofficial Dragon Register | `public/register.json` (rider names and photos dropped); no runs |
| Discord | not yet: planned, with admin consent and anonymised usernames |
Raw files (`raw/`, git-ignored) are never published: the current version of each source and the one before it.
`manifest.json` records the current raw file's checksum and when it last changed.
## What each column means
`data/dictionary.yaml` describes every published file and every column of `data/derived/runs_clean.csv`: its type,
`data/dictionary.yaml` describes every published file and every column of `data/public/runs_clean.csv`: its type,
what it means and every value it can take. `data/croissant.json` says the same in the
[Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so
dataset tools can load and cite the data. After changing the data or the dictionary, run `npm run metadata`; the
tests fail if a column or value is not described or the metadata is out of date.
dataset tools can load and cite the data. `python -m collection build` rewrites it; the tests fail if a column or
value is not described or the metadata is out of date.