Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+26 -24
View File
@@ -14,18 +14,21 @@ core/ the shared model every part uses. spec/ holds the declarative ru
form; game.yaml: the game); atlas/ is the Python package that reads them (coding, route model).
solver.js (the route advisor) and graph.js (the bot's text) are the only JavaScript, because
they run in the browser and in Ken's bot.
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
collection/ getting data in, as extract -> transform -> load: pipeline.yaml declares every source (the
community spreadsheet, Reddit threads, "Report a run", the bot, the route PDF, the register);
`python -m collection` runs it and builds data/public/runs_clean.csv from all of them.
analysis/ making sense of it: stats/ (measures.yaml and site_stats.py: every number on the page), network/ (builds
data/public/graph.json), reports/ (generated reports such as POLICY.md)
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
and experiments (docs/BOT_INTEGRATION.md)
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
data/ raw/ (source exports, not committed), public/ (what the page and the bot load), derived/
(coded runs, PDF fingerprint), bot/ (anonymised bot runs), sources.json (download links)
data/ public/ (everything published, including runs_clean.csv), dictionary.yaml (every column),
croissant.json (dataset metadata), manifest.json (current raw versions), bot/ (anonymised bot
runs), derived/ (PDF fingerprint), raw/ (current + previous raw files; never committed)
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
integration, colours, roadmap
tests/ core/ (rule, statistics, route model and advisor tests), parity/ (tools to compare two policies), fixtures/
tests/ core/ (rule, statistics, route model and advisor tests), collection/ (the pipeline), parity/
(tools to compare two policies), fixtures/
```
## Run it locally
@@ -34,17 +37,18 @@ Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
```
npm run dev # builds dist/ and serves it at http://localhost:8000
npm test # coding rules, statistics, route model, game spec, advisor and bot-text tests
npm test # pipeline, coding rules, statistics, route model, game spec, advisor and bot-text tests
```
## New data from the community sources
A GitHub Action (`.github/workflows/update-sources.yml`) runs every day. It downloads the Threshing Routes
responses, the Dragon Register and the route PDF, and if anything changed it rebuilds the data, runs the
tests and opens a pull request that says what changed: new responses, new or renamed questions, and for
the PDF the pages and lines that are new. The PDF itself is never committed, only a fingerprint
(`data/derived/pdf_fingerprint.json`). If a link stops working, the Action opens an issue. You can also
run it by hand from the Actions tab, or locally with `npm run check-sources`.
A GitHub Action (`.github/workflows/update-sources.yml`) runs the collection pipeline every day
(`python -m collection update`, see `collection/README.md`). It checks every source in `collection/pipeline.yaml`,
and if anything changed it rebuilds the runs and the analysis, runs the tests and opens a pull request that says
what changed: new responses, new or renamed questions, new run reports, new result links on Reddit, and for the PDF
the pages and lines that are new. The PDF itself is never committed, only a fingerprint. If a source cannot be
reached, or new answers do not match `data/dictionary.yaml`, the Action opens an issue instead. Locally:
`npm run collect`.
## Best routes (the solver)
@@ -60,29 +64,27 @@ shows) may depend on earlier choices; how much is learned from the data. See the
The private DragonKind bot's files are converted locally, and only the anonymised result is committed:
```
DK_ACCOUNT_SALT=... npm run import-bot -- --bot /path/to/DragonKind --tag 2026-11-01
DRAGONKIND_BOT_DIR=/path/to/DragonKind DK_ACCOUNT_SALT=... python -m collection update --source bot
npm run graph
```
This writes `data/bot/scenes.json` and `data/bot/trajectories-<tag>.jsonl`: scene fingerprints, short labels,
choice text and outcomes, with no game prose and no account names. The importer refuses to write anything that
looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
This writes `data/bot/scenes.json` and `data/bot/trajectories-<date>.jsonl`: scene fingerprints, short labels
(`collection/specs/bot.yaml`), choice text and outcomes, with no game prose and no account names, and adds the
bot's attempts to `runs_clean.csv` (source `bot`). The importer refuses to write anything that looks like prose.
See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
## Updating with new form data by hand
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
`core/spec/form.yaml` needs updating.
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
Download the responses spreadsheet (Google Sheets → File → Download → Microsoft Excel), then:
```
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
PYTHONPATH=core python -m atlas.coding data/public/form_responses.json data/derived/runs_clean.csv
python -m collection update --source reddit_spreadsheet --file reddit_spreadsheet=path/to/export.xlsx
npm test
```
Questions are matched by wording; a renamed question is listed in the report, and then `core/spec/form.yaml`
needs updating.
## Changing a coding rule
The tables (how each question is found, what each answer means, which answers count as missing) are in