Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
@@ -14,18 +14,21 @@ core/ the shared model every part uses. spec/ holds the declarative ru
|
||||
form; game.yaml: the game); atlas/ is the Python package that reads them (coding, route model).
|
||||
solver.js (the route advisor) and graph.js (the bot's text) are the only JavaScript, because
|
||||
they run in the browser and in Ken's bot.
|
||||
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
|
||||
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
|
||||
collection/ getting data in, as extract -> transform -> load: pipeline.yaml declares every source (the
|
||||
community spreadsheet, Reddit threads, "Report a run", the bot, the route PDF, the register);
|
||||
`python -m collection` runs it and builds data/public/runs_clean.csv from all of them.
|
||||
analysis/ making sense of it: stats/ (measures.yaml and site_stats.py: every number on the page), network/ (builds
|
||||
data/public/graph.json), reports/ (generated reports such as POLICY.md)
|
||||
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
|
||||
and experiments (docs/BOT_INTEGRATION.md)
|
||||
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
|
||||
data/ raw/ (source exports, not committed), public/ (what the page and the bot load), derived/
|
||||
(coded runs, PDF fingerprint), bot/ (anonymised bot runs), sources.json (download links)
|
||||
data/ public/ (everything published, including runs_clean.csv), dictionary.yaml (every column),
|
||||
croissant.json (dataset metadata), manifest.json (current raw versions), bot/ (anonymised bot
|
||||
runs), derived/ (PDF fingerprint), raw/ (current + previous raw files; never committed)
|
||||
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
|
||||
integration, colours, roadmap
|
||||
tests/ core/ (rule, statistics, route model and advisor tests), parity/ (tools to compare two policies), fixtures/
|
||||
tests/ core/ (rule, statistics, route model and advisor tests), collection/ (the pipeline), parity/
|
||||
(tools to compare two policies), fixtures/
|
||||
```
|
||||
|
||||
## Run it locally
|
||||
@@ -34,17 +37,18 @@ Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
|
||||
|
||||
```
|
||||
npm run dev # builds dist/ and serves it at http://localhost:8000
|
||||
npm test # coding rules, statistics, route model, game spec, advisor and bot-text tests
|
||||
npm test # pipeline, coding rules, statistics, route model, game spec, advisor and bot-text tests
|
||||
```
|
||||
|
||||
## New data from the community sources
|
||||
|
||||
A GitHub Action (`.github/workflows/update-sources.yml`) runs every day. It downloads the Threshing Routes
|
||||
responses, the Dragon Register and the route PDF, and if anything changed it rebuilds the data, runs the
|
||||
tests and opens a pull request that says what changed: new responses, new or renamed questions, and for
|
||||
the PDF the pages and lines that are new. The PDF itself is never committed, only a fingerprint
|
||||
(`data/derived/pdf_fingerprint.json`). If a link stops working, the Action opens an issue. You can also
|
||||
run it by hand from the Actions tab, or locally with `npm run check-sources`.
|
||||
A GitHub Action (`.github/workflows/update-sources.yml`) runs the collection pipeline every day
|
||||
(`python -m collection update`, see `collection/README.md`). It checks every source in `collection/pipeline.yaml`,
|
||||
and if anything changed it rebuilds the runs and the analysis, runs the tests and opens a pull request that says
|
||||
what changed: new responses, new or renamed questions, new run reports, new result links on Reddit, and for the PDF
|
||||
the pages and lines that are new. The PDF itself is never committed, only a fingerprint. If a source cannot be
|
||||
reached, or new answers do not match `data/dictionary.yaml`, the Action opens an issue instead. Locally:
|
||||
`npm run collect`.
|
||||
|
||||
## Best routes (the solver)
|
||||
|
||||
@@ -60,29 +64,27 @@ shows) may depend on earlier choices; how much is learned from the data. See the
|
||||
The private DragonKind bot's files are converted locally, and only the anonymised result is committed:
|
||||
|
||||
```
|
||||
DK_ACCOUNT_SALT=... npm run import-bot -- --bot /path/to/DragonKind --tag 2026-11-01
|
||||
DRAGONKIND_BOT_DIR=/path/to/DragonKind DK_ACCOUNT_SALT=... python -m collection update --source bot
|
||||
npm run graph
|
||||
```
|
||||
|
||||
This writes `data/bot/scenes.json` and `data/bot/trajectories-<tag>.jsonl`: scene fingerprints, short labels,
|
||||
choice text and outcomes, with no game prose and no account names. The importer refuses to write anything that
|
||||
looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
|
||||
This writes `data/bot/scenes.json` and `data/bot/trajectories-<date>.jsonl`: scene fingerprints, short labels
|
||||
(`collection/specs/bot.yaml`), choice text and outcomes, with no game prose and no account names, and adds the
|
||||
bot's attempts to `runs_clean.csv` (source `bot`). The importer refuses to write anything that looks like prose.
|
||||
See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
|
||||
|
||||
## Updating with new form data by hand
|
||||
|
||||
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
|
||||
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
|
||||
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
|
||||
`core/spec/form.yaml` needs updating.
|
||||
|
||||
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
|
||||
Download the responses spreadsheet (Google Sheets → File → Download → Microsoft Excel), then:
|
||||
|
||||
```
|
||||
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
|
||||
PYTHONPATH=core python -m atlas.coding data/public/form_responses.json data/derived/runs_clean.csv
|
||||
python -m collection update --source reddit_spreadsheet --file reddit_spreadsheet=path/to/export.xlsx
|
||||
npm test
|
||||
```
|
||||
|
||||
Questions are matched by wording; a renamed question is listed in the report, and then `core/spec/form.yaml`
|
||||
needs updating.
|
||||
|
||||
## Changing a coding rule
|
||||
|
||||
The tables (how each question is found, what each answer means, which answers count as missing) are in
|
||||
|
||||
Reference in New Issue
Block a user