Threshing Atlas: monorepo, route graph, bot importer, daily source checks

- apps/atlas: the web page (valley map, route flow, Dragon Register)
- packages/core: coding rules and statistics (engine.js) and the shared route
  graph (graph.js): scene graph, decision-variable graph, bot-to-form crosswalk,
  Beta credible intervals
- packages/ingest: DragonKind bot importer to anonymised trajectories (no prose,
  no account identifiers); data/bot holds Ken's sample
- research/python: independent reference implementation + update_sources.py
- tests: JS/Python parity checks and graph/importer unit tests
- CI, GitHub Pages deploy, daily check of the community sheets and route PDF
- Email addresses in free-text form answers are redacted; the raw spreadsheet
  is not committed
- MIT code licence, CC BY 4.0 data licence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-01 21:54:54 +00:00
co-authored by Claude Opus 5.5
parent 0052d1a5fd
commit c3d067b88c
48 changed files with 8334 additions and 0 deletions
+94
View File
@@ -0,0 +1,94 @@
# Threshing Atlas
Route probabilities and path finding for the Dragonkind threshing: which choices lead to which
dragon, how often, how sure we can be, and (in progress) the best route to each colour.
Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publishers.
## Layout
```
apps/atlas/ the web page: valley map, route flow, charts, Dragon Register, Reddit post generator
packages/core/ @threshing-atlas/core, no dependencies, runs in Node and the browser:
src/engine.js coding rules and statistics for the community form
src/graph.js the shared route graph: scene graph, decision-variable graph,
bot-to-form crosswalk, Beta credible intervals
packages/ingest/ @threshing-atlas/ingest: DragonKind bot importer (fingerprints and choice text only)
research/python/ independent Python implementation of the coding rules and statistics
tests/ JS/Python parity tests (they fail loudly if the two disagree)
scripts/build-site.js assembles the static site into dist/
scripts/build-graph.js combines every source into data/public/graph.json
data/raw/ source exports (form .xlsx; the register export stays local, git-ignored)
data/public/ what the page loads: form_responses.json, register.json, graph.json
data/sources.json where the community spreadsheets and the PDF are downloaded from
data/derived/ coded runs (runs_clean.csv)
data/bot/ anonymised DragonKind bot runs, added by pull request
docs/ ROADMAP.md, BOT_DATA_PROTOCOL.md, REDDIT_POST.md
```
## Run it locally
Requirements: Node 18+, Python 3.9+ (`pip install -r research/python/requirements.txt`).
```
npm run dev # builds dist/ and serves it at http://localhost:8000
npm test # JS engine + Python reference + parity checks
```
## New data from the community sources
A GitHub Action (`.github/workflows/update-sources.yml`) runs every day. It downloads the Threshing Routes
responses, the Dragon Register and the route PDF, and if anything changed it rebuilds the data, runs the
tests and opens a pull request that says what changed: new responses, new or renamed questions, and for
the PDF the pages and lines that are new. The PDF itself is never committed, only a fingerprint
(`data/derived/pdf_fingerprint.json`). If a link stops working, the Action opens an issue. You can also
run it by hand from the Actions tab, or locally with `npm run check-sources`.
## Bot data
The private DragonKind bot's files are converted locally, and only the anonymised result is committed:
```
DK_ACCOUNT_SALT=... npm run import-bot -- --bot /path/to/DragonKind --tag 2026-11-01
npm run graph
```
This writes `data/bot/scenes.json` and `data/bot/trajectories-<tag>.jsonl`: scene fingerprints, short labels,
choice text and outcomes, with no game prose and no account names. The importer refuses to write anything that
looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
## Updating with new form data by hand
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
`packages/core/src/engine.js` and `research/python/clean.py` need updating.
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
```
cd research/python
python export_raw.py ../../data/raw/form/NEW_EXPORT.xlsx ../../data/public/form_responses.json
python clean.py ../../data/raw/form/NEW_EXPORT.xlsx
cd ../..
npm test
```
## Changing a coding rule
Every rule lives in `codeRow()` in `packages/core/src/engine.js` and `code_row()` in
`research/python/clean.py`. Change both and run `npm test`.
## Statistics in one paragraph
Rates use 95% Wilson score intervals. Each answer is compared with the other answers to the same
question by odds ratio (Woolf CI, +0.5 Haldane correction for zero cells) and Fisher's exact test, with
Benjamini–Hochberg q-values. Because Blue- and Black-chasers report rare sightings far more often than
players who log every run, key outcomes are also modelled with ridge-penalised logistic regression
(λ = 1) that includes data source and chase goal, and route rankings use a Mantel–Haenszel odds ratio
stratified by the same factors. The page's Methods section has the details.
## Licences
Code: MIT (`LICENSE`). Data produced by this project: CC BY 4.0 (`DATA_LICENSE.md`), which also lists
third-party sources that keep their own terms.