Step 2 of the Python-first, declarative core. - core/spec/game.yaml: the game as data: every decision and its options, every roll and the earlier steps its odds may depend on, every bonding encounter, and the flow of a run from the entrance to the dragons (set / keep / drop / go / end / encounter). Also how the form's coded answers map onto the game. - core/atlas/model.py: what each coded run reveals about each step, count tables, empirical-Bayes shrinkage, and the posterior draws (same random numbers as before, reproducible from the seed). - core/atlas/solver.py: one interpreter of game.yaml used three ways (solve, evaluate a fixed policy, export the published tree) instead of the tree written out five times. Vectorised over the 200 posterior draws: the build takes about 2 minutes instead of 4, with no memory flag. - planner/build_policy.py and policy_report.py replace the JS build and report. Checked against the JavaScript model: every number in policy.json agrees to within 4e-12 (last-digit differences in log/cos). Three "still best" shares differ by one draw in 200, at near-ties under the "eventually" objective. POLICY.md is otherwise identical. tests/parity/compare_solver.py keeps checking JS against Python in CI until the site's JS advisor replaces solver.js; tests/core/test_game.py checks game.yaml is complete and consistent. Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
117 lines
6.5 KiB
Markdown
117 lines
6.5 KiB
Markdown
# Threshing Atlas
|
||
|
||
Route probabilities and path finding for the Dragonkind threshing: which choices lead to which
|
||
dragon, how often, how sure we can be, and (in progress) the best route to each colour.
|
||
|
||
Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publishers.
|
||
|
||
## Layout
|
||
|
||
The project has four parts, built on one shared core. Each folder has its own README.
|
||
|
||
```
|
||
core/ the shared model every part uses. spec/ holds the declarative rules (form.yaml: every question
|
||
and answer of the community form); atlas/ is the Python package that reads them (coding.py).
|
||
engine.js, graph.js and solver.js are the JavaScript the site and Ken's bot still use; they move
|
||
to Python + spec files step by step (docs/ROADMAP.md).
|
||
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
|
||
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
|
||
analysis/ making sense of it: stats/ (Python statistics, mirrored in core/engine.js), network/ (builds
|
||
data/public/graph.json), reports/ (generated reports such as POLICY.md)
|
||
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
|
||
and experiments (docs/BOT_INTEGRATION.md)
|
||
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
|
||
data/ raw/ (source exports, not committed), public/ (what the page and the bot load), derived/
|
||
(coded runs, PDF fingerprint), bot/ (anonymised bot runs), sources.json (download links)
|
||
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
|
||
integration, colours, roadmap
|
||
tests/ core/ (unit tests), parity/ (the JS and Python implementations must agree), fixtures/
|
||
```
|
||
|
||
## Run it locally
|
||
|
||
Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
|
||
|
||
```
|
||
npm run dev # builds dist/ and serves it at http://localhost:8000
|
||
npm test # JS engine + Python reference + parity checks
|
||
```
|
||
|
||
## New data from the community sources
|
||
|
||
A GitHub Action (`.github/workflows/update-sources.yml`) runs every day. It downloads the Threshing Routes
|
||
responses, the Dragon Register and the route PDF, and if anything changed it rebuilds the data, runs the
|
||
tests and opens a pull request that says what changed: new responses, new or renamed questions, and for
|
||
the PDF the pages and lines that are new. The PDF itself is never committed, only a fingerprint
|
||
(`data/derived/pdf_fingerprint.json`). If a link stops working, the Action opens an issue. You can also
|
||
run it by hand from the Actions tab, or locally with `npm run check-sources`.
|
||
|
||
## Best routes (the solver)
|
||
|
||
`npm run policy` (Python: `planner/build_policy.py`, from the game in `core/spec/game.yaml`) rebuilds `data/public/policy.json` and the readable `analysis/reports/POLICY.md` (about 2 minutes:
|
||
200 posterior draws for two data samples). For every colour it gives the best decision at every point under two
|
||
objectives: the chance of bonding that colour on this attempt, and the chance the account eventually ends on that
|
||
colour when you die on purpose whenever a run goes wrong. Chance events (which dragon appears, whether the shadow
|
||
shows) may depend on earlier choices; how much is learned from the data. See the top of
|
||
`core/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
|
||
|
||
## Bot data
|
||
|
||
The private DragonKind bot's files are converted locally, and only the anonymised result is committed:
|
||
|
||
```
|
||
DK_ACCOUNT_SALT=... npm run import-bot -- --bot /path/to/DragonKind --tag 2026-11-01
|
||
npm run graph
|
||
```
|
||
|
||
This writes `data/bot/scenes.json` and `data/bot/trajectories-<tag>.jsonl`: scene fingerprints, short labels,
|
||
choice text and outcomes, with no game prose and no account names. The importer refuses to write anything that
|
||
looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
|
||
|
||
## Updating with new form data by hand
|
||
|
||
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
|
||
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
|
||
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
|
||
`core/spec/form.yaml` (and, until the site stops coding rows itself, `core/engine.js`) need updating.
|
||
|
||
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
|
||
|
||
```
|
||
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
|
||
PYTHONPATH=core python -m atlas.coding data/public/form_responses.json data/derived/runs_clean.csv
|
||
npm test
|
||
```
|
||
|
||
## Changing a coding rule
|
||
|
||
The tables (how each question is found, what each answer means, which answers count as missing) are in
|
||
`core/spec/form.yaml`; the logic that combines them is `code_row()` in `core/atlas/coding.py`. Add a case to
|
||
`tests/core/test_coding.py` for every rule you change, then run `npm test`. Until the site switches to
|
||
precomputed statistics, `codeRow()` in `core/engine.js` must be changed the same way (the parity tests check it).
|
||
|
||
## Statistics in one paragraph
|
||
|
||
Rates use 95% Wilson score intervals. Each answer is compared with the other answers to the same
|
||
question by odds ratio (Woolf CI, +0.5 Haldane correction for zero cells) and Fisher's exact test, with
|
||
Benjamini–Hochberg q-values. Because Blue- and Black-chasers report rare sightings far more often than
|
||
players who log every run, key outcomes are also modelled with ridge-penalised logistic regression
|
||
(λ = 1) that includes data source and chase goal, and route rankings use a Mantel–Haenszel odds ratio
|
||
stratified by the same factors. The page's Methods section has the details.
|
||
|
||
## Privacy and licences
|
||
|
||
Code: MIT (`LICENSE`). Data produced by this project: CC BY 4.0 (`DATA_LICENSE.md`), which also lists
|
||
third-party sources that keep their own terms. The rules every part follows (what may be published, what stays
|
||
local, how bot accounts are anonymised) are in `docs/PRIVACY.md`.
|
||
|
||
## Release checks
|
||
|
||
```
|
||
npm test # parity and unit tests
|
||
python site/qa/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
|
||
npm run build && python3 -m http.server 8765 -d dist &
|
||
node site/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
|
||
node site/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
|
||
```
|