site built from 44ccd7f7

This commit is contained in:
Atlas data bot
2026-11-19 21:41:37 +00:00
commit e6332f7c0d
29 changed files with 22125 additions and 0 deletions
+124
View File
@@ -0,0 +1,124 @@
# Threshing Atlas
Route probabilities and path finding for the Dragonkind threshing: which choices lead to which
dragon, how often, how sure we can be, and (in progress) the best route to each colour.
Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publishers.
## Layout
The project has four parts, built on one shared core. Each folder has its own README.
```
core/ the shared model every part uses. spec/ holds the declarative rules (form.yaml: the community
form; game.yaml: the game); atlas/ is the Python package that reads them (coding, route model).
solver.js (the route advisor) and graph.js (the bot's text) are the only JavaScript, because
they run in the browser and in Ken's bot.
collection/ getting data in, as extract -> transform -> load: pipeline.yaml declares every source (the
community spreadsheet, Reddit threads, "Report a run", the bot, the route PDF, the register);
`python -m collection` runs it and builds data/public/runs_clean.csv from all of them.
analysis/ making sense of it: stats/ (measures.yaml and site_stats.py: every number on the page), network/ (builds
data/public/graph.json), reports/ (generated reports such as POLICY.md)
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
and experiments (docs/BOT_INTEGRATION.md)
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
data/ public/ (everything published, including runs_clean.csv), dictionary.yaml (every column),
croissant.json (dataset metadata), manifest.json (current raw versions), bot/ (anonymised bot
runs), derived/ (PDF fingerprint), raw/ (current + previous raw files; never committed)
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
integration, colours, roadmap
tests/ core/ (rule, statistics, route model and advisor tests), collection/ (the pipeline), parity/
(tools to compare two policies), fixtures/
```
## Run it locally
Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
```
npm run dev # builds dist/ and serves it at http://localhost:8000
npm test # pipeline, coding rules, statistics, route model, game spec, advisor and bot-text tests
```
## New data from the community sources
A scheduled workflow (`.gitea/workflows/update-sources.yml` on the Atlas's Gitea; `.github/workflows/` holds the
GitHub version) runs the collection pipeline every day
(`python -m collection update`, see `collection/README.md`). It checks every source in `collection/pipeline.yaml`,
and if anything changed it rebuilds the runs and the analysis, runs the tests and opens a pull request that says
what changed: new responses, new or renamed questions, new run reports, new result links on Reddit, and for the PDF
the pages and lines that are new. The PDF itself is never committed, only a fingerprint. If a source cannot be
reached, or new answers do not match `data/dictionary.yaml`, the Action opens an issue instead. Locally:
`npm run collect`.
## Best routes (the solver)
`npm run policy` (Python: `planner/build_policy.py`, from the game in `core/spec/game.yaml`) rebuilds `data/public/policy.json` and the readable `analysis/reports/POLICY.md` (about 2 minutes:
200 posterior draws for two data samples). For every colour it gives the best decision at every point under two
objectives: the chance of bonding that colour on this attempt, and the chance the account eventually ends on that
colour when you die on purpose whenever a run goes wrong. Chance events (which dragon appears, whether the shadow
shows) may depend on earlier choices; how much is learned from the data. See the top of
`core/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
## Bot data
The private DragonKind bot's files are converted locally, and only the anonymised result is committed:
```
DRAGONKIND_BOT_DIR=/path/to/DragonKind DK_ACCOUNT_SALT=... python -m collection update --source bot
npm run graph
```
This writes `data/bot/scenes.json` and `data/bot/trajectories-<date>.jsonl`: scene fingerprints, short labels
(`collection/specs/bot.yaml`), choice text and outcomes, with no game prose and no account names, and adds the
bot's attempts to `runs_clean.csv` (source `bot`). The importer refuses to write anything that looks like prose.
See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
## Running everything locally
`python -m collection refresh` does what the daily workflow does (`python -m collection rebuild` alone redoes everything built from the runs): checks every source, rebuilds the runs,
the graph, the route model and the statistics. Add `--bot <folder or zip>` to import bot runs, and
`python -m collection export-bot-kit` zips the files the bot uses (see `docs/BOT_INTEGRATION.md`).
## Updating with new form data by hand
Download the responses spreadsheet (Google Sheets → File → Download → Microsoft Excel), then:
```
python -m collection update --source reddit_spreadsheet --file reddit_spreadsheet=path/to/export.xlsx
npm test
```
Questions are matched by wording; a renamed question is listed in the report, and then `core/spec/form.yaml`
needs updating.
## Changing a coding rule
The tables (how each question is found, what each answer means, which answers count as missing) are in
`core/spec/form.yaml`; the logic that combines them is `code_row()` in `core/atlas/coding.py`. Add a case to
`tests/core/test_coding.py` for every rule you change, then run `npm test`.
## Statistics in one paragraph
Rates use 95% Wilson score intervals. Each answer is compared with the other answers to the same
question by odds ratio (Woolf CI, +0.5 Haldane correction for zero cells) and Fisher's exact test, with
Benjamini–Hochberg q-values. Because Blue- and Black-chasers report rare sightings far more often than
players who log every run, key outcomes are also modelled with ridge-penalised logistic regression
(λ = 1) that includes data source and chase goal, and route rankings use a Mantel–Haenszel odds ratio
stratified by the same factors. The page's Methods section has the details.
## Privacy and licences
Code: MIT (`LICENSE`). Data produced by this project: CC BY 4.0 (`DATA_LICENSE.md`), which also lists
third-party sources that keep their own terms. The rules every part follows (what may be published, what stays
local, how bot accounts are anonymised) are in `docs/PRIVACY.md`.
## Release checks
```
npm test # all tests
python site/qa/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
npm run build && python3 -m http.server 8765 -d dist &
node site/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
node site/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
```