Refactor: lay the repo out by section (core, collection, analysis, planner, site) (#8)

Moves only; no behaviour changes. Every file now lives under the part of the project it
belongs to, each section has a README, and docs/PRIVACY.md collects the rules all parts follow.

  core/        engine.js, graph.js, solver.js, python/clean.py
  collection/  sources/ (form, register, PDF check, daily update), bot/ (importer)
  analysis/    stats/ (Python statistics), network/ (build-graph.js), reports/ (POLICY.md)
  planner/     build-policy.js, policy-report.js
  site/        atlas/ (the page), build.js, qa/
  tests/       core/ (unit), parity/ (JS vs Python), fixtures/

Checked against the previous layout: tests pass; coded runs, test outputs, graph.json and
policy.json are byte-identical; the built site differs only in comments that name file paths.
The published file names Ken's bot downloads (solver.js, graph.js, data/policy.json) are unchanged.

Also removes the undefined/ folder of QA screenshots committed by mistake (the QA scripts now
default to a temp folder) and test.txt, and drops the unused npm workspaces.


Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ekcSchiano
2026-11-03 19:46:33 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent dcd5117373
commit 432e04c2c7
66 changed files with 201 additions and 2695 deletions
+31 -35
View File
@@ -7,32 +7,29 @@ Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publis
## Layout
The project has four parts, built on one shared core. Each folder has its own README.
```
apps/atlas/ the web page. The Valley (explorer.js: the map, the dragons, routes on demand; dragons.js:
original silhouettes), The Odds, The Dragons, Deep Dive, Data & Methods
packages/core/ @threshing-atlas/core, no dependencies, runs in Node and the browser:
src/engine.js coding rules and statistics for the community form
src/graph.js the shared route graph: scene graph, decision-variable graph,
bot-to-form crosswalk, Beta credible intervals
src/solver.js best decisions per dragon colour (history-dependent decision tree,
empirical-Bayes back-off, two objectives) + the bot advisor
packages/ingest/ @threshing-atlas/ingest: DragonKind bot importer (fingerprints and choice text only)
research/python/ independent Python implementation of the coding rules and statistics
tests/ JS/Python parity tests (they fail loudly if the two disagree)
scripts/build-site.js assembles the static site into dist/
scripts/build-graph.js combines every source into data/public/graph.json
scripts/build-policy.js solves for every colour -> data/public/policy.json; policy-report.js -> docs/POLICY.md
data/raw/ source exports (form .xlsx; the register export stays local, git-ignored)
data/public/ what the page loads: form_responses.json, register.json, graph.json
data/sources.json where the community spreadsheets and the PDF are downloaded from
data/derived/ coded runs (runs_clean.csv)
data/bot/ anonymised DragonKind bot runs, added by pull request
docs/ ROADMAP.md, BOT_DATA_PROTOCOL.md, REDDIT_POST.md
core/ the shared model every part uses: the run format and coding rules (engine.js, with the
Python mirror in core/python/clean.py), the route graph and bot crosswalk (graph.js), and the
decision model and bot advisor (solver.js). No dependencies; runs in Node and the browser.
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
analysis/ making sense of it: stats/ (Python statistics, mirrored in core/engine.js), network/ (builds
data/public/graph.json), reports/ (generated reports such as POLICY.md)
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
and experiments (docs/BOT_INTEGRATION.md)
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
data/ raw/ (source exports, not committed), public/ (what the page and the bot load), derived/
(coded runs, PDF fingerprint), bot/ (anonymised bot runs), sources.json (download links)
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
integration, colours, roadmap
tests/ core/ (unit tests), parity/ (the JS and Python implementations must agree), fixtures/
```
## Run it locally
Requirements: Node 18+, Python 3.9+ (`pip install -r research/python/requirements.txt`).
Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
```
npm run dev # builds dist/ and serves it at http://localhost:8000
@@ -50,12 +47,12 @@ run it by hand from the Actions tab, or locally with `npm run check-sources`.
## Best routes (the solver)
`npm run policy` rebuilds `data/public/policy.json` and the readable `docs/POLICY.md` (about 2.5 minutes:
`npm run policy` rebuilds `data/public/policy.json` and the readable `analysis/reports/POLICY.md` (about 4 minutes:
200 posterior draws for two data samples). For every colour it gives the best decision at every point under two
objectives: the chance of bonding that colour on this attempt, and the chance the account eventually ends on that
colour when you die on purpose whenever a run goes wrong. Chance events (which dragon appears, whether the shadow
shows) may depend on earlier choices; how much is learned from the data. See the top of
`packages/core/src/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
`core/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
## Bot data
@@ -75,22 +72,20 @@ looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
`packages/core/src/engine.js` and `research/python/clean.py` need updating.
`core/engine.js` and `core/python/clean.py` need updating.
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
```
cd research/python
python export_raw.py ../../data/raw/form/NEW_EXPORT.xlsx ../../data/public/form_responses.json
python clean.py ../../data/raw/form/NEW_EXPORT.xlsx
cd ../..
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
python core/python/clean.py data/public/form_responses.json data/derived/runs_clean.csv
npm test
```
## Changing a coding rule
Every rule lives in `codeRow()` in `packages/core/src/engine.js` and `code_row()` in
`research/python/clean.py`. Change both and run `npm test`.
Every rule lives in `codeRow()` in `core/engine.js` and `code_row()` in
`core/python/clean.py`. Change both and run `npm test`.
## Statistics in one paragraph
@@ -101,17 +96,18 @@ players who log every run, key outcomes are also modelled with ridge-penalised l
(λ = 1) that includes data source and chase goal, and route rankings use a Mantel–Haenszel odds ratio
stratified by the same factors. The page's Methods section has the details.
## Licences
## Privacy and licences
Code: MIT (`LICENSE`). Data produced by this project: CC BY 4.0 (`DATA_LICENSE.md`), which also lists
third-party sources that keep their own terms.
third-party sources that keep their own terms. The rules every part follows (what may be published, what stays
local, how bot accounts are anonymised) are in `docs/PRIVACY.md`.
## Release checks
```
npm test # parity and unit tests
python research/python/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
python site/qa/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
npm run build && python3 -m http.server 8765 -d dist &
node scripts/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
node scripts/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
node site/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
node site/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
```