Refactor: lay the repo out by section (core, collection, analysis, planner, site) (#8)
Moves only; no behaviour changes. Every file now lives under the part of the project it belongs to, each section has a README, and docs/PRIVACY.md collects the rules all parts follow. core/ engine.js, graph.js, solver.js, python/clean.py collection/ sources/ (form, register, PDF check, daily update), bot/ (importer) analysis/ stats/ (Python statistics), network/ (build-graph.js), reports/ (POLICY.md) planner/ build-policy.js, policy-report.js site/ atlas/ (the page), build.js, qa/ tests/ core/ (unit), parity/ (JS vs Python), fixtures/ Checked against the previous layout: tests pass; coded runs, test outputs, graph.json and policy.json are byte-identical; the built site differs only in comments that name file paths. The published file names Ken's bot downloads (solver.js, graph.js, data/policy.json) are unchanged. Also removes the undefined/ folder of QA screenshots committed by mistake (the QA scripts now default to a temp folder) and test.txt, and drops the unused npm workspaces. Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
dcd5117373
commit
432e04c2c7
@@ -7,32 +7,29 @@ Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publis
|
||||
|
||||
## Layout
|
||||
|
||||
The project has four parts, built on one shared core. Each folder has its own README.
|
||||
|
||||
```
|
||||
apps/atlas/ the web page. The Valley (explorer.js: the map, the dragons, routes on demand; dragons.js:
|
||||
original silhouettes), The Odds, The Dragons, Deep Dive, Data & Methods
|
||||
packages/core/ @threshing-atlas/core, no dependencies, runs in Node and the browser:
|
||||
src/engine.js coding rules and statistics for the community form
|
||||
src/graph.js the shared route graph: scene graph, decision-variable graph,
|
||||
bot-to-form crosswalk, Beta credible intervals
|
||||
src/solver.js best decisions per dragon colour (history-dependent decision tree,
|
||||
empirical-Bayes back-off, two objectives) + the bot advisor
|
||||
packages/ingest/ @threshing-atlas/ingest: DragonKind bot importer (fingerprints and choice text only)
|
||||
research/python/ independent Python implementation of the coding rules and statistics
|
||||
tests/ JS/Python parity tests (they fail loudly if the two disagree)
|
||||
scripts/build-site.js assembles the static site into dist/
|
||||
scripts/build-graph.js combines every source into data/public/graph.json
|
||||
scripts/build-policy.js solves for every colour -> data/public/policy.json; policy-report.js -> docs/POLICY.md
|
||||
data/raw/ source exports (form .xlsx; the register export stays local, git-ignored)
|
||||
data/public/ what the page loads: form_responses.json, register.json, graph.json
|
||||
data/sources.json where the community spreadsheets and the PDF are downloaded from
|
||||
data/derived/ coded runs (runs_clean.csv)
|
||||
data/bot/ anonymised DragonKind bot runs, added by pull request
|
||||
docs/ ROADMAP.md, BOT_DATA_PROTOCOL.md, REDDIT_POST.md
|
||||
core/ the shared model every part uses: the run format and coding rules (engine.js, with the
|
||||
Python mirror in core/python/clean.py), the route graph and bot crosswalk (graph.js), and the
|
||||
decision model and bot advisor (solver.js). No dependencies; runs in Node and the browser.
|
||||
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
|
||||
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
|
||||
analysis/ making sense of it: stats/ (Python statistics, mirrored in core/engine.js), network/ (builds
|
||||
data/public/graph.json), reports/ (generated reports such as POLICY.md)
|
||||
planner/ the route planner: builds data/public/policy.json and its report; the bot uses it for advice
|
||||
and experiments (docs/BOT_INTEGRATION.md)
|
||||
site/ the Atlas web page (site/atlas/), its build script (site/build.js) and QA checks (site/qa/)
|
||||
data/ raw/ (source exports, not committed), public/ (what the page and the bot load), derived/
|
||||
(coded runs, PDF fingerprint), bot/ (anonymised bot runs), sources.json (download links)
|
||||
docs/ guides that cross sections: privacy and licences, repo settings, bot data protocol, bot
|
||||
integration, colours, roadmap
|
||||
tests/ core/ (unit tests), parity/ (the JS and Python implementations must agree), fixtures/
|
||||
```
|
||||
|
||||
## Run it locally
|
||||
|
||||
Requirements: Node 18+, Python 3.9+ (`pip install -r research/python/requirements.txt`).
|
||||
Requirements: Node 18+, Python 3.9+ (`pip install -r requirements.txt`).
|
||||
|
||||
```
|
||||
npm run dev # builds dist/ and serves it at http://localhost:8000
|
||||
@@ -50,12 +47,12 @@ run it by hand from the Actions tab, or locally with `npm run check-sources`.
|
||||
|
||||
## Best routes (the solver)
|
||||
|
||||
`npm run policy` rebuilds `data/public/policy.json` and the readable `docs/POLICY.md` (about 2.5 minutes:
|
||||
`npm run policy` rebuilds `data/public/policy.json` and the readable `analysis/reports/POLICY.md` (about 4 minutes:
|
||||
200 posterior draws for two data samples). For every colour it gives the best decision at every point under two
|
||||
objectives: the chance of bonding that colour on this attempt, and the chance the account eventually ends on that
|
||||
colour when you die on purpose whenever a run goes wrong. Chance events (which dragon appears, whether the shadow
|
||||
shows) may depend on earlier choices; how much is learned from the data. See the top of
|
||||
`packages/core/src/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
|
||||
`core/solver.js` for the model and `docs/BOT_INTEGRATION.md` for how the bot uses it.
|
||||
|
||||
## Bot data
|
||||
|
||||
@@ -75,22 +72,20 @@ looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
|
||||
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
|
||||
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
|
||||
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
|
||||
`packages/core/src/engine.js` and `research/python/clean.py` need updating.
|
||||
`core/engine.js` and `core/python/clean.py` need updating.
|
||||
|
||||
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
|
||||
|
||||
```
|
||||
cd research/python
|
||||
python export_raw.py ../../data/raw/form/NEW_EXPORT.xlsx ../../data/public/form_responses.json
|
||||
python clean.py ../../data/raw/form/NEW_EXPORT.xlsx
|
||||
cd ../..
|
||||
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
|
||||
python core/python/clean.py data/public/form_responses.json data/derived/runs_clean.csv
|
||||
npm test
|
||||
```
|
||||
|
||||
## Changing a coding rule
|
||||
|
||||
Every rule lives in `codeRow()` in `packages/core/src/engine.js` and `code_row()` in
|
||||
`research/python/clean.py`. Change both and run `npm test`.
|
||||
Every rule lives in `codeRow()` in `core/engine.js` and `code_row()` in
|
||||
`core/python/clean.py`. Change both and run `npm test`.
|
||||
|
||||
## Statistics in one paragraph
|
||||
|
||||
@@ -101,17 +96,18 @@ players who log every run, key outcomes are also modelled with ridge-penalised l
|
||||
(λ = 1) that includes data source and chase goal, and route rankings use a Mantel–Haenszel odds ratio
|
||||
stratified by the same factors. The page's Methods section has the details.
|
||||
|
||||
## Licences
|
||||
## Privacy and licences
|
||||
|
||||
Code: MIT (`LICENSE`). Data produced by this project: CC BY 4.0 (`DATA_LICENSE.md`), which also lists
|
||||
third-party sources that keep their own terms.
|
||||
third-party sources that keep their own terms. The rules every part follows (what may be published, what stays
|
||||
local, how bot accounts are anonymised) are in `docs/PRIVACY.md`.
|
||||
|
||||
## Release checks
|
||||
|
||||
```
|
||||
npm test # parity and unit tests
|
||||
python research/python/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
|
||||
python site/qa/check_colours.py # dragon colours stay distinguishable (docs/COLOURS.md)
|
||||
npm run build && python3 -m http.server 8765 -d dist &
|
||||
node scripts/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
|
||||
node scripts/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
|
||||
node site/qa/crawl.js /tmp/qa # every tab on desktop and phone: text, links, buttons, errors, overflow
|
||||
node site/qa/mapcheck.js /tmp/qa # every dragon's routes: off-map labels, overlaps, too-faint dragons
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user