Refactor: lay the repo out by section (core, collection, analysis, planner, site) (#8)

Moves only; no behaviour changes. Every file now lives under the part of the project it
belongs to, each section has a README, and docs/PRIVACY.md collects the rules all parts follow.

  core/        engine.js, graph.js, solver.js, python/clean.py
  collection/  sources/ (form, register, PDF check, daily update), bot/ (importer)
  analysis/    stats/ (Python statistics), network/ (build-graph.js), reports/ (POLICY.md)
  planner/     build-policy.js, policy-report.js
  site/        atlas/ (the page), build.js, qa/
  tests/       core/ (unit), parity/ (JS vs Python), fixtures/

Checked against the previous layout: tests pass; coded runs, test outputs, graph.json and
policy.json are byte-identical; the built site differs only in comments that name file paths.
The published file names Ken's bot downloads (solver.js, graph.js, data/policy.json) are unchanged.

Also removes the undefined/ folder of QA screenshots committed by mistake (the QA scripts now
default to a temp folder) and test.txt, and drops the unused npm workspaces.


Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ekcSchiano
2026-11-03 19:46:33 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent dcd5117373
commit 432e04c2c7
66 changed files with 201 additions and 2695 deletions
+1 -1
View File
@@ -107,7 +107,7 @@ and it is exactly the randomised data the atlas needs to tell real history effec
- `samples.lowBias`: bulk logs and riders not chasing a rare dragon. Use it for absolute expectations, such as how
many attempts a Blue should take. Rare sightings are over-reported in form posts, so `all` is optimistic for Blue.
Current headline (from `docs/POLICY.md`): Black about 1.7% per attempt (about 60 attempts), Blue 3.7–7% per attempt
Current headline (from `analysis/reports/POLICY.md`): Black about 1.7% per attempt (about 60 attempts), Blue 3.7–7% per attempt
(14–28 attempts), depending on the sample.
## 2. From the bot to the atlas
+1 -1
View File
@@ -4,7 +4,7 @@ Two palettes, one per background. Both were chosen so that every pair of dragon
readers with the common colour-vision deficiencies, not only with normal vision. Check them with:
```
python research/python/check_colours.py
python site/qa/check_colours.py
```
It reports CIEDE2000 colour differences for every pair under normal vision and simulated protanopia, deuteranopia and
-1070
View File
File diff suppressed because it is too large Load Diff
+34
View File
@@ -0,0 +1,34 @@
# Privacy and licences
Rules every part of the project follows. If a change would break one, stop and ask first.
## What may be published
- **Code** is MIT (`LICENSE`). **Data this project produces** is CC BY 4.0 (`DATA_LICENSE.md`). Third-party
sources keep their own terms.
- **Community form responses:** published as exported, with email addresses in free-text answers redacted by
the coding rules (`core/engine.js`, `core/python/clean.py`).
- **Dragon Register:** only `data/public/register.json`. The raw export has rider names and stays local
(`data/raw/register/` is git-ignored).
- **The community route PDF:** never committed or redistributed. Only a fingerprint
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
names. The importer (`collection/bot/bot.js`) refuses to write anything that looks like either.
## The bot
- Ken's bot repository stays private. Only the anonymised import comes here.
- Account ids are salted and hashed. The salt (`DK_ACCOUNT_SALT`) lives in the environment and is never
committed.
- The bot's browser profiles, `.env` and credential files are never committed or shared.
## Sources we do not scrape
- **Discord:** not without the server admins' consent; never with a personal account token.
- **Reddit:** not collected yet. When it is, it should go through Reddit's official API with an app key, not page
scraping.
## Secrets
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
(`docs/REPO_SETTINGS.md`). Tokens go in secrets only: never in a file, issue, chat or commit.
+4 -4
View File
@@ -4,16 +4,16 @@ The aim is to move from summary statistics to path finding: for each dragon colo
policy that maximises the chance of bonding it, with honest uncertainty.
1. **Repo and packaging** (done): monorepo, CI, GitHub Pages, JS/Python parity tests.
2. **Graph format** (`packages/core`, first version done): typed scene, chance and terminal nodes; history-context keys;
2. **Graph format** (`core/`, first version done): typed scene, chance and terminal nodes; history-context keys;
per-source counts (bot, form, PDF, Discord); fingerprints and short labels only, no game prose.
Importers in `packages/ingest` for the bot's `knowledge.json`, `runs.jsonl` and `encounters.jsonl`
Importers in `collection/bot` for the bot's `knowledge.json`, `runs.jsonl` and `encounters.jsonl`
(done), the form (done, via the crosswalk in `graph.js`), and a PDF scene crosswalk (to do).
Daily source checks for the form, the register and the PDF (done).
3. **Context learning** (`research/`): tests of history dependence (G-tests, BH); variable-order
3. **Context learning** (`analysis/`): tests of history dependence (G-tests, BH); variable-order
Markov / context tree with BIC; temporal-tiered DAG structure learning (BDeu/BIC); latent
"character" model vs specific triggers vs memoryless baseline, compared on held-out likelihood;
hierarchical Dirichlet back-off pooling.
4. **Solver** (first version done: `packages/core/src/solver.js`, `docs/POLICY.md`; still to do: richer history models from step 3): backward induction on the augmented state (scene, history summary), i.e. an influence
4. **Solver** (first version done: `core/solver.js`, `analysis/reports/POLICY.md`; still to do: richer history models from step 3): backward induction on the augmented state (scene, history summary), i.e. an influence
diagram with perfect recall. Two objectives side by side per colour:
- P(target colour on this attempt);
- eventual success when deliberate deaths are free: maximise p_X / (p_X + p_other bond),