From ee07ee143e86cc871e66e84b5e44b011b6f4648e Mon Sep 17 00:00:00 2001 From: ekcSchiano Date: Thu, 12 Nov 2026 02:56:00 +0000 Subject: [PATCH] docs: project context for picking the project up in a new conversation Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ --- docs/PROJECT_CONTEXT.md | 262 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 262 insertions(+) create mode 100644 docs/PROJECT_CONTEXT.md diff --git a/docs/PROJECT_CONTEXT.md b/docs/PROJECT_CONTEXT.md new file mode 100644 index 0000000..f0f0b23 --- /dev/null +++ b/docs/PROJECT_CONTEXT.md @@ -0,0 +1,262 @@ +# Threshing Atlas: project context + +A briefing for picking this project up in a new conversation. Written 2026-11-12 from the working state on +Kathleen's Mac (`~/PycharmProjects/threshing-atlas`, latest commit `4837332`). + +--- + +## 1. What this is + +**Dragonkind** is a browser game set in the world of *Fourth Wing*. Its centrepiece, **the threshing**, is a short +choose-your-path run: a cadet walks into a valley, makes a series of choices, the game rolls dice at certain points, +and the run ends with the cadet either **bonded to a dragon** (Red, Orange, Brown, Green, Blue or Black) or +**incinerated**. There's a cooldown of about four hours between attempts, Blue and Black are rare, and players want +to know which choices give the best odds of the dragon they want. + +The **Threshing Atlas** is a hobby, non-commercial, open-data project by **Kathleen Schiano (GitHub: ekcSchiano)** +that: + +1. **collects** every logged run it can find (a community Google Form and spreadsheet, Reddit, a bot, direct reports), +2. **codes** them into one clean table, one run per row (`data/public/runs_clean.csv`), +3. **analyses** them (factor tables, models, route rankings) for a public page, +4. **plans** routes: a Bayesian model of the game's dice, solved as a decision tree, that recommends the best + choice at every step for each target colour, and +5. **feeds a bot**: Ken's DragonKind bot plays the game, uses the Atlas's route model to choose, and sends its runs + back, so the model keeps learning. + +Kathleen is a PhD researcher (informatics); she prefers **Python**, **declarative specs (YAML, data dictionaries, +Croissant metadata)**, and JavaScript only where it must run in a browser or in Ken's Node bot. + +## 2. People, places and accounts + +| What | Where | Notes | +|---|---|---| +| The Atlas repo (canonical copy) | `~/PycharmProjects/threshing-atlas` on Kathleen's Mac | Python venv at `.venv` (Python 3.9); PyCharm. Has `scipy`; **Node is not installed** (so `graph.json` can't be rebuilt there; `brew install node` fixes it). | +| GitHub repo | `github.com/ekcSchiano/threshing-atlas` (public) | **Account suspended** (flagged by GitHub's abuse detection, probably bursts of automated API activity from an AI assistant). Reinstatement request open with GitHub Support; they asked how she plans to use GitHub (answered: pet projects). GitHub Pages site paused. The Mac copy's `origin` still points here. | +| Gitea | `git.kenschiano.net` (Ken's server) | The Atlas is prepared to move there (under Kathleen's own account, visibility "limited"), but **Ken's Gitea has a problem; skipped for now**. | +| Ken | Gitea `kschiano`; GitHub issues author | Collaborator. Runs the **DragonKind bot** (private repo, now on his Gitea at `kschiano/DragonKind`). Kathleen has a local clone at `~/Desktop/DragonKind`: **read-only for us, never modify it** without asking. | +| The page | Claude artifact "Threshing Route Atlas": https://claude.ai/artifact/Amr4D5Av7sBHkWvjJuXM8A | Last republished 2026-11-04 (older data). Kathleen will view new builds locally / as an artifact while GitHub Pages is paused. | +| Community sources | u/ClumsyBadger's "Threshing Routes" Google Form (https://forms.gle/DRpj9tG8gb5E4Hcu7) and its responses spreadsheet; the "Unofficial Dragon Register"; the community route PDF; r/fourthwing Dragonkind megathreads (Part 3 `1wvwfp9`, Part 4 `1wz5cnj`) | Linked from the megathread post. u/Old_Regular3018 computes the PDF's percentages from the same spreadsheet; u/Smart_Trifle_2099 posted "precise Blue pathways" and Black theories. | +| Backups | `threshing-atlas.bundle` sent in chat (git bundle of all branches) | `git clone threshing-atlas.bundle threshing-atlas` | + +## 3. The game, as the Atlas models it + +The game is written down declaratively in **`core/spec/game.yaml`** (the single source of truth for the route +model, the advisor and the bot). In order: + +1. **Company**: enter alone, with 1 or with 2 squadmates. **Leave** your squadmates or stay together. +2. **First direction**: East, West or Center (the forest). +3. **First dragon** (roll): Red, Brown, Green or Orange. **React**: hold still, respect/face it, run, hide, weapon, + meet its gaze, protect squadmate (which reactions exist depends on the dragon). **Survive the first dragon** (roll). +4. **The scream**: investigate it or keep walking. + - **Investigate → the clearing** (your squadmate attacked by two bully cadets): help, or walk away. + Help → **the fight**: pause to assess (then the dagger: engage the smaller cadet or charge the bigger one + anyway), charge the smaller cadet ("even the numbers"), or charge the bigger one ("put yourself in the middle"). + **Survive the fight** (roll). Then **the bullies' fate**: kill them yourself, squadmate's revenge, or leave them + to the dragons. Then a **fork**: Forest, West or East. + - **Keep walking → the wounded cadet**: tourniquet, leave them to die, or take their weapons. +5. **Survive the hour** (roll). **After an hour**: climb the tallest tree, or go to the valley centre. +6. **Tree**: the **shadow** may appear (roll; **gated**, see below) → the shadow's colour (Black or Blue) → bonding + encounter. If no shadow: **climb down quickly** (the Red below the tree: jump on it, or let it pass → the two + Greens) or **slowly** (the two Greens). +7. **Valley**: **face the Red**, or pass it into the **meadow**, where a dragon waits (roll: Orange, Brown, or a Blue). +8. **Bonding encounters** (`encounters` in game.yaml): valley Red, tree Red, the two Greens (an Orange or Brown can + swoop in and rescue/bond you), meadow Orange/Brown/Blue, the Blue at the redwood (shadow), the Black (shadow). + A rider can attempt the bond or **die on purpose** ("seek Malek") rather than bond the wrong dragon. + +Key modelling facts now in game.yaml: +- **The shadow is gated**: it can only appear if you **investigated the scream and helped your squadmate** (the route + PDF's "Black Dragon Gates"; data agree: 78 shadows in 4,199 gated tree runs vs 2 in 646 others, both likely + mis-entered). A closed gate fixes the roll at "no" and those runs aren't counted for it. +- **Possible bonds per encounter**: each encounter gives only its own colour (the two Greens also Orange/Brown). + Anything else in the data is treated as a data-entry error (a form row once claimed a Green at the shadow's Blue, + which broke the Green recommendation). + +## 4. The four prongs + core (repo layout) + +``` +core/ spec/form.yaml (how each form answer is coded), spec/game.yaml (the game) + atlas/ (Python): coding.py (form → runs), model.py (counts, back-off, Params), solver.py (decision + tree, posterior draws, policy export), bias.py (reporting bias), runs.py (read runs_clean.csv), spec.py + solver.js (the small JS advisor), graph.js (route graph helpers + the bot crosswalk, generated block) +collection/ pipeline.yaml (every source, declaratively), run.py (`python -m collection ...`), + etl/ (extract, transform/*, validate, load, runs.py coders, metadata.py (Croissant), codegen.py, + report_form.py, handoff.py (zips, local refresh), forge.py (Gitea/GitHub API)), + specs/bot.yaml (bot labels + crosswalk), specs/report_form.yaml, manual/reddit_results.yaml +analysis/ stats/measures.yaml (what the page measures), site_stats.py, stats_core.py; + network/build-graph.js (graph.json); reports/POLICY.md (written by policy_report.py) +planner/ policy.yaml (which runs the model learns from, low-bias sample, reporting-bias settings), + build_policy.py, policy_report.py +site/ atlas/ (index.html, app.js, ui.js, explorer.js (valley map), valley.js, dragons.js, register.js), + build.js (→ dist/), qa/ (crawl.js, mapcheck.js: Playwright checks) +data/ public/ (everything published: runs_clean.csv, form_responses.json, register.json, reports.json, + stats.json, policy.json, graph.json, links.json), dictionary.yaml, croissant.json, manifest.json, + bot/ (scenes.json, trajectories-*.jsonl: anonymised bot runs), derived/pdf_fingerprint.json, + raw/ (git-ignored: current + previous raw file per source; never published) +tests/ core/ (coding rules, game, solver, stats, bias + JS advisor/graph tests), collection/ (pipeline), + fixtures/, parity/ +docs/ BOT_INTEGRATION.md, BOT_DATA_PROTOCOL.md, PRIVACY.md, REPO_SETTINGS.md, ROADMAP.md, COLOURS.md +forge.yaml where the repo is hosted (gitea, git.kenschiano.net, repo: YOUR-GITEA-USERNAME/threshing-atlas placeholder) +.gitea/workflows/ CI + daily update for Gitea .github/workflows/ CI, daily update, Pages for GitHub (paused) +``` + +### 4a. Data collection: ETL (`collection/`) + +`collection/pipeline.yaml` declares every source: how to **extract** it, which **transform**, its **privacy** rules, +where it **loads**, and which **coder** turns it into runs. + +- **extract** only fetches into `data/raw//current.` (the replaced one becomes `previous.`; + nothing older is kept). `data/manifest.json` records the current raw checksum and change date. +- **transform** is pure (no network, no clock) → the public source file. +- **validate** checks runs against `data/dictionary.yaml` (types, every allowed value, unique ids). A new answer the + rules don't know **stops the load** and is reported (add it to `core/spec/form.yaml`). +- **load** is the only stage that writes; byte-stable. +- **build** codes every public source file into `data/public/runs_clean.csv` and writes generated files: the + "Report a run" issue form, the crosswalk block in `core/graph.js`, `data/public/links.json`, `data/croissant.json`. + +| `source` in runs_clean | Pipeline source | Runs now | Notes | +|---|---|---|---| +| reddit spreadsheet | `reddit_spreadsheet` (the Threshing Routes form + sheet) | 8,132 | coded by `core/spec/form.yaml`; `entry` column keeps the sheet's own label ("Form" = form submission; others = bulk-pasted logs) | +| bot | `bot` (Ken's bot, imported from a folder or zip) | 421 | per-account folders; account names salted-hashed (salt in `data/raw/bot/account-salt.txt` on the Mac, never committed); `bot_mode` column | +| reddit thread | `reddit_thread` | 12 | dragonkind.com result pages read by hand (`collection/manual/reddit_results.yaml`); Reddit API extractor ready but needs `REDDIT_CLIENT_ID/SECRET` | +| reported | `reported` ("Report a run" issue form) | 0 | GitHub issues; switched to Gitea via forge.yaml (currently 404s: GitHub suspended, Gitea not set up) | +| reddit pdf | `reddit_pdf` | 0 | the route PDF is a flowchart, not runs: only its **fingerprint** is kept (never redistributed); changes are reported line by line | +| (none) | `register` (Unofficial Dragon Register) | 0 | bonded dragons without routes; names and photos dropped | + +Every column is described in `data/dictionary.yaml`; `data/croissant.json` is Croissant 1.0 metadata built from it. + +### 4b. Data analysis (`analysis/`) + +`analysis/stats/site_stats.py` reads runs_clean (sources in `measures.yaml`: **spreadsheet only** for now) and writes +`data/public/stats.json`: every number the page shows, for six fixed filter presets (all, form only, bulk only, not +chasing Blue/Black, chasing, unflagged). Factor tables (Fisher/Mantel–Haenszel, BH q-values), ridge-logistic models, +route rankings at thresholds 5/10/20/30, per-stage route map counts. The page computes nothing itself. +`analysis/network/build-graph.js` writes `graph.json` (route graph per source + the bot's scene graph). + +### 4c. Route planner (`planner/` + `core/atlas/`) + +- **Model**: each roll (first dragon, survive sighting/fight/hour, shadow, shadow colour, meadow dragon) is a + categorical with a **hierarchical Dirichlet back-off** over its context (earlier choices); the shrinkage strength + (alpha) is fitted by empirical Bayes. Bond outcomes per encounter likewise. Malek/deliberate burns are not counted + as failed bonds. +- **Learns from** (`planner/policy.yaml`): reddit spreadsheet + reported + **bot** runs. +- **Reporting-bias correction** (`core/atlas/bias.py`): form submitters over-report rare sightings. For each rare roll + and each group, rho = odds(rare | group) / odds(rare | complete logs), where complete logs = **bot + bulk logs**. + Rare outcomes from biased groups count 1/rho. Current estimates (meadow Blue): **form, chasing Blue/Black rho 3.85 + [2.98–5.14]; form, other 3.17 [2.33–4.41]**. For the shadow, complete logs have **0 shadows in 579 gated tree runs**, + so the bias can't be estimated and the shadow is learned from complete logs only. A roll seen only one way is pooled + fully (alpha max). Caveats: assumes the bias is the same on every route; intervals don't include rho's uncertainty. +- **Solver** (`core/atlas/solver.py`): walks game.yaml as a decision tree; two goals per colour: "this attempt" + (maximise P(colour)) and "eventually" (Dinkelbach ratio: P(X) / (P(X)+P(other bond)), i.e. dying on purpose + otherwise). 200 posterior draws give intervals and **stillBest** (share of draws where the recommended choice is best). + Two samples: **all** and **lowBias** (bot + bulk + riders not chasing rare dragons). +- **Output**: `data/public/policy.json` (game, count tables, alphas, reportingBias, support, trees per colour). +- **JS advisor** (`core/solver.js`): reads policy.json, re-solves, `advisor(sample, colour, objective, opts)`, + `recommendOption()` maps on-screen text to options (game.yaml `bot_text`). **Thompson sampling**: + `{ sampling: 'thompson', seed }` follows one posterior draw per attempt and returns a **propensity**. + +Current odds per attempt (sample "all", latest build `538bc1e3d694`): + +| Colour | Per attempt (90% range) | Low-bias sample | stillBest of first choice | +|---|---|---|---| +| Red | 93.1% (86.9–97.4) | 94.1% | 0.30 | +| Green | 25.2% (21.7–28.8) | 24.2% | 0.42 | +| Orange | 13.5% (11.3–15.7) | 16.7% | 0.55 | +| Brown | 9.3% (7.7–11.2) | 10.7% | 0.52 | +| Blue | 2.5% (2.1–3.0) | 1.9% | 0.45 | +| Black | 0.05% (0.00–0.25) | 0.04% | 0.31 | + +(Before the bot data and bias correction, the model said Blue 7.3% and Black 2.1%: inflated by reporting bias.) + +### 4d. Data visualisation (`site/`) + +A single static page (tabs: Valley map, Odds, Dragons, Deep dive, Data & Methods, **Report a run**). The **valley +map** (`explorer.js`) uses progressive disclosure: key moments only, a point's options appear when selected, picking +a dragon shows its best path and choices; later choices keep the path you picked. The page shows precomputed +`stats.json` and runs the JS advisor on `policy.json`. Links come from `data/public/links.json` (filled in by +`site/build.js`). Build: `node site/build.js` → `dist/`. QA: `site/qa/crawl.js` (needs a local server on :8765), +`site/qa/mapcheck.js`. Published so far as the Claude artifact above. + +## 5. The bot loop (Ken) + +- **Atlas → bot**: `python3 -m collection export-bot-kit` → `exports/atlas-for-bot--.zip` + (policy.json, solver.js, graph.js, README with checksums). Ken unzips into the bot's `vendor/threshing-atlas/`. + Latest: `atlas-for-bot-2026-11-12-538bc1e3d694.zip`. +- **Bot → atlas**: Ken zips the bot's `data/` and `logs/` folders; save in `inbox/` and run + `python3 -m collection refresh --bot inbox/`. The importer reads `data/knowledge.json` and every + `data/accounts//runs.jsonl` + `logs/accounts//encounters.jsonl` (top-level files mirror the first + account and are skipped). Bot **schema-2** run records carry steps, what was seen (shadow, dragons), propensity, + decisionMode, policyId. No game prose or account names are published (`check_public` refuses them). +- **How the bot should choose** (docs/BOT_INTEGRATION.md): goal accounts follow the best route + (`advisor(policy.samples.all, target, 'eventually')`); **research accounts use Thompson sampling** with a new seed + per attempt and log `propensity` with `decisionMode: "atlas_thompson"`. +- Ken's data so far: 13 accounts, 415 attempts (to 2026-11-10); research accounts burn on purpose when no shadow at + the tree / no Blue in the meadow (`TREE_NO_SHADOW_AUTO_BURN`, `MEADOW_NO_BLUE_AUTO_BURN`). + +## 6. Findings so far + +- **Reporting bias is large.** Meadow Blue: complete logs ~10% (bot 16/126, bulk 30/328) vs form ~33%. +- **Black is very rare.** 0 shadows in 579 gated tree runs from complete logs; form data show ~2% (inflated). Model: + ~0.05% per attempt. Fits Smart_Trifle_2099's guess of 1/500–1/1000 tree runs for a Black sighting. +- **Route doesn't seem to move the rare rolls.** Shadow rate doesn't differ across routes (chi² 44 on 40 df); the + "precise Blue pathways" (Smart_Trifle_2099 and the megathread post's ~47%/~63% routes) did no better than other + meadow routes (23–30% vs 30% in the spreadsheet). Reaching the meadow seems to be the "unlock". +- **The gate is real**: the shadow needs investigate + help. +- **Tree Red is the hardest bond**: 1.7% (38/2,207), matching the post's "2%". +- **First attempts** see the meadow Blue less often (16% vs 31%), surviving adjustment in the site's model. +- Weekly swings in rates (and "the game was patched" claims) are confounded by who reports when. + +## 7. How to run things (on the Mac, PyCharm terminal, venv active) + +``` +python3 -m collection refresh # the daily update locally (~3 min): sources → runs → graph* → model → stats → build +python3 -m collection refresh --bot inbox/x.zip # same, importing a bot zip +python3 -m collection update --source NAME # one source; --file NAME=PATH to use a local file +python3 -m collection build [--check] # rebuild runs_clean + generated files (CI uses --check) +python3 -m collection export-bot-kit # zip for Ken +python3 planner/build_policy.py # route model (~30–75 s); python3 planner/policy_report.py +python3 analysis/stats/site_stats.py # page statistics (needs scipy) +node analysis/network/build-graph.js # graph.json (*needs Node) +node site/build.js # the page → dist/ +npm test # = build --check + Python unit tests + JS tests +``` + +Reports land in `exports/last-report.md`. If a refresh stops with "values not in the dictionary", add the new form +answer to `core/spec/form.yaml` (or the value to `data/dictionary.yaml`) and run `python3 -m collection build`. + +## 8. Conventions and constraints (keep these) + +- **Python first; declarative specs** (YAML for the form, game, pipeline, measures, planner, bot crosswalk, report + form; dictionary + Croissant). JS only for the browser page and Ken's Node bot (`solver.js`, `graph.js`). +- **Generated files are generated**: don't hand-edit `runs_clean.csv`, `croissant.json`, `links.json`, the + crosswalk block in `core/graph.js`, `.github/ISSUE_TEMPLATE/report-a-run.yml`. Edit the spec, then build. +- **Privacy**: no game prose, no account names/ids, no emails (redacted), no rider names (register), no Reddit or + GitHub usernames; the route PDF is never redistributed (fingerprint only). Never commit `data/raw/`, `.env`, + `.browser-profile/`, salts or tokens; tokens only in repository secrets. Don't scrape Discord with a personal token; + Reddit only via the official API. +- **Don't write to Ken's bot repo** (`~/Desktop/DragonKind`) without asking. +- **Commits** as `ekcSchiano` / Kathleen's email, ending with `Co-Authored-By: Claude …` and `Claude-Session: …` + lines. On GitHub, main was protected (PR + required `test` check, auto-merge). **Go easy on automated API + activity** (that's the likely cause of the GitHub suspension). +- Kathleen likes: progressive disclosure on the map, honest uncertainty, explaining what changed and why. + +## 9. Status and open threads + +- **Uncommitted / pending**: nothing important; Mac repo is clean at `4837332` (plus `.idea/`, `exports/`, `inbox/` + ignored). The artifact page is older than the data; rebuild with `node site/build.js` (needs Node) and republish. +- **GitHub**: suspended; reinstatement in progress. When back: `git push` to a branch + PR (main is protected); + GitHub Pages resumes from `.github/workflows/pages.yml`; set `forge.yaml` `kind: github` (or keep Gitea + mirror). +- **Gitea**: ready but on hold (Ken's server has a problem). Steps in `docs/REPO_SETTINGS.md`; set your username in + `forge.yaml`; the `.gitea/workflows/` are untested (runner label assumed `ubuntu-latest`). +- **"Report a run"** button points at a Gitea URL that doesn't exist yet; the Google Form is the working intake. +- **Not done yet / ideas**: + - Include bot and reported runs in the page's statistics (currently spreadsheet only; bot choices follow a strategy). + - Propagate the bias estimate's uncertainty (rho) into the route model's intervals; consider a per-source + reporting-bias model inside the posterior draws. + - Inverse-propensity weighting of bot runs using logged propensities. + - Port `build-graph.js` to Python; Reddit extractor live once API secrets exist; "Community claims, checked" + section (claims YAML recomputed against data); a "random run?" question on the report form. + - Reviews still pending from the original plan: analysis, planner (experiments.json, encounter answers), site. +- **Environment gotchas** (for Claude sessions): the cloud workspace and the Mac's Linux helper can't reach Google, + Gitea or PyPI; Kathleen's own Terminal can. The helper lacks `scipy`. Git in the helper needs delete permission + for its lock files.