# Threshing Atlas: project context A briefing for picking this project up in a new conversation. Written 2026-11-12 from the working state on Kathleen's Mac (`~/PycharmProjects/threshing-atlas`, latest commit `4837332`). --- ## 1. What this is **Dragonkind** is a browser game set in the world of *Fourth Wing*. Its centrepiece, **the threshing**, is a short choose-your-path run: a cadet walks into a valley, makes a series of choices, the game rolls dice at certain points, and the run ends with the cadet either **bonded to a dragon** (Red, Orange, Brown, Green, Blue or Black) or **incinerated**. There's a cooldown of about four hours between attempts, Blue and Black are rare, and players want to know which choices give the best odds of the dragon they want. The **Threshing Atlas** is a hobby, non-commercial, open-data project by **Kathleen Schiano (GitHub: ekcSchiano)** that: 1. **collects** every logged run it can find (a community Google Form and spreadsheet, Reddit, a bot, direct reports), 2. **codes** them into one clean table, one run per row (`data/public/runs_clean.csv`), 3. **analyses** them (factor tables, models, route rankings) for a public page, 4. **plans** routes: a Bayesian model of the game's dice, solved as a decision tree, that recommends the best choice at every step for each target colour, and 5. **feeds a bot**: Ken's DragonKind bot plays the game, uses the Atlas's route model to choose, and sends its runs back, so the model keeps learning. Kathleen is a PhD researcher (informatics); she prefers **Python**, **declarative specs (YAML, data dictionaries, Croissant metadata)**, and JavaScript only where it must run in a browser or in Ken's Node bot. ## 2. People, places and accounts | What | Where | Notes | |---|---|---| | The Atlas repo (canonical copy) | `~/PycharmProjects/threshing-atlas` on Kathleen's Mac | Python venv at `.venv` (Python 3.9); PyCharm. Has `scipy`; **Node is not installed** (so `graph.json` can't be rebuilt there; `brew install node` fixes it). | | GitHub repo | `github.com/ekcSchiano/threshing-atlas` (public) | **Account suspended** (flagged by GitHub's abuse detection, probably bursts of automated API activity from an AI assistant). Reinstatement request open with GitHub Support; they asked how she plans to use GitHub (answered: pet projects). GitHub Pages site paused. The Mac copy's `origin` still points here. | | Gitea | `git.kenschiano.net` (Ken's server) | The Atlas is prepared to move there (under Kathleen's own account, visibility "limited"), but **Ken's Gitea has a problem; skipped for now**. | | Ken | Gitea `kschiano`; GitHub issues author | Collaborator. Runs the **DragonKind bot** (private repo, now on his Gitea at `kschiano/DragonKind`). Kathleen has a local clone at `~/Desktop/DragonKind`: **read-only for us, never modify it** without asking. | | The page | Claude artifact "Threshing Route Atlas": https://claude.ai/artifact/Amr4D5Av7sBHkWvjJuXM8A | Last republished 2026-11-04 (older data). Kathleen will view new builds locally / as an artifact while GitHub Pages is paused. | | Community sources | u/ClumsyBadger's "Threshing Routes" Google Form (https://forms.gle/DRpj9tG8gb5E4Hcu7) and its responses spreadsheet; the "Unofficial Dragon Register"; the community route PDF; r/fourthwing Dragonkind megathreads (Part 3 `1wvwfp9`, Part 4 `1wz5cnj`) | Linked from the megathread post. u/Old_Regular3018 computes the PDF's percentages from the same spreadsheet; u/Smart_Trifle_2099 posted "precise Blue pathways" and Black theories. | | Backups | `threshing-atlas.bundle` sent in chat (git bundle of all branches) | `git clone threshing-atlas.bundle threshing-atlas` | ## 3. The game, as the Atlas models it The game is written down declaratively in **`core/spec/game.yaml`** (the single source of truth for the route model, the advisor and the bot). In order: 1. **Company**: enter alone, with 1 or with 2 squadmates. **Leave** your squadmates or stay together. 2. **First direction**: East, West or Center (the forest). 3. **First dragon** (roll): Red, Brown, Green or Orange. **React**: hold still, respect/face it, run, hide, weapon, meet its gaze, protect squadmate (which reactions exist depends on the dragon). **Survive the first dragon** (roll). 4. **The scream**: investigate it or keep walking. - **Investigate → the clearing** (your squadmate attacked by two bully cadets): help, or walk away. Help → **the fight**: pause to assess (then the dagger: engage the smaller cadet or charge the bigger one anyway), charge the smaller cadet ("even the numbers"), or charge the bigger one ("put yourself in the middle"). **Survive the fight** (roll). Then **the bullies' fate**: kill them yourself, squadmate's revenge, or leave them to the dragons. Then a **fork**: Forest, West or East. - **Keep walking → the wounded cadet**: tourniquet, leave them to die, or take their weapons. 5. **Survive the hour** (roll). **After an hour**: climb the tallest tree, or go to the valley centre. 6. **Tree**: the **shadow** may appear (roll; **gated**, see below) → the shadow's colour (Black or Blue) → bonding encounter. If no shadow: **climb down quickly** (the Red below the tree: jump on it, or let it pass → the two Greens) or **slowly** (the two Greens). 7. **Valley**: **face the Red**, or pass it into the **meadow**, where a dragon waits (roll: Orange, Brown, or a Blue). 8. **Bonding encounters** (`encounters` in game.yaml): valley Red, tree Red, the two Greens (an Orange or Brown can swoop in and rescue/bond you), meadow Orange/Brown/Blue, the Blue at the redwood (shadow), the Black (shadow). A rider can attempt the bond or **die on purpose** ("seek Malek") rather than bond the wrong dragon. Key modelling facts now in game.yaml: - **The shadow is gated**: it can only appear if you **investigated the scream and helped your squadmate** (the route PDF's "Black Dragon Gates"; data agree: 78 shadows in 4,199 gated tree runs vs 2 in 646 others, both likely mis-entered). A closed gate fixes the roll at "no" and those runs aren't counted for it. - **Possible bonds per encounter**: each encounter gives only its own colour (the two Greens also Orange/Brown). Anything else in the data is treated as a data-entry error (a form row once claimed a Green at the shadow's Blue, which broke the Green recommendation). ## 4. The four prongs + core (repo layout) ``` core/ spec/form.yaml (how each form answer is coded), spec/game.yaml (the game) atlas/ (Python): coding.py (form → runs), model.py (counts, back-off, Params), solver.py (decision tree, posterior draws, policy export), bias.py (reporting bias), runs.py (read runs_clean.csv), spec.py solver.js (the small JS advisor), graph.js (route graph helpers + the bot crosswalk, generated block) collection/ pipeline.yaml (every source, declaratively), run.py (`python -m collection ...`), etl/ (extract, transform/*, validate, load, runs.py coders, metadata.py (Croissant), codegen.py, report_form.py, handoff.py (zips, local refresh), forge.py (Gitea/GitHub API)), specs/bot.yaml (bot labels + crosswalk), specs/report_form.yaml, manual/reddit_results.yaml analysis/ stats/measures.yaml (what the page measures), site_stats.py, stats_core.py; network/build-graph.js (graph.json); reports/POLICY.md (written by policy_report.py) planner/ policy.yaml (which runs the model learns from, low-bias sample, reporting-bias settings), build_policy.py, policy_report.py site/ atlas/ (index.html, app.js, ui.js, explorer.js (valley map), valley.js, dragons.js, register.js), build.js (→ dist/), qa/ (crawl.js, mapcheck.js: Playwright checks) data/ public/ (everything published: runs_clean.csv, form_responses.json, register.json, reports.json, stats.json, policy.json, graph.json, links.json), dictionary.yaml, croissant.json, manifest.json, bot/ (scenes.json, trajectories-*.jsonl: anonymised bot runs), derived/pdf_fingerprint.json, raw/ (git-ignored: current + previous raw file per source; never published) tests/ core/ (coding rules, game, solver, stats, bias + JS advisor/graph tests), collection/ (pipeline), fixtures/, parity/ docs/ BOT_INTEGRATION.md, BOT_DATA_PROTOCOL.md, PRIVACY.md, REPO_SETTINGS.md, ROADMAP.md, COLOURS.md forge.yaml where the repo is hosted (gitea, git.kenschiano.net, repo: YOUR-GITEA-USERNAME/threshing-atlas placeholder) .gitea/workflows/ CI + daily update for Gitea .github/workflows/ CI, daily update, Pages for GitHub (paused) ``` ### 4a. Data collection: ETL (`collection/`) `collection/pipeline.yaml` declares every source: how to **extract** it, which **transform**, its **privacy** rules, where it **loads**, and which **coder** turns it into runs. - **extract** only fetches into `data/raw//current.` (the replaced one becomes `previous.`; nothing older is kept). `data/manifest.json` records the current raw checksum and change date. - **transform** is pure (no network, no clock) → the public source file. - **validate** checks runs against `data/dictionary.yaml` (types, every allowed value, unique ids). A new answer the rules don't know **stops the load** and is reported (add it to `core/spec/form.yaml`). - **load** is the only stage that writes; byte-stable. - **build** codes every public source file into `data/public/runs_clean.csv` and writes generated files: the "Report a run" issue form, the crosswalk block in `core/graph.js`, `data/public/links.json`, `data/croissant.json`. | `source` in runs_clean | Pipeline source | Runs now | Notes | |---|---|---|---| | reddit spreadsheet | `reddit_spreadsheet` (the Threshing Routes form + sheet) | 8,132 | coded by `core/spec/form.yaml`; `entry` column keeps the sheet's own label ("Form" = form submission; others = bulk-pasted logs) | | bot | `bot` (Ken's bot, imported from a folder or zip) | 421 | per-account folders; account names salted-hashed (salt in `data/raw/bot/account-salt.txt` on the Mac, never committed); `bot_mode` column | | reddit thread | `reddit_thread` | 12 | dragonkind.com result pages read by hand (`collection/manual/reddit_results.yaml`); Reddit API extractor ready but needs `REDDIT_CLIENT_ID/SECRET` | | reported | `reported` ("Report a run" issue form) | 0 | GitHub issues; switched to Gitea via forge.yaml (currently 404s: GitHub suspended, Gitea not set up) | | reddit pdf | `reddit_pdf` | 0 | the route PDF is a flowchart, not runs: only its **fingerprint** is kept (never redistributed); changes are reported line by line | | (none) | `register` (Unofficial Dragon Register) | 0 | bonded dragons without routes; names and photos dropped | Every column is described in `data/dictionary.yaml`; `data/croissant.json` is Croissant 1.0 metadata built from it. ### 4b. Data analysis (`analysis/`) `analysis/stats/site_stats.py` reads runs_clean (sources in `measures.yaml`: **spreadsheet only** for now) and writes `data/public/stats.json`: every number the page shows, for six fixed filter presets (all, form only, bulk only, not chasing Blue/Black, chasing, unflagged). Factor tables (Fisher/Mantel–Haenszel, BH q-values), ridge-logistic models, route rankings at thresholds 5/10/20/30, per-stage route map counts. The page computes nothing itself. `analysis/network/build-graph.js` writes `graph.json` (route graph per source + the bot's scene graph). ### 4c. Route planner (`planner/` + `core/atlas/`) - **Model**: each roll (first dragon, survive sighting/fight/hour, shadow, shadow colour, meadow dragon) is a categorical with a **hierarchical Dirichlet back-off** over its context (earlier choices); the shrinkage strength (alpha) is fitted by empirical Bayes. Bond outcomes per encounter likewise. Malek/deliberate burns are not counted as failed bonds. - **Learns from** (`planner/policy.yaml`): reddit spreadsheet + reported + **bot** runs. - **Reporting-bias correction** (`core/atlas/bias.py`): form submitters over-report rare sightings. For each rare roll and each group, rho = odds(rare | group) / odds(rare | complete logs), where complete logs = **bot + bulk logs**. Rare outcomes from biased groups count 1/rho. Current estimates (meadow Blue): **form, chasing Blue/Black rho 3.85 [2.98–5.14]; form, other 3.17 [2.33–4.41]**. For the shadow, complete logs have **0 shadows in 579 gated tree runs**, so the bias can't be estimated and the shadow is learned from complete logs only. A roll seen only one way is pooled fully (alpha max). Caveats: assumes the bias is the same on every route; intervals don't include rho's uncertainty. - **Solver** (`core/atlas/solver.py`): walks game.yaml as a decision tree; two goals per colour: "this attempt" (maximise P(colour)) and "eventually" (Dinkelbach ratio: P(X) / (P(X)+P(other bond)), i.e. dying on purpose otherwise). 200 posterior draws give intervals and **stillBest** (share of draws where the recommended choice is best). Two samples: **all** and **lowBias** (bot + bulk + riders not chasing rare dragons). - **Output**: `data/public/policy.json` (game, count tables, alphas, reportingBias, support, trees per colour). - **JS advisor** (`core/solver.js`): reads policy.json, re-solves, `advisor(sample, colour, objective, opts)`, `recommendOption()` maps on-screen text to options (game.yaml `bot_text`). **Thompson sampling**: `{ sampling: 'thompson', seed }` follows one posterior draw per attempt and returns a **propensity**. Current odds per attempt (sample "all", latest build `538bc1e3d694`): | Colour | Per attempt (90% range) | Low-bias sample | stillBest of first choice | |---|---|---|---| | Red | 93.1% (86.9–97.4) | 94.1% | 0.30 | | Green | 25.2% (21.7–28.8) | 24.2% | 0.42 | | Orange | 13.5% (11.3–15.7) | 16.7% | 0.55 | | Brown | 9.3% (7.7–11.2) | 10.7% | 0.52 | | Blue | 2.5% (2.1–3.0) | 1.9% | 0.45 | | Black | 0.05% (0.00–0.25) | 0.04% | 0.31 | (Before the bot data and bias correction, the model said Blue 7.3% and Black 2.1%: inflated by reporting bias.) ### 4d. Data visualisation (`site/`) A single static page (tabs: Valley map, Odds, Dragons, Deep dive, Data & Methods, **Report a run**). The **valley map** (`explorer.js`) uses progressive disclosure: key moments only, a point's options appear when selected, picking a dragon shows its best path and choices; later choices keep the path you picked. The page shows precomputed `stats.json` and runs the JS advisor on `policy.json`. Links come from `data/public/links.json` (filled in by `site/build.js`). Build: `node site/build.js` → `dist/`. QA: `site/qa/crawl.js` (needs a local server on :8765), `site/qa/mapcheck.js`. Published so far as the Claude artifact above. ## 5. The bot loop (Ken) - **Atlas → bot**: `python3 -m collection export-bot-kit` → `exports/atlas-for-bot--.zip` (policy.json, solver.js, graph.js, README with checksums). Ken unzips into the bot's `vendor/threshing-atlas/`. Latest: `atlas-for-bot-2026-11-12-538bc1e3d694.zip`. - **Bot → atlas**: Ken zips the bot's `data/` and `logs/` folders; save in `inbox/` and run `python3 -m collection refresh --bot inbox/`. The importer reads `data/knowledge.json` and every `data/accounts//runs.jsonl` + `logs/accounts//encounters.jsonl` (top-level files mirror the first account and are skipped). Bot **schema-2** run records carry steps, what was seen (shadow, dragons), propensity, decisionMode, policyId. No game prose or account names are published (`check_public` refuses them). - **How the bot should choose** (docs/BOT_INTEGRATION.md): goal accounts follow the best route (`advisor(policy.samples.all, target, 'eventually')`); **research accounts use Thompson sampling** with a new seed per attempt and log `propensity` with `decisionMode: "atlas_thompson"`. - Ken's data so far: 13 accounts, 415 attempts (to 2026-11-10); research accounts burn on purpose when no shadow at the tree / no Blue in the meadow (`TREE_NO_SHADOW_AUTO_BURN`, `MEADOW_NO_BLUE_AUTO_BURN`). ## 6. Findings so far - **Reporting bias is large.** Meadow Blue: complete logs ~10% (bot 16/126, bulk 30/328) vs form ~33%. - **Black is very rare.** 0 shadows in 579 gated tree runs from complete logs; form data show ~2% (inflated). Model: ~0.05% per attempt. Fits Smart_Trifle_2099's guess of 1/500–1/1000 tree runs for a Black sighting. - **Route doesn't seem to move the rare rolls.** Shadow rate doesn't differ across routes (chi² 44 on 40 df); the "precise Blue pathways" (Smart_Trifle_2099 and the megathread post's ~47%/~63% routes) did no better than other meadow routes (23–30% vs 30% in the spreadsheet). Reaching the meadow seems to be the "unlock". - **The gate is real**: the shadow needs investigate + help. - **Tree Red is the hardest bond**: 1.7% (38/2,207), matching the post's "2%". - **First attempts** see the meadow Blue less often (16% vs 31%), surviving adjustment in the site's model. - Weekly swings in rates (and "the game was patched" claims) are confounded by who reports when. ## 7. How to run things (on the Mac, PyCharm terminal, venv active) ``` python3 -m collection refresh # the daily update locally (~3 min): sources → runs → graph* → model → stats → build python3 -m collection refresh --bot inbox/x.zip # same, importing a bot zip python3 -m collection update --source NAME # one source; --file NAME=PATH to use a local file python3 -m collection build [--check] # rebuild runs_clean + generated files (CI uses --check) python3 -m collection export-bot-kit # zip for Ken python3 planner/build_policy.py # route model (~30–75 s); python3 planner/policy_report.py python3 analysis/stats/site_stats.py # page statistics (needs scipy) node analysis/network/build-graph.js # graph.json (*needs Node) node site/build.js # the page → dist/ npm test # = build --check + Python unit tests + JS tests ``` Reports land in `exports/last-report.md`. If a refresh stops with "values not in the dictionary", add the new form answer to `core/spec/form.yaml` (or the value to `data/dictionary.yaml`) and run `python3 -m collection build`. ## 8. Conventions and constraints (keep these) - **Python first; declarative specs** (YAML for the form, game, pipeline, measures, planner, bot crosswalk, report form; dictionary + Croissant). JS only for the browser page and Ken's Node bot (`solver.js`, `graph.js`). - **Generated files are generated**: don't hand-edit `runs_clean.csv`, `croissant.json`, `links.json`, the crosswalk block in `core/graph.js`, `.github/ISSUE_TEMPLATE/report-a-run.yml`. Edit the spec, then build. - **Privacy**: no game prose, no account names/ids, no emails (redacted), no rider names (register), no Reddit or GitHub usernames; the route PDF is never redistributed (fingerprint only). Never commit `data/raw/`, `.env`, `.browser-profile/`, salts or tokens; tokens only in repository secrets. Don't scrape Discord with a personal token; Reddit only via the official API. - **Don't write to Ken's bot repo** (`~/Desktop/DragonKind`) without asking. - **Commits** as `ekcSchiano` / Kathleen's email, ending with `Co-Authored-By: Claude …` and `Claude-Session: …` lines. On GitHub, main was protected (PR + required `test` check, auto-merge). **Go easy on automated API activity** (that's the likely cause of the GitHub suspension). - Kathleen likes: progressive disclosure on the map, honest uncertainty, explaining what changed and why. ## 9. Status and open threads - **Uncommitted / pending**: nothing important; Mac repo is clean at `4837332` (plus `.idea/`, `exports/`, `inbox/` ignored). The artifact page is older than the data; rebuild with `node site/build.js` (needs Node) and republish. - **GitHub**: suspended; reinstatement in progress. When back: `git push` to a branch + PR (main is protected); GitHub Pages resumes from `.github/workflows/pages.yml`; set `forge.yaml` `kind: github` (or keep Gitea + mirror). - **Gitea**: ready but on hold (Ken's server has a problem). Steps in `docs/REPO_SETTINGS.md`; set your username in `forge.yaml`; the `.gitea/workflows/` are untested (runner label assumed `ubuntu-latest`). - **"Report a run"** button points at a Gitea URL that doesn't exist yet; the Google Form is the working intake. - **Not done yet / ideas**: - Include bot and reported runs in the page's statistics (currently spreadsheet only; bot choices follow a strategy). - Propagate the bias estimate's uncertainty (rho) into the route model's intervals; consider a per-source reporting-bias model inside the posterior draws. - Inverse-propensity weighting of bot runs using logged propensities. - Port `build-graph.js` to Python; Reddit extractor live once API secrets exist; "Community claims, checked" section (claims YAML recomputed against data); a "random run?" question on the report form. - Reviews still pending from the original plan: analysis, planner (experiments.json, encounter answers), site. - **Environment gotchas** (for Claude sessions): the cloud workspace and the Mac's Linux helper can't reach Google, Gitea or PyPI; Kathleen's own Terminal can. The helper lacks `scipy`. Git in the helper needs delete permission for its lock files.