- collection/etl/runs.py fingerprints(): a form or register run's id is now <prefix>-<hash of the row's non-empty
answers, timestamp included> instead of its row number, so deleting or re-sorting rows in the sheet moves no
other run's id, and a column added later changes none. The 32 rows that are exact copies of another row get
-2, -3 (identical, so which is which does not matter). A row edited in the sheet gets a new id. Every id in
runs_clean.csv and overlap.json changes once; nothing else does (no statistic or model reads the ids).
- tests/core/test_site_stats.py (9 tests): measures.yaml conditions, source groups and presets, bond
denominators without deliberate deaths (cards, encounters, route map), the riders' paths, and a JSON-safe
stats.json, on a fixed sample of the published runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- collection/etl/review_reddit.py (`python -m collection review-reddit`) and .gitea/workflows/reddit-review.yml:
on every scraper pull request, one comment (refreshed on each push) listing the runs added, changed and gone, the
runs that break a game rule (left out of the model and stats until corrected), other flags, runs that may also
be in the form, and result pages already counted; plus how to reject / fix / add. forge.upsert_comment keeps it
to one comment. collection/README.md: "Reviewing Reddit runs".
- analysis/overlap/overlap.py -> data/public/overlap.json: a Reddit run with at least 4 route answers "matches" a
form row giving the same answers within a day; chance is the same test a few weeks off. 9 of 36 comparable runs
match, ~7.3 by chance: about 2 of 82 look reported twice, too few to matter. Nothing is removed. Shown on the
page's Data & methods tab; in the dataset metadata; checked in test:py; part of `rebuild`.
- Neither the form nor the register records a result-page id, so the only proven copy (comment <-> result page)
stays the one #18 applies.
- data/public/stats.json: the statistics of #42 (impossible runs left out), which did not make it into that merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- core/spec/form.yaml impossible_flags: the flags that mean a run cannot have happened as recorded (a dragon an
encounter does not have, a Green or the Black in the valley, two bonds, two burns, a cadet path that does not
follow, an early death that still reached the hour...). atlas.runs.impossible() reads it.
- The route model and the site statistics (and the riders' paths) leave those runs out, from every source; every
other run is used, however partial. The runs stay in runs_clean.csv, flagged. 80 runs: 75 form, 5 Reddit.
Route-model odds move by at most 0.2 points.
- collection/etl/runs.py: the community form gets the same game-rule checks as "Report a run" and the Reddit
comments (10 form rows newly flagged); new rule: the Black in the valley.
- The first-sighting pool stays a flag only: it is what riders have seen, not a rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- planner/policy.yaml: `reddit comment` is a model source. Only runs that break a game rule (the rule_flags
checks) are left out; partial runs are kept. Its own reporting-bias group, and out of the low-bias sample
(riders posting a run pick the notable ones). 77 of the 82 runs are used.
- analysis/stats/measures.yaml: the site statistics use every source answered in the form's terms (form,
"Report a run", Reddit comments).
- analysis/black/evidence.py: a shadow run is "off the gates" only when an answer contradicts a gate (3); runs
that do not say are counted separately (14).
- croissant isBasedOn lists both megathreads (pipeline.yaml `cite`).
- Rebuilt: policy.json, stats.json, BLACK.md, black.json, croissant.json.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
Reported and Reddit-comment runs now get the checks the community form
gets in core/atlas/coding.py, as data_flags (the run is kept):
- first dragon outside the documented pool for the company/direction
- a bond or burn by a dragon the encounter does not have (new
encounter_colours table in core/spec/form.yaml; Orange/Brown after the
two Greens allowed, as a Green's rescue)
- a Green on a valley run
- a cadet outcome off its scream path, fight answers or a post-fight path
on a run that did not help, a second fight move without assessing
- died before the hour, yet reached the tree/valley or an encounter
planner/policy.yaml drop_flagged: flagged runs of sources read from free
text (reddit comment) stay out of the route model; provenance notes do not
count as flags. No effect today: the source is not in `sources` yet.
On the first 82 scraped runs: 6 flagged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
analysis/known/build_known.py writes data/public/known_answers.json from pdf_encounters.json and tree.json;
solver.js knownAnswer(known, offered, { target, sceneColour }) matches a screen's choices against it (the guide's
paraphrases by their words, the bot's texts exactly) and returns a pick when one choice is clearly on the guide's
recommended path to the target, plus the evidence for every choice to hand to Clef. Rebuilt daily, checked in CI,
and sent to the bot with the route model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs
riders describe in the r/fourthwing megathreads and delivers them here by
pull request as data/public/reddit_comments.json: each run is the "Report a
run" answers (option labels of collection/specs/report_form.yaml), with the
comment id and day only; no usernames, no comment text.
- pipeline.yaml: a reddit_comments source of extract kind "delivered" (the
atlas fetches nothing; `update` skips it with a note)
- runs.py: the coder, sharing one answers_row() with the reported coder;
every row is flagged as read from a Reddit comment (and "not checked by
hand" until corrected); a result page a comment already carries is not
counted twice as a "reddit thread" row
- collection/manual/reddit_comments.yaml: reject / fix / add, applied at
build time
- dictionary, SOURCES, README, croissant: the new source
The route model and statistics do not use the source until it is named in
planner/policy.yaml / analysis/stats/measures.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daily check keeps the community route PDF only in its private raw store (never published here). When that
copy differs from "Threshing Options.pdf" in the bot's private repo, `python -m collection send-pdf` opens (or
updates) a pull request there with the new PDF and the lines its text gained and lost, so Ken can update the
hand-written routes in src/pdfRoutes.js (the bot does not read the PDF itself).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- update-sources: download DragonKind/DragonKind (read with ATLAS_BOT_TOKEN, or DRAGONKIND_TOKEN if set) and
import it as the bot source. Needs DK_ACCOUNT_SALT (the same salt as earlier imports, so accounts keep their keys);
without it the bot source is skipped as before.
- The bot load keeps one file, data/bot/trajectories-bot.jsonl: the bot's repo holds every account's full history,
so each import is merged by attempt id (attempts already published stay; older per-import files are folded in).
One copy of each attempt also keeps build-graph.js from counting an attempt twice.
- New workflow push-policy: when policy.json, solver.js or graph.js change on main, open (or update) a pull request
on DragonKind/DragonKind that replaces vendor/threshing-atlas/ with them. Ken merges it.
- forge open-pr: --repo/--base for another repository, and reuse the open pull request for the same branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- spreadsheet 8,132 -> 8,748 rows (about 600 new; the sheet's keeper fixed 'Sqaudmate' in 72 answers,
corrected 15 attempt numbers to 'First Attempt' and re-pasted the daily bulk logs); register 288 -> 292
- form.yaml squad_any_case: a typed 'second' is Second (it stopped the build)
- collection rebuild: the runs are built and checked first, so a failed check stops it before the model,
plan, reports and statistics are rebuilt from stale runs
- route model, plan, Black evidence, screen tree, statistics, graph and metadata rebuilt (9,473 runs)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- python -m collection rebuild: graph, route model, experiment plan, the Black evidence, the screen
tree, statistics, build. refresh, cycle and both update-sources workflows use it, so the morning
update also rebuilds the reports the tests check (it rebuilt only graph, model and stats before)
- graph.json had not been rebuilt since the played-steps fix (no Node on the Mac)
- docs: the branch is pushed; README mentions rebuild
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- analysis/tree/build_tree.py -> data/public/tree.json, analysis/reports/TREE.md: every screen the bot has seen,
every option, where each led (fixed or dice, with ranges), best known picks per colour chosen cautiously, where
any picks end, and the untested options. solver.js screenPick(): least-tried and Thompson picks from the tree.
- The bot logs a pick it then overrode as a step: 37 runs were coded with choices they never made. Importer and
graph.js now use the played steps only.
- runs_clean.csv: the Unofficial Dragon Register's 288 bonded dragons are in the master list as outcome-only
rows, flagged (no analysis uses them).
- Cycle, refresh and the bot kit rebuild and carry the tree; tests check every generated report is current.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
- planner/knowledge.yaml: what the game's developer has said (paraphrased, dated, graded), what the Atlas does
with each statement, and the open questions with the experiment that tests each.
- analysis/black/evidence.py -> analysis/reports/BLACK.md: counts from the runs. On the same recorded route the
shadow shows on 21 of 79 first attempts and 1 of 80 later ones; no single route; Blue then Black after the shadow.
- game.yaml: the shadow is a guard (not a roll); what follows it is the Blue, then the Black; on-screen text for
the squad's direction screen.
- planner/experiments.yaml v2: one exact path on every fresh account (fresh-replicate), then one choice changed
at a time (fresh-ablate); shadow-again, second-attempt-same-path, veteran-repeat, veteran-variety; what the bot
records about the account; the sequence to follow once the shadow shows.
- solver.js: assign() takes account facts, account offsets and cycled arms; recommendOption() answers the
squad screen (stay + direction) in one pick.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
- forge.yaml: the repository's host (Gitea at git.kenschiano.net, limited visibility);
the "Report a run" collector, the page's links (data/public/links.json, filled in by
site/build.js) and the dataset metadata read it
- collection/etl/forge.py: issues, pull requests (merge when checks succeed) and labels
for Gitea and GitHub; `python -m collection forge ...`
- .gitea/workflows: CI and the daily update (pushes a branch, opens a pull request that
merges itself, opens an issue when a source breaks); .github/workflows kept for GitHub
- docs: Gitea setup (REPO_SETTINGS), the bot reading atlas files from the repository
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- planner/policy.yaml: which sources the route model learns from (spreadsheet,
reported, bot) and the low-bias sample
- bot runs: deaths coded by stage, Malek intent for runs with a target
- model: survival of the first dragon is not counted when nothing after it is
known; the meadow dragon falls back to what the bot saw
- solver.js: advisor(..., {sampling: "thompson", seed, draws}) follows one
posterior draw per attempt and reports each pick's propensity
Not shipped yet: mixing sources lowers the fitted shrinkage for the shadow roll
(alpha 500 -> 50 in the low-bias sample), see the conversation of 2026-11-09.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- the importer reads data/accounts/*/runs.jsonl and logs/accounts/*/encounters.jsonl;
with account folders, the top-level files (a mirror of the first account) are skipped
- run records that carry their steps (schema 2) become trajectories directly, keeping
what the bot saw (shadow, meadow dragons), propensities, decision mode and policy id;
older runs are still rebuilt from the decision log; bonded runs without steps are kept
- account names become salted hashes (DK_ACCOUNT_SALT, or a private salt in data/raw)
- runs_clean.csv: new bot_mode column; saw_shadow and saw_blue_meadow from what the bot
saw; deliberate burns say why
- crosswalk: the valley Red, leaving squadmates, squad direction, hold your ground,
hide/approach reactions, the wounded cadet's other two choices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- python -m collection export-bot-kit: policy.json, solver.js, graph.js and a
README with checksums, zipped for the bot
- the bot import takes a folder or a zip (files may sit under any top folder)
- python -m collection refresh: the daily workflow, run locally
- trajectories files are named by date and raw checksum, so two imports on
one day never overwrite each other
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- collection/pipeline.yaml declares every source (extract kind, transform,
privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
`source` column (reddit spreadsheet, reddit thread, reported, bot,
reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
collection/specs/report_form.yaml, collected as source "reported", plus a
site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
measures.yaml); stats.json and policy.json unchanged apart from the
factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
the Actions cache, encrypted with RAW_ARCHIVE_KEY
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
- data/dictionary.yaml: every published file, and every column of runs_clean.csv
with its type, meaning and allowed values
- collection/publish_metadata.py writes data/croissant.json (Croissant 1.0) from
it, with a sha256 per file; `npm run metadata`, run by the daily update
- tests/core/test_dictionary.py: every column and value described, types hold,
croissant.json up to date and checksums match
- attempt is now written as a whole number (was 1.0, 2.0); no statistic changes
- the site ships both files and the Methods text links them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
Step 1 of moving the repo to Python with declarative rules.
- core/spec/form.yaml: every question (how it is found in an export), every answer and what it
is coded as, missing answers, email redaction, burn order, first-dragon pools.
- core/atlas/coding.py reads it and replaces core/python/clean.py (python -m atlas.coding).
- tests/core/test_coding.py: 16 rule tests, one hand-written row per rule, checking the rules
are right rather than only that JS and Python agree.
- analysis/stats/analyze.py takes the colours and missing answers from the spec instead of its
own copy; the daily update job codes rows through the package.
- pyproject.toml for the package; pyyaml and pytest added to requirements.
The coded runs are byte-identical to before (5,988 runs); all parity checks still match.
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Moves only; no behaviour changes. Every file now lives under the part of the project it
belongs to, each section has a README, and docs/PRIVACY.md collects the rules all parts follow.
core/ engine.js, graph.js, solver.js, python/clean.py
collection/ sources/ (form, register, PDF check, daily update), bot/ (importer)
analysis/ stats/ (Python statistics), network/ (build-graph.js), reports/ (POLICY.md)
planner/ build-policy.js, policy-report.js
site/ atlas/ (the page), build.js, qa/
tests/ core/ (unit), parity/ (JS vs Python), fixtures/
Checked against the previous layout: tests pass; coded runs, test outputs, graph.json and
policy.json are byte-identical; the built site differs only in comments that name file paths.
The published file names Ken's bot downloads (solver.js, graph.js, data/policy.json) are unchanged.
Also removes the undefined/ folder of QA screenshots committed by mistake (the QA scripts now
default to a temp folder) and test.txt, and drops the unused npm workspaces.
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>