- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
4.4 KiB
Roadmap
The aim is to move from summary statistics to path finding: for each dragon colour, the decision policy that maximises the chance of bonding it, with honest uncertainty.
- Repo and packaging (done): monorepo, CI, GitHub Pages, JS/Python parity tests.
- Graph format (
core/, first version done): typed scene, chance and terminal nodes; history-context keys; per-source counts (bot, form, PDF, Discord); fingerprints and short labels only, no game prose. Importers incollection/botfor the bot'sknowledge.json,runs.jsonlandencounters.jsonl(done), the form (done, via the crosswalk ingraph.js), and a PDF scene crosswalk (to do). Daily source checks for the form, the register and the PDF (done). - Context learning (
analysis/): tests of history dependence (G-tests, BH); variable-order Markov / context tree with BIC; temporal-tiered DAG structure learning (BDeu/BIC); latent "character" model vs specific triggers vs memoryless baseline, compared on held-out likelihood; hierarchical Dirichlet back-off pooling. - Solver (first version done:
core/solver.js,analysis/reports/POLICY.md; still to do: richer history models from step 3): backward induction on the augmented state (scene, history summary), i.e. an influence diagram with perfect recall. Two objectives side by side per colour:- P(target colour on this attempt);
- eventual success when deliberate deaths are free: maximise p_X / (p_X + p_other bond), solved by Dinkelbach iteration. Monte Carlo credible intervals, top-k routes, required steps and bottlenecks.
- Valley graph UI: every option at every point, with P(survive) and P(bond) per colour, the policy overlay for a chosen colour, and untested branches marked.
- Bot loop (advisor and sync published, see
docs/BOT_INTEGRATION.md; experiments.json to do): the atlas publishespolicy.jsonandexperiments.json(ranked by value of information); the private DragonKind bot consumes them and returns anonymised runs.
Python first, declarative rules (November 2026)
The repo is moving to Python, with the rules and the game written as declarative spec files. JavaScript stays only where it has to run: the site's drawing and interaction, and a small route advisor the site and Ken's bot share.
- Coding rules:
core/spec/form.yaml+core/atlas/coding.py, with rule tests. Done. - Route model in Python, built from
core/spec/game.yaml. Done (the bot's text,core/spec/crosswalk.yaml, moves with the bot importer and the JS advisor in step 4). - Done. Fix the coding bugs found in the core review (letting the Red pass, "left your squadmates" before the redesigned form, missing bot text), each with a rule test.
- Done. The site shows precomputed statistics with fixed filter presets (
analysis/stats/site_stats.py,measures.yaml,data/public/stats.json); the upload button and the click-to-select route map are gone; a small JS advisor (core/solver.js, about 250 lines) reads the game and the Python-built model frompolicy.jsonand replaces the 900-line JS solver;engine.jsis gone. - Done.
data/dictionary.yamldescribes every published file and every column of the coded runs (type, meaning, allowed values);collection/etl/metadata.pywritesdata/croissant.json(Croissant 1.0) from it, with a sha256 for every file.tests/core/test_dictionary.pyfails when a column or value is undescribed or the metadata is stale. It also caughtattemptbeing written as1.0,2.0: now a whole number.
Data collection review (November 2026)
Collection is an extract → transform → load pipeline declared in collection/pipeline.yaml (python -m collection).
data/public/runs_clean.csv holds every run from every source with a source column (reddit spreadsheet, reddit
thread, reported, bot, reddit pdf). The bot importer is Python (collection/etl/transform/bot.py, identical output to
the old bot.js), and its crosswalk is collection/specs/bot.yaml, written into core/graph.js for the bot.
Reddit goes through the official API; "Report a run" is a GitHub issue form generated from
collection/specs/report_form.yaml. Only the previous raw version is kept, encrypted, in the Actions cache.
Still to do: decide which non-spreadsheet sources join the statistics and the route model (analysis/stats/measures.yaml
sources); the bot follows a strategy, so its runs need weighting rather than pooling.