Files
ekcSchianoandClaude Opus 5.5 3b24a11f5a Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-04 02:46:48 +00:00

4.4 KiB

Roadmap

The aim is to move from summary statistics to path finding: for each dragon colour, the decision policy that maximises the chance of bonding it, with honest uncertainty.

  1. Repo and packaging (done): monorepo, CI, GitHub Pages, JS/Python parity tests.
  2. Graph format (core/, first version done): typed scene, chance and terminal nodes; history-context keys; per-source counts (bot, form, PDF, Discord); fingerprints and short labels only, no game prose. Importers in collection/bot for the bot's knowledge.json, runs.jsonl and encounters.jsonl (done), the form (done, via the crosswalk in graph.js), and a PDF scene crosswalk (to do). Daily source checks for the form, the register and the PDF (done).
  3. Context learning (analysis/): tests of history dependence (G-tests, BH); variable-order Markov / context tree with BIC; temporal-tiered DAG structure learning (BDeu/BIC); latent "character" model vs specific triggers vs memoryless baseline, compared on held-out likelihood; hierarchical Dirichlet back-off pooling.
  4. Solver (first version done: core/solver.js, analysis/reports/POLICY.md; still to do: richer history models from step 3): backward induction on the augmented state (scene, history summary), i.e. an influence diagram with perfect recall. Two objectives side by side per colour:
    • P(target colour on this attempt);
    • eventual success when deliberate deaths are free: maximise p_X / (p_X + p_other bond), solved by Dinkelbach iteration. Monte Carlo credible intervals, top-k routes, required steps and bottlenecks.
  5. Valley graph UI: every option at every point, with P(survive) and P(bond) per colour, the policy overlay for a chosen colour, and untested branches marked.
  6. Bot loop (advisor and sync published, see docs/BOT_INTEGRATION.md; experiments.json to do): the atlas publishes policy.json and experiments.json (ranked by value of information); the private DragonKind bot consumes them and returns anonymised runs.

Python first, declarative rules (November 2026)

The repo is moving to Python, with the rules and the game written as declarative spec files. JavaScript stays only where it has to run: the site's drawing and interaction, and a small route advisor the site and Ken's bot share.

  1. Coding rules: core/spec/form.yaml + core/atlas/coding.py, with rule tests. Done.
  2. Route model in Python, built from core/spec/game.yaml. Done (the bot's text, core/spec/crosswalk.yaml, moves with the bot importer and the JS advisor in step 4).
  3. Done. Fix the coding bugs found in the core review (letting the Red pass, "left your squadmates" before the redesigned form, missing bot text), each with a rule test.
  4. Done. The site shows precomputed statistics with fixed filter presets (analysis/stats/site_stats.py, measures.yaml, data/public/stats.json); the upload button and the click-to-select route map are gone; a small JS advisor (core/solver.js, about 250 lines) reads the game and the Python-built model from policy.json and replaces the 900-line JS solver; engine.js is gone.
  5. Done. data/dictionary.yaml describes every published file and every column of the coded runs (type, meaning, allowed values); collection/etl/metadata.py writes data/croissant.json (Croissant 1.0) from it, with a sha256 for every file. tests/core/test_dictionary.py fails when a column or value is undescribed or the metadata is stale. It also caught attempt being written as 1.0, 2.0: now a whole number.

Data collection review (November 2026)

Collection is an extract → transform → load pipeline declared in collection/pipeline.yaml (python -m collection). data/public/runs_clean.csv holds every run from every source with a source column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf). The bot importer is Python (collection/etl/transform/bot.py, identical output to the old bot.js), and its crosswalk is collection/specs/bot.yaml, written into core/graph.js for the bot. Reddit goes through the official API; "Report a run" is a GitHub issue form generated from collection/specs/report_form.yaml. Only the previous raw version is kept, encrypted, in the Actions cache.

Still to do: decide which non-spreadsheet sources join the statistics and the route model (analysis/stats/measures.yaml sources); the bot follows a strategy, so its runs need weighting rather than pooling.