Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+1 -1
View File
@@ -125,7 +125,7 @@ rescue branch (the wounded cadet) to be tried in both directions in phase 1.
## 4. Getting data to the atlas
- The atlas repo already has the exporter: `npm run import-bot -- --bot /path/to/DragonKind`. It reads
- The atlas repo already has the exporter: `DRAGONKIND_BOT_DIR=/path/to/DragonKind python -m collection update --source bot`. It reads
`data/knowledge.json`, `data/runs.jsonl` and `logs/encounters.jsonl` from the bot folder and writes
`data/bot/scenes.json` and `data/bot/trajectories-<date>.jsonl`, with:
- prose removed (scene fingerprints, short labels, choice text and outcomes only). The importer
+4 -4
View File
@@ -3,10 +3,10 @@
Data flows both ways:
```
community form, register, PDF ──┐
community sources, reports ─────┐
├─► threshing-atlas ──► policy.json + solver.js ──► bot (advice per screen)
bot runs (anonymised) ──────────┘ ▲ │
└──────────── import-bot + pull request ◄──┘
└──────────── collection --source bot + PR ◄┘
```
## 1. From the atlas to the bot
@@ -129,7 +129,7 @@ Current headline (from `analysis/reports/POLICY.md`): Black about 1.7% per attem
## 2. From the bot to the atlas
Already in place: `npm run import-bot -- --bot /path/to/DragonKind` in the atlas repo converts the bot's
Already in place: `DRAGONKIND_BOT_DIR=/path/to/DragonKind python -m collection update --source bot` in the atlas repo converts the bot's
`knowledge.json`, `runs.jsonl` and `logs/encounters.jsonl` into anonymised files under `data/bot/`, and refuses to
write prose or account names. Commit them by pull request; CI rebuilds the graph and checks everything.
@@ -147,4 +147,4 @@ What would make the bot's data far more useful (details in `docs/BOT_DATA_PROTOC
- The model learns how much history matters from the data. Today it finds that the meadow colour does not depend on
earlier choices at all, and the shadow only weakly. The bot's randomised runs are what can confirm or overturn that.
- The policy is only as good as the crosswalk between the bot's screen text and the form's variables. If the game
changes its wording, `OPTION_PATTERNS` in `solver.js` and `CROSSWALK` in `graph.js` need updating.
changes its wording, `OPTION_PATTERNS` in `solver.js` and the crosswalk in `collection/specs/bot.yaml` (written into `graph.js` by `python -m collection build`) need updating.
+14 -4
View File
@@ -13,7 +13,7 @@ Rules every part of the project follows. If a change would break one, stop and a
- **The community route PDF:** never committed or redistributed. Only a fingerprint
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
names. The importer (`collection/bot/bot.js`) refuses to write anything that looks like either.
names. The importer (`collection/etl/transform/bot.py`, rules in `collection/pipeline.yaml`) refuses to write anything that looks like either.
## The bot
@@ -25,10 +25,20 @@ Rules every part of the project follows. If a change would break one, stop and a
## Sources we do not scrape
- **Discord:** not without the server admins' consent; never with a personal account token.
- **Reddit:** not collected yet. When it is, it should go through Reddit's official API with an app key, not page
scraping.
- **Reddit:** only through Reddit's official API with an app key (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`),
never by page scraping. Only dragonkind.com result links are taken from the threads; usernames and comment text
are never kept, and each result page is read and added by hand (`collection/manual/reddit_results.yaml`).
## Run reports
"Report a run" creates a public GitHub issue. The pipeline keeps only the issue number, the date and the answers
(`data/public/reports.json`); GitHub usernames are dropped. The form asks people not to include account names. A
report closed as "not planned" or labelled `invalid` or `duplicate` is left out of the data.
## Secrets
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
(`docs/REPO_SETTINGS.md`). Tokens go in secrets only: never in a file, issue, chat or commit.
(`docs/REPO_SETTINGS.md`).
- `RAW_ARCHIVE_KEY` encrypts the raw store (the current and previous raw file of each source, which include the
register's rider names) in the Actions cache between daily runs. Without it nothing is archived.
- `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` let the daily update read the megathreads through the Reddit API. Tokens go in secrets only: never in a file, issue, chat or commit.
+12
View File
@@ -59,3 +59,15 @@ merge.
If Ken's bot should also open pull requests here (for example to send new bot runs), create a **second** token
with the same settings and store it as a secret in the bot's own repository. Separate tokens can be revoked
separately, and a leak in one place does not expose the other.
## Data collection secrets (optional)
Both are repository secrets (Settings → Secrets and variables → Actions → New repository secret). Without them the
daily update still runs and says in its report what it skipped.
| Secret | What it is for | How to make it |
|---|---|---|
| `RAW_ARCHIVE_KEY` | Encrypts the private raw store (current and previous raw file of each source) kept in the Actions cache between runs. | Any long random string, e.g. the output of `openssl rand -base64 32`. Keep a copy somewhere safe: without it the archive cannot be read. |
| `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET` | Reading the r/fourthwing megathreads through the official Reddit API. | reddit.com/prefs/apps → "create another app…" → type **script**, any name, redirect uri `http://localhost`. The id is under the app name; the secret is labelled "secret". |
The "Report a run" form adds the label `run report` to each report; the label must exist (Issues → Labels).
+13 -1
View File
@@ -39,6 +39,18 @@ share.
small JS advisor (`core/solver.js`, about 250 lines) reads the game and the Python-built model from `policy.json`
and replaces the 900-line JS solver; `engine.js` is gone.
5. **Done.** `data/dictionary.yaml` describes every published file and every column of the coded runs (type, meaning,
allowed values); `collection/publish_metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
allowed values); `collection/etl/metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
sha256 for every file. `tests/core/test_dictionary.py` fails when a column or value is undescribed or the metadata
is stale. It also caught `attempt` being written as `1.0`, `2.0`: now a whole number.
## Data collection review (November 2026)
Collection is an extract → transform → load pipeline declared in `collection/pipeline.yaml` (`python -m collection`).
`data/public/runs_clean.csv` holds every run from every source with a `source` column (reddit spreadsheet, reddit
thread, reported, bot, reddit pdf). The bot importer is Python (`collection/etl/transform/bot.py`, identical output to
the old `bot.js`), and its crosswalk is `collection/specs/bot.yaml`, written into `core/graph.js` for the bot.
Reddit goes through the official API; "Report a run" is a GitHub issue form generated from
`collection/specs/report_form.yaml`. Only the previous raw version is kept, encrypted, in the Actions cache.
Still to do: decide which non-spreadsheet sources join the statistics and the route model (`analysis/stats/measures.yaml`
`sources`); the bot follows a strategy, so its runs need weighting rather than pooling.