Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
@@ -125,7 +125,7 @@ rescue branch (the wounded cadet) to be tried in both directions in phase 1.
|
||||
|
||||
## 4. Getting data to the atlas
|
||||
|
||||
- The atlas repo already has the exporter: `npm run import-bot -- --bot /path/to/DragonKind`. It reads
|
||||
- The atlas repo already has the exporter: `DRAGONKIND_BOT_DIR=/path/to/DragonKind python -m collection update --source bot`. It reads
|
||||
`data/knowledge.json`, `data/runs.jsonl` and `logs/encounters.jsonl` from the bot folder and writes
|
||||
`data/bot/scenes.json` and `data/bot/trajectories-<date>.jsonl`, with:
|
||||
- prose removed (scene fingerprints, short labels, choice text and outcomes only). The importer
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
Data flows both ways:
|
||||
|
||||
```
|
||||
community form, register, PDF ──┐
|
||||
community sources, reports ─────┐
|
||||
├─► threshing-atlas ──► policy.json + solver.js ──► bot (advice per screen)
|
||||
bot runs (anonymised) ──────────┘ ▲ │
|
||||
└──────────── import-bot + pull request ◄──┘
|
||||
└──────────── collection --source bot + PR ◄┘
|
||||
```
|
||||
|
||||
## 1. From the atlas to the bot
|
||||
@@ -129,7 +129,7 @@ Current headline (from `analysis/reports/POLICY.md`): Black about 1.7% per attem
|
||||
|
||||
## 2. From the bot to the atlas
|
||||
|
||||
Already in place: `npm run import-bot -- --bot /path/to/DragonKind` in the atlas repo converts the bot's
|
||||
Already in place: `DRAGONKIND_BOT_DIR=/path/to/DragonKind python -m collection update --source bot` in the atlas repo converts the bot's
|
||||
`knowledge.json`, `runs.jsonl` and `logs/encounters.jsonl` into anonymised files under `data/bot/`, and refuses to
|
||||
write prose or account names. Commit them by pull request; CI rebuilds the graph and checks everything.
|
||||
|
||||
@@ -147,4 +147,4 @@ What would make the bot's data far more useful (details in `docs/BOT_DATA_PROTOC
|
||||
- The model learns how much history matters from the data. Today it finds that the meadow colour does not depend on
|
||||
earlier choices at all, and the shadow only weakly. The bot's randomised runs are what can confirm or overturn that.
|
||||
- The policy is only as good as the crosswalk between the bot's screen text and the form's variables. If the game
|
||||
changes its wording, `OPTION_PATTERNS` in `solver.js` and `CROSSWALK` in `graph.js` need updating.
|
||||
changes its wording, `OPTION_PATTERNS` in `solver.js` and the crosswalk in `collection/specs/bot.yaml` (written into `graph.js` by `python -m collection build`) need updating.
|
||||
|
||||
+14
-4
@@ -13,7 +13,7 @@ Rules every part of the project follows. If a change would break one, stop and a
|
||||
- **The community route PDF:** never committed or redistributed. Only a fingerprint
|
||||
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
|
||||
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
|
||||
names. The importer (`collection/bot/bot.js`) refuses to write anything that looks like either.
|
||||
names. The importer (`collection/etl/transform/bot.py`, rules in `collection/pipeline.yaml`) refuses to write anything that looks like either.
|
||||
|
||||
## The bot
|
||||
|
||||
@@ -25,10 +25,20 @@ Rules every part of the project follows. If a change would break one, stop and a
|
||||
## Sources we do not scrape
|
||||
|
||||
- **Discord:** not without the server admins' consent; never with a personal account token.
|
||||
- **Reddit:** not collected yet. When it is, it should go through Reddit's official API with an app key, not page
|
||||
scraping.
|
||||
- **Reddit:** only through Reddit's official API with an app key (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`),
|
||||
never by page scraping. Only dragonkind.com result links are taken from the threads; usernames and comment text
|
||||
are never kept, and each result page is read and added by hand (`collection/manual/reddit_results.yaml`).
|
||||
|
||||
## Run reports
|
||||
|
||||
"Report a run" creates a public GitHub issue. The pipeline keeps only the issue number, the date and the answers
|
||||
(`data/public/reports.json`); GitHub usernames are dropped. The form asks people not to include account names. A
|
||||
report closed as "not planned" or labelled `invalid` or `duplicate` is left out of the data.
|
||||
|
||||
## Secrets
|
||||
|
||||
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
|
||||
(`docs/REPO_SETTINGS.md`). Tokens go in secrets only: never in a file, issue, chat or commit.
|
||||
(`docs/REPO_SETTINGS.md`).
|
||||
- `RAW_ARCHIVE_KEY` encrypts the raw store (the current and previous raw file of each source, which include the
|
||||
register's rider names) in the Actions cache between daily runs. Without it nothing is archived.
|
||||
- `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` let the daily update read the megathreads through the Reddit API. Tokens go in secrets only: never in a file, issue, chat or commit.
|
||||
|
||||
@@ -59,3 +59,15 @@ merge.
|
||||
If Ken's bot should also open pull requests here (for example to send new bot runs), create a **second** token
|
||||
with the same settings and store it as a secret in the bot's own repository. Separate tokens can be revoked
|
||||
separately, and a leak in one place does not expose the other.
|
||||
|
||||
## Data collection secrets (optional)
|
||||
|
||||
Both are repository secrets (Settings → Secrets and variables → Actions → New repository secret). Without them the
|
||||
daily update still runs and says in its report what it skipped.
|
||||
|
||||
| Secret | What it is for | How to make it |
|
||||
|---|---|---|
|
||||
| `RAW_ARCHIVE_KEY` | Encrypts the private raw store (current and previous raw file of each source) kept in the Actions cache between runs. | Any long random string, e.g. the output of `openssl rand -base64 32`. Keep a copy somewhere safe: without it the archive cannot be read. |
|
||||
| `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET` | Reading the r/fourthwing megathreads through the official Reddit API. | reddit.com/prefs/apps → "create another app…" → type **script**, any name, redirect uri `http://localhost`. The id is under the app name; the secret is labelled "secret". |
|
||||
|
||||
The "Report a run" form adds the label `run report` to each report; the label must exist (Issues → Labels).
|
||||
|
||||
+13
-1
@@ -39,6 +39,18 @@ share.
|
||||
small JS advisor (`core/solver.js`, about 250 lines) reads the game and the Python-built model from `policy.json`
|
||||
and replaces the 900-line JS solver; `engine.js` is gone.
|
||||
5. **Done.** `data/dictionary.yaml` describes every published file and every column of the coded runs (type, meaning,
|
||||
allowed values); `collection/publish_metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
|
||||
allowed values); `collection/etl/metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
|
||||
sha256 for every file. `tests/core/test_dictionary.py` fails when a column or value is undescribed or the metadata
|
||||
is stale. It also caught `attempt` being written as `1.0`, `2.0`: now a whole number.
|
||||
|
||||
## Data collection review (November 2026)
|
||||
|
||||
Collection is an extract → transform → load pipeline declared in `collection/pipeline.yaml` (`python -m collection`).
|
||||
`data/public/runs_clean.csv` holds every run from every source with a `source` column (reddit spreadsheet, reddit
|
||||
thread, reported, bot, reddit pdf). The bot importer is Python (`collection/etl/transform/bot.py`, identical output to
|
||||
the old `bot.js`), and its crosswalk is `collection/specs/bot.yaml`, written into `core/graph.js` for the bot.
|
||||
Reddit goes through the official API; "Report a run" is a GitHub issue form generated from
|
||||
`collection/specs/report_form.yaml`. Only the previous raw version is kept, encrypted, in the Actions cache.
|
||||
|
||||
Still to do: decide which non-spreadsheet sources join the statistics and the route model (`analysis/stats/measures.yaml`
|
||||
`sources`); the bot follows a strategy, so its runs need weighting rather than pooling.
|
||||
|
||||
Reference in New Issue
Block a user