Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+13 -1
View File
@@ -39,6 +39,18 @@ share.
small JS advisor (`core/solver.js`, about 250 lines) reads the game and the Python-built model from `policy.json`
and replaces the 900-line JS solver; `engine.js` is gone.
5. **Done.** `data/dictionary.yaml` describes every published file and every column of the coded runs (type, meaning,
allowed values); `collection/publish_metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
allowed values); `collection/etl/metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
sha256 for every file. `tests/core/test_dictionary.py` fails when a column or value is undescribed or the metadata
is stale. It also caught `attempt` being written as `1.0`, `2.0`: now a whole number.
## Data collection review (November 2026)
Collection is an extract → transform → load pipeline declared in `collection/pipeline.yaml` (`python -m collection`).
`data/public/runs_clean.csv` holds every run from every source with a `source` column (reddit spreadsheet, reddit
thread, reported, bot, reddit pdf). The bot importer is Python (`collection/etl/transform/bot.py`, identical output to
the old `bot.js`), and its crosswalk is `collection/specs/bot.yaml`, written into `core/graph.js` for the bot.
Reddit goes through the official API; "Report a run" is a GitHub issue form generated from
`collection/specs/report_form.yaml`. Only the previous raw version is kept, encrypted, in the Actions cache.
Still to do: decide which non-spreadsheet sources join the statistics and the route model (`analysis/stats/measures.yaml`
`sources`); the bot follows a strategy, so its runs need weighting rather than pooling.