Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+13
-1
@@ -39,6 +39,18 @@ share.
|
||||
small JS advisor (`core/solver.js`, about 250 lines) reads the game and the Python-built model from `policy.json`
|
||||
and replaces the 900-line JS solver; `engine.js` is gone.
|
||||
5. **Done.** `data/dictionary.yaml` describes every published file and every column of the coded runs (type, meaning,
|
||||
allowed values); `collection/publish_metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
|
||||
allowed values); `collection/etl/metadata.py` writes `data/croissant.json` (Croissant 1.0) from it, with a
|
||||
sha256 for every file. `tests/core/test_dictionary.py` fails when a column or value is undescribed or the metadata
|
||||
is stale. It also caught `attempt` being written as `1.0`, `2.0`: now a whole number.
|
||||
|
||||
## Data collection review (November 2026)
|
||||
|
||||
Collection is an extract → transform → load pipeline declared in `collection/pipeline.yaml` (`python -m collection`).
|
||||
`data/public/runs_clean.csv` holds every run from every source with a `source` column (reddit spreadsheet, reddit
|
||||
thread, reported, bot, reddit pdf). The bot importer is Python (`collection/etl/transform/bot.py`, identical output to
|
||||
the old `bot.js`), and its crosswalk is `collection/specs/bot.yaml`, written into `core/graph.js` for the bot.
|
||||
Reddit goes through the official API; "Report a run" is a GitHub issue form generated from
|
||||
`collection/specs/report_form.yaml`. Only the previous raw version is kept, encrypted, in the Actions cache.
|
||||
|
||||
Still to do: decide which non-spreadsheet sources join the statistics and the route model (`analysis/stats/measures.yaml`
|
||||
`sources`); the bot follows a strategy, so its runs need weighting rather than pooling.
|
||||
|
||||
Reference in New Issue
Block a user