Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+81
View File
@@ -0,0 +1,81 @@
# The DragonKind bot: how its screens and choices become public data and runs. collection/etl/transform/bot.py.
#
# The bot's raw data (game prose, account ids) stays on the machine that runs it. What is published:
# scene fingerprints, the short labels below (our own paraphrases, not game prose), choice text and outcomes.
version: 1
# Must match normalize() in the bot's src/strategy.js, so fingerprints line up (applied in this order).
normalize:
- ["[\u2018\u2019\u201b`']", "'"]
- ["[\u201c\u201d\"]", "\""]
- ["[\u2013\u2014-]", " "]
- ["\u2026", " "]
- ["[^a-z0-9'\\s]", " "]
- ["\\s+", " "]
# Events in the bot's log that record a choice it acted on
choice_events: [DECISION, MANUAL_DECISION, MANUAL_EDGE_PICK]
# runs.jsonl and the log are matched when the same attempt starts within this many hours
match_hours: 6
# Short public labels, by scene fingerprint. A scene without one is labelled "<stage> · <tint>" and listed by the
# importer so a label can be added here.
labels:
"db7786392626": "Start: choose company"
"a68f681076c6": "Valley edge: choose direction"
"bc4ee4754b9c": "First sighting: Red (forest centre)"
"abe3011378ac": "First sighting: Brown (forest centre)"
"84780dab01e3": "Survived the first sighting"
"d25f2759ace0": "Deeper in the forest: a scream"
"f72199f24b91": "Clearing: squadmate vs two bullies"
"b209bacacec3": "Bully fight: first move"
"fbe0f0bea700": "Bully fight: smaller cadet has a dagger"
"a2368b0dc212": "Blocked the bigger cadet: bullies’ fate"
"f4b4944d3f3b": "Beat the smaller cadet: bullies’ fate"
"0cfd06cf8939": "After the fight: choose a direction"
"0a561d53ad19": "An hour later: valley or tree"
"96b68612eaf4": "Top of the tree (shadow check)"
"10bc7353b8eb": "Descended quickly: a Red below"
"c62f1f1a2b3a": "Let the Red pass: two Greens in the clearing"
"07e24681e3b3": "Descended slowly: two Greens in the clearing"
# Crosswalk: a bot choice (option text, normalised as in core/graph.js norm()) -> a column of the coded runs.
# First match wins; a column keeps its first value. Unmatched choices are listed, never guessed.
# Two keys are not columns: scream_investigate (the scream step on its own) and red (the Red below the tree).
crosswalk:
- ["^enter the valley alone", company, "Alone"]
- ["^enter the valley with one of your squadmates", company, "1 Squadmate"]
- ["^enter the valley with two of your squadmates", company, "2 Squadmates"]
- ["^head east toward the roar", direction, "East"]
- ["^head west toward the scent", direction, "West"]
- ["^head toward the center of the forest", direction, "Center"]
- ["^keep walking the cadets are just as dangerous", scream, "Ignore the scream"]
- ["^run and hope", sighting_action, "Retreat / run"]
- ["hold still", sighting_action, "Hold still"]
- ["lower your eyes in respect|lower your eyes and turn to face it", sighting_action, "Respect / face it"]
- ["meeting its gaze", sighting_action, "Meet its gaze"]
- ["^hide behind a tree", sighting_action, "Hide"]
- ["^investigate the scream", scream_investigate, true]
- ["^help your squadmate", scream, "Investigate & help"]
- ["^play it safe and leave them", scream, "Investigate & leave"]
- ["^pause to assess the situation", combat_first, "Assess the situation"]
- ["^charge the other cadet", combat_first, "Even the numbers (smaller cadet)"]
- ["^unsheathe your weapon and charge between", combat_first, "Put yourself in the middle (bigger cadet)"]
- ["^engage the smaller cadet", combat_second, "Engage the smaller cadet"]
- ["^daggers or not", combat_second, "Daggers or not (protect squadmate)"]
- ["^end the lives of the bullies", resolution, "Killed the bullies yourself"]
- ["^allow your squadmate to exact", resolution, "Squadmate's revenge"]
- ["^leave the bullies alive", resolution, "Left bullies to the dragons"]
- ["tourniquet", resolution, "Tourniquet"]
# the wounded cadet's other two choices (the bot has not logged this screen yet, so these are kept loose)
- ["leave (them|him|her) to die", resolution, "Left injured cadet"]
- ["take (their|his|her) weapons?", resolution, "Took their weapons"]
- ["^walk toward the east", post_path, "East"]
- ["^walk toward the west", post_path, "West"]
- ["^walk farther into the forest", post_path, "Forest"]
- ["^continue toward the center of the valley", hour, "Valley"]
- ["^find the tallest tree", hour, "Tree"]
- ["^descend quickly", branch, "Descend quickly: Red"]
- ["^climb down slowly to pursue the green", branch, "Descend slowly: two Greens"]
- ["^reds are too hot tempered", red, "Let it pass (two Greens)"]
- ["^jump (now|in a few seconds)", red, "Jump onto the Red"]