Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+81
View File
@@ -0,0 +1,81 @@
# The DragonKind bot: how its screens and choices become public data and runs. collection/etl/transform/bot.py.
#
# The bot's raw data (game prose, account ids) stays on the machine that runs it. What is published:
# scene fingerprints, the short labels below (our own paraphrases, not game prose), choice text and outcomes.
version: 1
# Must match normalize() in the bot's src/strategy.js, so fingerprints line up (applied in this order).
normalize:
- ["[\u2018\u2019\u201b`']", "'"]
- ["[\u201c\u201d\"]", "\""]
- ["[\u2013\u2014-]", " "]
- ["\u2026", " "]
- ["[^a-z0-9'\\s]", " "]
- ["\\s+", " "]
# Events in the bot's log that record a choice it acted on
choice_events: [DECISION, MANUAL_DECISION, MANUAL_EDGE_PICK]
# runs.jsonl and the log are matched when the same attempt starts within this many hours
match_hours: 6
# Short public labels, by scene fingerprint. A scene without one is labelled "<stage> · <tint>" and listed by the
# importer so a label can be added here.
labels:
"db7786392626": "Start: choose company"
"a68f681076c6": "Valley edge: choose direction"
"bc4ee4754b9c": "First sighting: Red (forest centre)"
"abe3011378ac": "First sighting: Brown (forest centre)"
"84780dab01e3": "Survived the first sighting"
"d25f2759ace0": "Deeper in the forest: a scream"
"f72199f24b91": "Clearing: squadmate vs two bullies"
"b209bacacec3": "Bully fight: first move"
"fbe0f0bea700": "Bully fight: smaller cadet has a dagger"
"a2368b0dc212": "Blocked the bigger cadet: bullies’ fate"
"f4b4944d3f3b": "Beat the smaller cadet: bullies’ fate"
"0cfd06cf8939": "After the fight: choose a direction"
"0a561d53ad19": "An hour later: valley or tree"
"96b68612eaf4": "Top of the tree (shadow check)"
"10bc7353b8eb": "Descended quickly: a Red below"
"c62f1f1a2b3a": "Let the Red pass: two Greens in the clearing"
"07e24681e3b3": "Descended slowly: two Greens in the clearing"
# Crosswalk: a bot choice (option text, normalised as in core/graph.js norm()) -> a column of the coded runs.
# First match wins; a column keeps its first value. Unmatched choices are listed, never guessed.
# Two keys are not columns: scream_investigate (the scream step on its own) and red (the Red below the tree).
crosswalk:
- ["^enter the valley alone", company, "Alone"]
- ["^enter the valley with one of your squadmates", company, "1 Squadmate"]
- ["^enter the valley with two of your squadmates", company, "2 Squadmates"]
- ["^head east toward the roar", direction, "East"]
- ["^head west toward the scent", direction, "West"]
- ["^head toward the center of the forest", direction, "Center"]
- ["^keep walking the cadets are just as dangerous", scream, "Ignore the scream"]
- ["^run and hope", sighting_action, "Retreat / run"]
- ["hold still", sighting_action, "Hold still"]
- ["lower your eyes in respect|lower your eyes and turn to face it", sighting_action, "Respect / face it"]
- ["meeting its gaze", sighting_action, "Meet its gaze"]
- ["^hide behind a tree", sighting_action, "Hide"]
- ["^investigate the scream", scream_investigate, true]
- ["^help your squadmate", scream, "Investigate & help"]
- ["^play it safe and leave them", scream, "Investigate & leave"]
- ["^pause to assess the situation", combat_first, "Assess the situation"]
- ["^charge the other cadet", combat_first, "Even the numbers (smaller cadet)"]
- ["^unsheathe your weapon and charge between", combat_first, "Put yourself in the middle (bigger cadet)"]
- ["^engage the smaller cadet", combat_second, "Engage the smaller cadet"]
- ["^daggers or not", combat_second, "Daggers or not (protect squadmate)"]
- ["^end the lives of the bullies", resolution, "Killed the bullies yourself"]
- ["^allow your squadmate to exact", resolution, "Squadmate's revenge"]
- ["^leave the bullies alive", resolution, "Left bullies to the dragons"]
- ["tourniquet", resolution, "Tourniquet"]
# the wounded cadet's other two choices (the bot has not logged this screen yet, so these are kept loose)
- ["leave (them|him|her) to die", resolution, "Left injured cadet"]
- ["take (their|his|her) weapons?", resolution, "Took their weapons"]
- ["^walk toward the east", post_path, "East"]
- ["^walk toward the west", post_path, "West"]
- ["^walk farther into the forest", post_path, "Forest"]
- ["^continue toward the center of the valley", hour, "Valley"]
- ["^find the tallest tree", hour, "Tree"]
- ["^descend quickly", branch, "Descend quickly: Red"]
- ["^climb down slowly to pursue the green", branch, "Descend slowly: two Greens"]
- ["^reds are too hot tempered", red, "Let it pass (two Greens)"]
- ["^jump (now|in a few seconds)", red, "Jump onto the Red"]
+118
View File
@@ -0,0 +1,118 @@
# "Report a run": the questions of the GitHub issue form, and how each answer becomes a column of runs_clean.csv.
#
# collection/etl/report_form.py writes .github/ISSUE_TEMPLATE/report-a-run.yml from this file (a test checks it is
# up to date), and collection/etl/runs.py codes every report with it. Each option is "label shown: column value";
# an option with value null (or no answer) means "not known". Values must be listed in data/dictionary.yaml.
version: 1
name: Report a run
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
title: "Run report"
label: run report
intro: |
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
Please don't include your DragonKind account name or anything personal.
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
fields:
- id: goal
label: Which dragon were you trying to bond?
column: goal
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black, "Any, just for fun": Any (for fun) }
- id: attempt
label: Which attempt was this for your account? (a number; 1 = your first)
column: attempt
type: integer
- id: company
label: Who entered the valley with you?
column: company
required: true
options: { Alone: Alone, One squadmate: 1 Squadmate, Two squadmates: 2 Squadmates }
- id: left_squadmates
label: If you went with squadmates, did you leave them?
column: left_squadmates
options: { "Yes, I left them": true, "No, we stayed together": false, "I went alone": null }
- id: direction
label: Which way did you go first?
column: direction
required: true
options: { East (toward the roar): East, West (toward the scent): West, Toward the forest centre: Center }
- id: first_sighting
label: Which dragon found you first?
column: first_sighting
options: { Red: Red, Brown: Brown, Green: Green, Orange: Orange }
- id: sighting_action
label: What did you do when it found you?
column: sighting_action
options:
Hold still: Hold still
Lower your eyes in respect / turn to face it: Respect / face it
Run: Retreat / run
Hide: Hide
Draw a weapon: Weapon
Meet its gaze: Meet its gaze
Protect your squadmate: Protect squadmate
- id: scream
label: The scream
column: scream
required: true
options:
I died at the first dragon: DIED at first sighting
Investigated, and helped my squadmate: Investigate & help
Investigated, and walked away: Investigate & leave
Kept walking (ignored it): Ignore the scream
- id: combat_first
label: "If you helped: your first move in the fight"
column: combat_first
options:
Pause to assess: Assess the situation
Charge the smaller cadet: Even the numbers (smaller cadet)
Charge between the bigger cadet and your squadmate: Put yourself in the middle (bigger cadet)
- id: combat_second
label: "If you paused to assess: the smaller cadet's dagger"
column: combat_second
options: { Engage the smaller cadet: Engage the smaller cadet, Charge the bigger cadet anyway: Daggers or not (protect squadmate) }
- id: resolution
label: How did the cadet encounter end?
column: resolution
options:
I died in the fight: DIED in combat
"Bullies: killed them myself": Killed the bullies yourself
"Bullies: my squadmate's revenge": Squadmate's revenge
"Bullies: left them to the dragons": Left bullies to the dragons
"Wounded cadet: tourniquet": Tourniquet
"Wounded cadet: left them": Left injured cadet
"Wounded cadet: took their weapons": Took their weapons
"I walked away from the clearing": Left them to it
- id: post_path
label: "After the fight: which way did you walk?"
column: post_path
options: { Deeper into the forest: Forest, West: West, East: East, There was no fork: (no fork) }
- id: hour
label: After an hour's walk
column: hour
options: { Climbed the tallest tree: Tree, Went to the valley centre: Valley, I died before the hour: DIED before the hour }
- id: branch
label: Which bonding encounter did you reach?
column: branch
options:
"Tree: the shadow (Blue or Black)": Shadow (rare)
"Tree: climbed down quickly, jumped on the Red": "Descend quickly: Red"
"Tree: climbed down quickly, let the Red pass": "Descend quickly: let the Red pass"
"Tree: climbed down slowly to the two Greens": "Descend slowly: two Greens"
"Valley: faced the Red": Valley Red
"Valley: passed the Red, the meadow": "Passed the Red: meadow"
- id: outcome
label: How did the run end?
column: outcome
required: true
options: { Bonded: Bonded, Incinerated at a bonding encounter: Incinerated, Died before the hour: Died before the hour }
- id: outcome_color
label: Which dragon bonded you (or incinerated you)?
column: outcome_color
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black }
- id: tail
label: "If you bonded: the dragon's tail"
column: tail
options: { Clubtail: Clubtail, Daggertail: Daggertail, Morningstartail: Morningstartail, Scorpiontail: Scorpiontail, Swordtail: Swordtail }