Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
@@ -0,0 +1,81 @@
|
||||
# The DragonKind bot: how its screens and choices become public data and runs. collection/etl/transform/bot.py.
|
||||
#
|
||||
# The bot's raw data (game prose, account ids) stays on the machine that runs it. What is published:
|
||||
# scene fingerprints, the short labels below (our own paraphrases, not game prose), choice text and outcomes.
|
||||
version: 1
|
||||
|
||||
# Must match normalize() in the bot's src/strategy.js, so fingerprints line up (applied in this order).
|
||||
normalize:
|
||||
- ["[\u2018\u2019\u201b`']", "'"]
|
||||
- ["[\u201c\u201d\"]", "\""]
|
||||
- ["[\u2013\u2014-]", " "]
|
||||
- ["\u2026", " "]
|
||||
- ["[^a-z0-9'\\s]", " "]
|
||||
- ["\\s+", " "]
|
||||
|
||||
# Events in the bot's log that record a choice it acted on
|
||||
choice_events: [DECISION, MANUAL_DECISION, MANUAL_EDGE_PICK]
|
||||
# runs.jsonl and the log are matched when the same attempt starts within this many hours
|
||||
match_hours: 6
|
||||
|
||||
# Short public labels, by scene fingerprint. A scene without one is labelled "<stage> · <tint>" and listed by the
|
||||
# importer so a label can be added here.
|
||||
labels:
|
||||
"db7786392626": "Start: choose company"
|
||||
"a68f681076c6": "Valley edge: choose direction"
|
||||
"bc4ee4754b9c": "First sighting: Red (forest centre)"
|
||||
"abe3011378ac": "First sighting: Brown (forest centre)"
|
||||
"84780dab01e3": "Survived the first sighting"
|
||||
"d25f2759ace0": "Deeper in the forest: a scream"
|
||||
"f72199f24b91": "Clearing: squadmate vs two bullies"
|
||||
"b209bacacec3": "Bully fight: first move"
|
||||
"fbe0f0bea700": "Bully fight: smaller cadet has a dagger"
|
||||
"a2368b0dc212": "Blocked the bigger cadet: bullies’ fate"
|
||||
"f4b4944d3f3b": "Beat the smaller cadet: bullies’ fate"
|
||||
"0cfd06cf8939": "After the fight: choose a direction"
|
||||
"0a561d53ad19": "An hour later: valley or tree"
|
||||
"96b68612eaf4": "Top of the tree (shadow check)"
|
||||
"10bc7353b8eb": "Descended quickly: a Red below"
|
||||
"c62f1f1a2b3a": "Let the Red pass: two Greens in the clearing"
|
||||
"07e24681e3b3": "Descended slowly: two Greens in the clearing"
|
||||
|
||||
# Crosswalk: a bot choice (option text, normalised as in core/graph.js norm()) -> a column of the coded runs.
|
||||
# First match wins; a column keeps its first value. Unmatched choices are listed, never guessed.
|
||||
# Two keys are not columns: scream_investigate (the scream step on its own) and red (the Red below the tree).
|
||||
crosswalk:
|
||||
- ["^enter the valley alone", company, "Alone"]
|
||||
- ["^enter the valley with one of your squadmates", company, "1 Squadmate"]
|
||||
- ["^enter the valley with two of your squadmates", company, "2 Squadmates"]
|
||||
- ["^head east toward the roar", direction, "East"]
|
||||
- ["^head west toward the scent", direction, "West"]
|
||||
- ["^head toward the center of the forest", direction, "Center"]
|
||||
- ["^keep walking the cadets are just as dangerous", scream, "Ignore the scream"]
|
||||
- ["^run and hope", sighting_action, "Retreat / run"]
|
||||
- ["hold still", sighting_action, "Hold still"]
|
||||
- ["lower your eyes in respect|lower your eyes and turn to face it", sighting_action, "Respect / face it"]
|
||||
- ["meeting its gaze", sighting_action, "Meet its gaze"]
|
||||
- ["^hide behind a tree", sighting_action, "Hide"]
|
||||
- ["^investigate the scream", scream_investigate, true]
|
||||
- ["^help your squadmate", scream, "Investigate & help"]
|
||||
- ["^play it safe and leave them", scream, "Investigate & leave"]
|
||||
- ["^pause to assess the situation", combat_first, "Assess the situation"]
|
||||
- ["^charge the other cadet", combat_first, "Even the numbers (smaller cadet)"]
|
||||
- ["^unsheathe your weapon and charge between", combat_first, "Put yourself in the middle (bigger cadet)"]
|
||||
- ["^engage the smaller cadet", combat_second, "Engage the smaller cadet"]
|
||||
- ["^daggers or not", combat_second, "Daggers or not (protect squadmate)"]
|
||||
- ["^end the lives of the bullies", resolution, "Killed the bullies yourself"]
|
||||
- ["^allow your squadmate to exact", resolution, "Squadmate's revenge"]
|
||||
- ["^leave the bullies alive", resolution, "Left bullies to the dragons"]
|
||||
- ["tourniquet", resolution, "Tourniquet"]
|
||||
# the wounded cadet's other two choices (the bot has not logged this screen yet, so these are kept loose)
|
||||
- ["leave (them|him|her) to die", resolution, "Left injured cadet"]
|
||||
- ["take (their|his|her) weapons?", resolution, "Took their weapons"]
|
||||
- ["^walk toward the east", post_path, "East"]
|
||||
- ["^walk toward the west", post_path, "West"]
|
||||
- ["^walk farther into the forest", post_path, "Forest"]
|
||||
- ["^continue toward the center of the valley", hour, "Valley"]
|
||||
- ["^find the tallest tree", hour, "Tree"]
|
||||
- ["^descend quickly", branch, "Descend quickly: Red"]
|
||||
- ["^climb down slowly to pursue the green", branch, "Descend slowly: two Greens"]
|
||||
- ["^reds are too hot tempered", red, "Let it pass (two Greens)"]
|
||||
- ["^jump (now|in a few seconds)", red, "Jump onto the Red"]
|
||||
@@ -0,0 +1,118 @@
|
||||
# "Report a run": the questions of the GitHub issue form, and how each answer becomes a column of runs_clean.csv.
|
||||
#
|
||||
# collection/etl/report_form.py writes .github/ISSUE_TEMPLATE/report-a-run.yml from this file (a test checks it is
|
||||
# up to date), and collection/etl/runs.py codes every report with it. Each option is "label shown: column value";
|
||||
# an option with value null (or no answer) means "not known". Values must be listed in data/dictionary.yaml.
|
||||
version: 1
|
||||
|
||||
name: Report a run
|
||||
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
|
||||
title: "Run report"
|
||||
label: run report
|
||||
intro: |
|
||||
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
|
||||
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
|
||||
Please don't include your DragonKind account name or anything personal.
|
||||
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
|
||||
|
||||
fields:
|
||||
- id: goal
|
||||
label: Which dragon were you trying to bond?
|
||||
column: goal
|
||||
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black, "Any, just for fun": Any (for fun) }
|
||||
- id: attempt
|
||||
label: Which attempt was this for your account? (a number; 1 = your first)
|
||||
column: attempt
|
||||
type: integer
|
||||
- id: company
|
||||
label: Who entered the valley with you?
|
||||
column: company
|
||||
required: true
|
||||
options: { Alone: Alone, One squadmate: 1 Squadmate, Two squadmates: 2 Squadmates }
|
||||
- id: left_squadmates
|
||||
label: If you went with squadmates, did you leave them?
|
||||
column: left_squadmates
|
||||
options: { "Yes, I left them": true, "No, we stayed together": false, "I went alone": null }
|
||||
- id: direction
|
||||
label: Which way did you go first?
|
||||
column: direction
|
||||
required: true
|
||||
options: { East (toward the roar): East, West (toward the scent): West, Toward the forest centre: Center }
|
||||
- id: first_sighting
|
||||
label: Which dragon found you first?
|
||||
column: first_sighting
|
||||
options: { Red: Red, Brown: Brown, Green: Green, Orange: Orange }
|
||||
- id: sighting_action
|
||||
label: What did you do when it found you?
|
||||
column: sighting_action
|
||||
options:
|
||||
Hold still: Hold still
|
||||
Lower your eyes in respect / turn to face it: Respect / face it
|
||||
Run: Retreat / run
|
||||
Hide: Hide
|
||||
Draw a weapon: Weapon
|
||||
Meet its gaze: Meet its gaze
|
||||
Protect your squadmate: Protect squadmate
|
||||
- id: scream
|
||||
label: The scream
|
||||
column: scream
|
||||
required: true
|
||||
options:
|
||||
I died at the first dragon: DIED at first sighting
|
||||
Investigated, and helped my squadmate: Investigate & help
|
||||
Investigated, and walked away: Investigate & leave
|
||||
Kept walking (ignored it): Ignore the scream
|
||||
- id: combat_first
|
||||
label: "If you helped: your first move in the fight"
|
||||
column: combat_first
|
||||
options:
|
||||
Pause to assess: Assess the situation
|
||||
Charge the smaller cadet: Even the numbers (smaller cadet)
|
||||
Charge between the bigger cadet and your squadmate: Put yourself in the middle (bigger cadet)
|
||||
- id: combat_second
|
||||
label: "If you paused to assess: the smaller cadet's dagger"
|
||||
column: combat_second
|
||||
options: { Engage the smaller cadet: Engage the smaller cadet, Charge the bigger cadet anyway: Daggers or not (protect squadmate) }
|
||||
- id: resolution
|
||||
label: How did the cadet encounter end?
|
||||
column: resolution
|
||||
options:
|
||||
I died in the fight: DIED in combat
|
||||
"Bullies: killed them myself": Killed the bullies yourself
|
||||
"Bullies: my squadmate's revenge": Squadmate's revenge
|
||||
"Bullies: left them to the dragons": Left bullies to the dragons
|
||||
"Wounded cadet: tourniquet": Tourniquet
|
||||
"Wounded cadet: left them": Left injured cadet
|
||||
"Wounded cadet: took their weapons": Took their weapons
|
||||
"I walked away from the clearing": Left them to it
|
||||
- id: post_path
|
||||
label: "After the fight: which way did you walk?"
|
||||
column: post_path
|
||||
options: { Deeper into the forest: Forest, West: West, East: East, There was no fork: (no fork) }
|
||||
- id: hour
|
||||
label: After an hour's walk
|
||||
column: hour
|
||||
options: { Climbed the tallest tree: Tree, Went to the valley centre: Valley, I died before the hour: DIED before the hour }
|
||||
- id: branch
|
||||
label: Which bonding encounter did you reach?
|
||||
column: branch
|
||||
options:
|
||||
"Tree: the shadow (Blue or Black)": Shadow (rare)
|
||||
"Tree: climbed down quickly, jumped on the Red": "Descend quickly: Red"
|
||||
"Tree: climbed down quickly, let the Red pass": "Descend quickly: let the Red pass"
|
||||
"Tree: climbed down slowly to the two Greens": "Descend slowly: two Greens"
|
||||
"Valley: faced the Red": Valley Red
|
||||
"Valley: passed the Red, the meadow": "Passed the Red: meadow"
|
||||
- id: outcome
|
||||
label: How did the run end?
|
||||
column: outcome
|
||||
required: true
|
||||
options: { Bonded: Bonded, Incinerated at a bonding encounter: Incinerated, Died before the hour: Died before the hour }
|
||||
- id: outcome_color
|
||||
label: Which dragon bonded you (or incinerated you)?
|
||||
column: outcome_color
|
||||
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black }
|
||||
- id: tail
|
||||
label: "If you bonded: the dragon's tail"
|
||||
column: tail
|
||||
options: { Clubtail: Clubtail, Daggertail: Daggertail, Morningstartail: Morningstartail, Scorpiontail: Scorpiontail, Swordtail: Swordtail }
|
||||
Reference in New Issue
Block a user