Files
threshing-atlas/data/dictionary.yaml
T
ekcSchianoandClaude Opus 5.5 2271fe0976 Route model: correct for reporting bias
- core/atlas/bias.py: for each rare roll (shadow, meadow Blue) and each source group,
  the factor by which the group over-reports the rare outcome relative to complete
  logging (bot, bulk logs), with a 90% range; rare outcomes from biased groups count
  1/rho; a roll the complete logs have (almost) never shown is learned from them only
- planner/policy.yaml: groups, rolls and thresholds; estimates published in
  policy.json under model.reportingBias
- fit_alpha: a roll seen only one way is pooled fully
- docs: goal accounts follow the best route, research accounts use Thompson sampling
  (decisionMode atlas_thompson)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:29:09 +00:00

156 lines
10 KiB
YAML

# Data dictionary: every published file and every column of the coded runs.
#
# collection/etl/metadata.py builds data/croissant.json (Croissant, the MLCommons metadata format) from this
# file, collection/pipeline.yaml and the licences. tests/core/test_dictionary.py checks that every column of the coded runs
# is described here and that every value it takes is listed.
#
# Missing answers: "Unknown", "Unsure", "(not asked)" and the other values in core/spec/form.yaml missing_values carry
# no information about the choice; every analysis treats them, and empty cells, as missing.
version: 1
dataset:
name: Threshing Atlas
description: >-
Community-logged runs of the Dragonkind "threshing" (the dragon-bonding game based on Fourth Wing), coded into
one record per run: every choice, what the game rolled, and how the run ended. Also the route model built from
them, the statistics shown on the Threshing Atlas page, and the route graph shared with a DragonKind bot.
url: https://ekcschiano.github.io/threshing-atlas/
repository: https://github.com/ekcSchiano/threshing-atlas
license: https://creativecommons.org/licenses/by/4.0/
code_license: MIT
creator: ekcSchiano
keywords: [Dragonkind, Fourth Wing, threshing, decision tree, route planning, community data]
citation: "ekcSchiano and contributors (2026). Threshing Atlas. https://github.com/ekcSchiano/threshing-atlas"
# Third-party sources keep their own terms (DATA_LICENSE.md); the route PDF is never redistributed.
files:
runs_clean:
path: data/public/runs_clean.csv
url: https://ekcschiano.github.io/threshing-atlas/data/runs_clean.csv
format: text/csv
description: >-
Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's
rules are core/spec/form.yaml). The `source` column says where each run came from.
form_responses:
path: data/public/form_responses.json
format: application/json
description: >-
The community "Threshing Routes" form export as published (headers and rows), with email addresses in
free-text answers redacted. The source of runs_clean.
reports:
path: data/public/reports.json
format: application/json
description: Runs reported through the Atlas's "Report a run" form (GitHub issues), answers only, no usernames.
register:
path: data/public/register.json
url: https://ekcschiano.github.io/threshing-atlas/data/register.json
format: application/json
description: The Unofficial Dragon Register (which dragon each rider bonded), without rider names.
stats:
path: data/public/stats.json
url: https://ekcschiano.github.io/threshing-atlas/data/stats.json
format: application/json
description: >-
Every number the Atlas page shows, for six filter presets (analysis/stats/site_stats.py; what is measured is
analysis/stats/measures.yaml).
policy:
path: data/public/policy.json
url: https://ekcschiano.github.io/threshing-atlas/data/policy.json
format: application/json
description: >-
The route model: the game (core/spec/game.yaml), the count tables behind every roll, and the best decision
tree for every dragon colour under two goals, with posterior intervals (planner/build_policy.py).
graph:
path: data/public/graph.json
url: https://ekcschiano.github.io/threshing-atlas/data/graph.json
format: application/json
description: The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).
# The columns of runs_clean. type: text, integer, boolean or date. values: every value the column may take
# (empty cells are always allowed and mean "does not apply" or "not recorded").
columns:
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>." }
source:
type: text
required: true
description: >-
Where the run came from. "reddit spreadsheet": the community Threshing Routes form and spreadsheet linked from
the r/fourthwing megathreads. "reddit thread": a dragonkind.com result page posted in a megathread (bonded,
route not recorded). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
DragonKind bot. "reddit pdf": the community route PDF (a flowchart of the routes, not a log of runs, so it
adds no rows today).
values: [reddit spreadsheet, reddit thread, reported, bot, reddit pdf]
entry:
type: text
description: >-
For spreadsheet runs, the sheet's own row label: "Form" for a form submission (it has a timestamp);
bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept
them, so they are the least selectively reported rows. Empty for other sources.
values: [Form, Data Import, BulkUpload3, Manual Upload, Historical import, Unknown import]
bot_mode:
type: text
description: >-
For bot runs, how the bot chose: "auto_learn" (its own rules and the route PDF), "atlas_policy" (the Atlas's
route model), "atlas_thompson" (the route model, sampling routes in proportion to how likely each is best), "explore" (trying untested options), "experiment" (a randomised test), "manual" (a person
clicked). Matters for analysis: the bot follows a strategy, so its choices are not like a player's. Empty for
other sources.
values: [auto_learn, atlas_policy, atlas_thompson, explore, experiment, manual]
date: { type: date, description: "Date of the run or its report (YYYY-MM-DD): the form submission, the issue, or the bot attempt. Empty for bulk-pasted rows and result pages." }
wing: { type: text, description: The rider's wing, from the form., values: [First, Second, Third, Fourth, Multiple, Unknown] }
section: { type: text, description: The rider's section within the wing (free text as answered). }
squad: { type: text, description: The rider's squad., values: [First, Second, Third, "Don't know", Unknown] }
attempt: { type: integer, description: Which attempt this was for the account (1 = first). Empty if not given. }
attempt_group: { type: text, description: First attempt or a later one ("I've lost count" counts as later)., values: [First attempt, Later attempt, Unknown] }
goal:
type: text
description: Which dragon the rider was trying to bond (older form) or why they did the run (redesigned form).
values: [Red, Green, Orange, Brown, Blue, Black, Blue or Black, Red (tree), Any (for fun), Daily run data, "(not asked/blank)"]
company: { type: text, description: Who entered the valley with the rider., values: [Alone, 1 Squadmate, 2 Squadmates, Unknown] }
left_squadmates:
type: boolean
description: >-
Whether a rider who entered with squadmates later left them. Only asked from 2026-10-27: earlier and undated
squad runs are empty (unknown).
direction: { type: text, description: First direction., values: [East, West, Center, Multiple, Unknown] }
first_sighting: { type: text, description: The first dragon that found the rider (a roll of the game)., values: [Red, Brown, Green, Orange, Unsure, Unknown] }
sighting_action:
type: text
description: How the rider reacted to the first dragon (redesigned form only).
values: [Hold still, Respect / face it, Retreat / run, Hide, Weapon, Meet its gaze, Protect squadmate, Approach, Stand alone, Unknown, "(not asked)"]
scream:
type: text
description: What the rider did about the scream, or that they died before it.
values: [Investigate & help, Investigate & leave, Ignore the scream, DIED at first sighting, Unknown]
resolution:
type: text
description: How the cadet encounter ended.
values: [Killed the bullies yourself, Squadmate's revenge, Left bullies to the dragons, Tourniquet, Left injured cadet, Took their weapons, Left them to it, DIED in combat, DIED at first sighting, Unknown]
combat_first:
type: text
description: The rider's first move in the bully fight.
values: [Assess the situation, Even the numbers (smaller cadet), Put yourself in the middle (bigger cadet), "(not recorded)"]
combat_second: { type: text, description: After assessing, what the rider did about the smaller cadet's dagger., values: [Engage the smaller cadet, Daggers or not (protect squadmate), "(not recorded)"] }
post_path: { type: text, description: The fork after the bully fight (only that path has one)., values: [Forest, West, East, (no fork), Unknown] }
hour: { type: text, description: After an hour's walk, the tallest tree or the valley centre., values: [Tree, Valley, DIED before the hour, Unknown] }
branch:
type: text
description: Which bonding encounter the run reached after the hour, worked out from the dragons recorded.
values: [Shadow (rare), "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens", Valley Red, "Passed the Red: meadow"]
enc_Red: &enc { type: text, description: The rider's answer for this dragon in the bonding grid., values: [Bonded!, Incinerated, Changed my mind, Saved by another] }
enc_Green: *enc
enc_Orange: *enc
enc_Brown: *enc
enc_Blue: *enc
enc_Black: *enc
outcome: { type: text, description: How the run ended., values: [Bonded, Incinerated, Died before the hour, No outcome recorded, Ambiguous] }
outcome_color: { type: text, description: The dragon bonded, or the one that incinerated the rider (the last met)., values: [Red, Green, Orange, Brown, Blue, Black] }
rescued: { type: boolean, description: The Greens' encounter ended with an Orange or Brown swooping in and bonding the rider. }
saw_blue_meadow: { type: boolean, description: A Blue waited in the meadow (valley runs). }
saw_shadow: { type: boolean, description: The shadow appeared at the tree (tree runs). }
rare_reported: { type: text, description: The rider's own answer to "did you meet a rare dragon"., values: ["No", Blue (valley), Shadow, Unknown, "(not asked)"] }
malek_intent: { type: text, description: Whether the rider would seek Malek (die on purpose) rather than bond a dragon they did not want., values: ["Yes", "No", Unknown, "(not asked)"] }
tail: { type: text, description: The tail of the dragon bonded (redesigned form)., values: [Clubtail, Daggertail, Morningstartail, Scorpiontail, Swordtail] }
data_flags: { type: text, description: Inconsistencies found while coding the row, separated by "; ". Empty when none. }
death_note: { type: text, description: Free-text note from riders who died (email addresses removed). }
bond_note: { type: text, description: Free-text note from riders who bonded (email addresses removed). }