Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+50 -13
View File
@@ -1,19 +1,56 @@
# collection
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
under `data/`.
Getting data in, as a strict **extract → transform → load** pipeline. Every source is declared in
[`pipeline.yaml`](pipeline.yaml); the code only carries those declarations out.
| Source | Code | Output |
```
python -m collection update # every source: extract, then transform + load if it changed; then build
python -m collection update --source reported --source register
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
python -m collection build # code every public source file into data/public/runs_clean.csv
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
```
## The stages
| Stage | Does | Never does |
|---|---|---|
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/atlas/coding.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
| Reddit megathreads | not collected yet | — |
| **extract** (`etl/extract.py`) | Fetches the source as it is now into `data/raw/<source>/current.<ext>`. The version it replaces becomes `previous.<ext>`; nothing older is kept. `data/manifest.json` records the current raw file's sha256 and the day it last changed. | Change the data. |
| **transform** (`etl/transform/`) | Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
| **validate** (`etl/validate.py`) | Checks the combined runs against `data/dictionary.yaml` (types, every value listed, unique ids). A failure stops the load and is reported. | Fix anything silently. |
| **load** (`etl/load.py`) | The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
merges itself when CI passes. A broken link opens an issue.
`build` then codes every public source file into one run per row (`etl/runs.py`, one coder per source) and writes
`data/public/runs_clean.csv`, plus the files generated from the specs: the "Report a run" issue form, the bot
crosswalk inside `core/graph.js`, and `data/croissant.json`. Every input to `build` is committed, so CI rebuilds
it exactly.
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.
## The sources
| `source` in runs_clean.csv | Pipeline source | Extract | Public source file |
|---|---|---|---|
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
| — | `register`: the Unofficial Dragon Register | the sheet's .xlsx export | `data/public/register.json` (rider names and photos dropped) |
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to `runs_clean.csv`; its
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
led there, so it adds no rows either.
## Raw files and privacy
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
with the `RAW_ARCHIVE_KEY` secret (see `docs/REPO_SETTINGS.md`). Privacy rules sit with each source in
`pipeline.yaml`: emails redacted, register names and photos dropped, the bot's fields refused if they look like
prose or account ids, GitHub and Reddit usernames never kept.
## Adding a source
1. Add a block to `pipeline.yaml` (extract kind, transform, privacy, load target, and `run_source` / `runs` if it
has runs).
2. If it needs a new transform or coder, add one module to `etl/transform/` and one function to `etl/runs.py`.
3. Add its `source` value to `data/dictionary.yaml`, then `python -m collection build` and `npm test`.
View File
+3
View File
@@ -0,0 +1,3 @@
from collection.run import main
raise SystemExit(main())
-211
View File
@@ -1,211 +0,0 @@
'use strict';
/**
* Import DragonKind bot data into anonymised trajectories and a scene catalogue.
*
* Inputs (all from the PRIVATE bot repo, read locally, never committed):
* data/knowledge.json scene atlas: nodes (prose, choices, tint, backdrop) and edges
* data/runs.jsonl one record per attempt (newer bot versions)
* logs/encounters.jsonl every decision the bot made, with the options on screen
*
* Outputs (public, CC BY 4.0): no game prose, only fingerprints, short labels and choice text.
* scenes.json { sceneId: { stage, tint, backdrop, label, options: [{ id, text }] } }
* trajectories.jsonl one line per attempt: { id, source, meta, steps: [...], outcome }
*/
const crypto = require('node:crypto');
const fs = require('node:fs');
const path = require('node:path');
const sha1 = (s, n = 12) => crypto.createHash('sha1').update(String(s)).digest('hex').slice(0, n);
// Must match normalize() in the bot's src/strategy.js, so fingerprints line up.
function normalize(text) {
return String(text || '')
.toLowerCase()
.replace(/[‘’‛`']/g, "'")
.replace(/[“”"]/g, '"')
.replace(/[–—-]/g, ' ')
.replace(/…/g, ' ')
.replace(/[^a-z0-9'\s]/g, ' ')
.replace(/\s+/g, ' ')
.trim();
}
// Must match computeSceneFingerprint() in the bot's src/learner.js.
function sceneId(prose, choices) {
const proseNorm = normalize(prose).slice(0, 500);
const sorted = choices.map(normalize).filter(Boolean).sort();
return sha1(`${proseNorm} ### ${sorted.join(' || ')}`);
}
const optionId = (text) => sha1(normalize(text), 10);
const readJsonl = (file) => {
if (!file || !fs.existsSync(file)) return [];
return fs.readFileSync(file, 'utf8').split(/\r?\n/).filter((l) => l.trim()).map((l) => {
try { return JSON.parse(l); } catch { return null; }
}).filter(Boolean);
};
/**
* Short public labels. Keys are scene fingerprints; values are our own paraphrases, not game prose.
* Unknown scenes fall back to "<stage> · <tint>" and are listed by the importer so labels can be added.
*/
const LABELS = {
db7786392626: 'Start: choose company',
a68f681076c6: 'Valley edge: choose direction',
bc4ee4754b9c: 'First sighting: Red (forest centre)',
abe3011378ac: 'First sighting: Brown (forest centre)',
'84780dab01e3': 'Survived the first sighting',
d25f2759ace0: 'Deeper in the forest: a scream',
f72199f24b91: 'Clearing: squadmate vs two bullies',
b209bacacec3: 'Bully fight: first move',
fbe0f0bea700: 'Bully fight: smaller cadet has a dagger',
a2368b0dc212: 'Blocked the bigger cadet: bullies’ fate',
f4b4944d3f3b: 'Beat the smaller cadet: bullies’ fate',
'0cfd06cf8939': 'After the fight: choose a direction',
'0a561d53ad19': 'An hour later: valley or tree',
'96b68612eaf4': 'Top of the tree (shadow check)',
'10bc7353b8eb': 'Descended quickly: a Red below',
c62f1f1a2b3a: 'Let the Red pass: two Greens in the clearing',
'07e24681e3b3': 'Descended slowly: two Greens in the clearing'
};
function labelFor(id, node) {
if (LABELS[id]) return LABELS[id];
const bits = [node.stage || 'Scene'];
if (node.tint) bits.push(node.tint);
return bits.join(' · ');
}
/** Scene catalogue from knowledge.json (and, for scenes only in the log, from the log). */
function buildScenes(knowledge, encounters) {
const scenes = {};
for (const [id, n] of Object.entries((knowledge && knowledge.nodes) || {})) {
scenes[id] = {
stage: n.stage || 'UNKNOWN',
tint: n.tint || null,
backdrop: n.backdrop || null,
label: labelFor(id, n),
options: Object.values(n.choices || {}).map((c) => ({ id: optionId(c.label || c.choiceNorm), text: c.label || c.choiceNorm }))
};
}
for (const e of encounters) {
if (!Array.isArray(e.choices) || !e.prose) continue;
const id = e.nodeId || sceneId(e.prose, e.choices);
if (scenes[id]) continue;
const stage = (e.decision && e.decision.stage) || e.stage || 'UNKNOWN';
scenes[id] = { stage, tint: null, backdrop: null, label: labelFor(id, { stage }), options: e.choices.map((t) => ({ id: optionId(t), text: t })) };
}
return scenes;
}
const CHOICE_EVENTS = new Set(['DECISION', 'MANUAL_DECISION', 'MANUAL_EDGE_PICK']);
/**
* Trajectories from the encounter log: group events by attempt, keep the decisions that were acted on,
* and close each attempt at its first BURNED / DRAGON_BONDED event. Repeated BURNED events with no
* decisions in between (the cooldown screen being re-detected) are ignored.
*/
function trajectoriesFromEncounters(encounters, scenes) {
const byAttempt = new Map();
for (const e of encounters) {
const key = e.account ? `${e.account}:${e.attempt}` : String(e.attempt);
if (!byAttempt.has(key)) byAttempt.set(key, []);
byAttempt.get(key).push(e);
}
const out = [];
for (const [key, events] of byAttempt) {
events.sort((a, b) => String(a.ts).localeCompare(String(b.ts)));
const steps = [];
let outcome = null;
for (const e of events) {
if (outcome) break;
if (CHOICE_EVENTS.has(e.event)) {
const d = e.decision || {};
const text = d.choiceText || e.choiceText;
if (d.action && d.action !== 'choose') continue; // paused, nothing happened
if (!text || !Array.isArray(e.choices)) continue;
const scene = e.nodeId || sceneId(e.prose || '', e.choices);
const manual = e.event !== 'DECISION';
steps.push({
scene,
stage: d.stage || (scenes[scene] && scenes[scene].stage) || 'UNKNOWN',
tint: (scenes[scene] && scenes[scene].tint) || null,
option: optionId(text),
optionText: text,
offered: e.choices.map(optionId),
rule: d.rule || null,
decisionMode: manual ? 'manual' : d.source || null,
manual,
at: e.ts
});
} else if (e.event === 'BURNED' && steps.length) {
outcome = { type: 'BURNED', colour: null, intentional: Boolean(e.intentional) };
} else if (e.event === 'DRAGON_BONDED' && steps.length) {
outcome = { type: 'BONDED', colour: e.dragon ? String(e.dragon).toLowerCase() : null, intentional: false };
}
}
// A decision repeated on the same scene (the bot re-detecting the screen) counts once.
const dedup = steps.filter((s, i) => !(i > 0 && steps[i - 1].scene === s.scene && steps[i - 1].option === s.option));
for (let i = 0; i < dedup.length; i++) dedup[i].to = i + 1 < dedup.length ? dedup[i + 1].scene : null;
if (!dedup.length) continue;
const last = dedup[dedup.length - 1];
if (outcome) last.to = outcome.type === 'BURNED' ? 'BURNED' : `BONDED:${outcome.colour || 'unknown'}`;
out.push({
id: `bot-${sha1(`${key}:${dedup[0].at}`, 10)}`,
source: 'bot',
meta: { attempt: Number(String(key).split(':').pop()) || null, startedAt: dedup[0].at, accountKey: null, target: null, decisionMode: null },
steps: dedup,
outcome: outcome || { type: 'ABORTED', colour: null, intentional: false }
});
}
return out;
}
/** Fill run-level metadata (target colour, mode) from runs.jsonl where an attempt is covered by both. */
function attachRunMeta(trajs, runs, salt) {
for (const r of runs) {
const t = trajs.find((x) => x.meta.attempt === r.attempt && Math.abs(Date.parse(x.meta.startedAt) - Date.parse(r.startedAt)) < 6 * 3600e3);
if (!t) continue;
t.meta.target = r.targetDragon || null;
t.meta.decisionMode = r.decisionMode || null;
t.meta.policyId = r.policyId || null;
t.meta.experimentId = r.experimentId || null;
t.meta.arm = r.arm || null;
if (r.accountKey) t.meta.accountKey = r.accountKey;
else if (r.accountId && salt) t.meta.accountKey = sha1(`${salt}:${r.accountId}`, 10);
if (r.outcome === 'BONDED' && r.dragonColor) t.outcome = { type: 'BONDED', colour: String(r.dragonColor).toLowerCase(), intentional: false };
if (r.intentionalBurn) t.outcome.intentional = true;
}
return trajs;
}
function importBot({ dataDir, logsDir, salt = process.env.DK_ACCOUNT_SALT || null }) {
const knowledgeFile = path.join(dataDir, 'knowledge.json');
const knowledge = fs.existsSync(knowledgeFile) ? JSON.parse(fs.readFileSync(knowledgeFile, 'utf8')) : { nodes: {} };
const encounters = readJsonl(path.join(logsDir, 'encounters.jsonl'));
const runs = readJsonl(path.join(dataDir, 'runs.jsonl'));
const scenes = buildScenes(knowledge, encounters);
const trajectories = attachRunMeta(trajectoriesFromEncounters(encounters, scenes), runs, salt);
const unlabeled = Object.entries(scenes).filter(([id]) => !LABELS[id]).map(([id, s]) => `${id} (${s.stage}${s.tint ? `, ${s.tint}` : ''})`);
return { scenes, trajectories, unlabeled };
}
/** Guard: nothing that looks like game prose or an account identifier may leave this module. */
function assertPublicSafe(obj) {
const banned = ['prose', 'proseNorm', 'proseExcerpt', 'email', 'password', 'accountId', 'accountName'];
const walk = (v, p) => {
if (Array.isArray(v)) return v.forEach((x, i) => walk(x, `${p}[${i}]`));
if (v && typeof v === 'object') {
for (const [k, x] of Object.entries(v)) {
if (banned.includes(k)) throw new Error(`refusing to export field "${k}" at ${p}`);
walk(x, `${p}.${k}`);
}
} else if (typeof v === 'string' && v.length > 200) {
throw new Error(`refusing to export a ${v.length}-character string at ${p} (looks like prose)`);
}
};
walk(obj, '$');
}
module.exports = { normalize, sceneId, optionId, buildScenes, trajectoriesFromEncounters, attachRunMeta, importBot, assertPublicSafe, LABELS };
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env node
'use strict';
/**
* Convert a local copy of the DragonKind bot's data into the public, anonymised format.
*
* node collection/bot/import-bot.js --bot /path/to/DragonKind [--out data/bot] [--tag 2026-11-01]
*
* Reads <bot>/data/knowledge.json, <bot>/data/runs.jsonl and <bot>/logs/encounters.jsonl.
* Writes <out>/scenes.json (merged with any existing catalogue) and <out>/trajectories-<tag>.jsonl.
* Set DK_ACCOUNT_SALT in the environment to hash account ids; never commit the salt.
*/
const fs = require('node:fs');
const path = require('node:path');
const { importBot, assertPublicSafe } = require('./bot');
const args = Object.fromEntries(process.argv.slice(2).reduce((acc, a, i, all) => {
if (a.startsWith('--')) acc.push([a.slice(2), all[i + 1]]);
return acc;
}, []));
if (!args.bot) {
console.error('usage: import-bot.js --bot /path/to/DragonKind [--out data/bot] [--tag YYYY-MM-DD]');
process.exit(2);
}
const out = path.resolve(args.out || 'data/bot');
const tag = args.tag || new Date().toISOString().slice(0, 10);
const { scenes, trajectories, unlabeled } = importBot({
dataDir: path.join(args.bot, 'data'),
logsDir: path.join(args.bot, 'logs')
});
assertPublicSafe(scenes);
assertPublicSafe(trajectories);
fs.mkdirSync(out, { recursive: true });
const sceneFile = path.join(out, 'scenes.json');
const existing = fs.existsSync(sceneFile) ? JSON.parse(fs.readFileSync(sceneFile, 'utf8')) : {};
for (const [id, s] of Object.entries(scenes)) {
const old = existing[id];
if (!old) { existing[id] = s; continue; }
const ids = new Set(old.options.map((o) => o.id));
for (const o of s.options) if (!ids.has(o.id)) old.options.push(o);
old.tint = old.tint || s.tint;
old.backdrop = old.backdrop || s.backdrop;
if (old.label.includes('·') && !s.label.includes('·')) old.label = s.label;
}
const sorted = Object.fromEntries(Object.entries(existing).sort(([a], [b]) => a.localeCompare(b)));
fs.writeFileSync(sceneFile, `${JSON.stringify(sorted, null, 1)}\n`);
const trajFile = path.join(out, `trajectories-${tag}.jsonl`);
fs.writeFileSync(trajFile, trajectories.map((t) => JSON.stringify(t)).join('\n') + '\n');
const outcomes = trajectories.reduce((m, t) => ((m[t.outcome.type] = (m[t.outcome.type] || 0) + 1), m), {});
console.log(`${Object.keys(scenes).length} scenes, ${trajectories.length} attempts ${JSON.stringify(outcomes)}`);
console.log(`wrote ${path.relative(process.cwd(), sceneFile)} and ${path.relative(process.cwd(), trajFile)}`);
if (unlabeled.length) console.log(`scenes without a short label (add them to LABELS in collection/bot/bot.js):\n ${unlabeled.join('\n ')}`);
View File
+26
View File
@@ -0,0 +1,26 @@
"""Files written from the declarative specs, so each rule has one source of truth:
.github/ISSUE_TEMPLATE/report-a-run.yml from collection/specs/report_form.yaml (report_form.render)
the CROSSWALK block of core/graph.js from collection/specs/bot.yaml (Ken's bot uses it at run time)
"""
import json
from .common import path, read_yaml
GRAPH_JS = "core/graph.js"
BEGIN = " // BEGIN CROSSWALK"
END = " // END CROSSWALK\n"
def graph_js():
"""core/graph.js with its CROSSWALK block rewritten from the bot spec."""
text = path(GRAPH_JS).read_text(encoding="utf-8")
a = text.index(BEGIN)
a = text.index("\n", a) + 1
b = text.index(END, a)
rules = read_yaml("collection/specs/bot.yaml")["crosswalk"]
lines = [" const CROSSWALK = ["]
for i, (rx, col, val) in enumerate(rules):
sep = "," if i < len(rules) - 1 else ""
lines.append(f" [/{rx.replace('/', chr(92) + '/')}/, {json.dumps(col)}, {json.dumps(val, ensure_ascii=False)}]{sep}")
lines.append(" ];")
return text[:a] + "\n".join(lines) + "\n" + text[b:]
+76
View File
@@ -0,0 +1,76 @@
"""Paths, the pipeline spec, and byte-stable writers shared by every stage."""
import hashlib
import json
import sys
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[2]
if str(ROOT / "core") not in sys.path:
sys.path.insert(0, str(ROOT / "core"))
def path(p):
return ROOT / p
def read_yaml(p):
with open(path(p), encoding="utf-8") as f:
return yaml.safe_load(f)
def pipeline():
return read_yaml("collection/pipeline.yaml")
def sha256_bytes(data):
return hashlib.sha256(data).hexdigest()
def sha256_file(p):
h = hashlib.sha256()
with open(p, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
def json_bytes(obj, compact=True):
if compact:
return (json.dumps(obj, ensure_ascii=False, separators=(",", ":"))).encode("utf-8")
return (json.dumps(obj, ensure_ascii=False, indent=1) + "\n").encode("utf-8")
def jsonl_bytes(rows):
return "".join(json.dumps(r, ensure_ascii=False, separators=(",", ":")) + "\n" for r in rows).encode("utf-8")
def read_jsonl(p):
p = Path(p)
if not p.exists():
return []
out = []
for line in p.read_text(encoding="utf-8").splitlines():
if line.strip():
try:
out.append(json.loads(line))
except json.JSONDecodeError:
pass
return out
class Report:
"""What a pipeline run did, as Markdown lines (the daily pull request's body)."""
def __init__(self):
self.lines = []
self.changed = False
self.failed = []
def add(self, line):
self.lines.append(line)
print(line)
def text(self, title):
return "\n".join([f"## {title}", ""] + self.lines) + "\n"
+132
View File
@@ -0,0 +1,132 @@
"""Extract: fetch each source as it is now into the raw store. No transformation happens here.
The raw store is data/raw/<source>/ (git-ignored, never published):
current.<ext> the latest version
previous.<ext> the version before it (kept only in case it is needed; the daily workflow keeps it in an
encrypted, private cache). Nothing older is kept and old versions are not hashed.
data/manifest.json records, for each source, the sha256 of the current raw file and the date it last changed.
Each extractor returns the raw bytes, or raises Skip when the source cannot be reached from here (a missing
secret, the bot's private repo not on this machine).
"""
import base64
import json
import os
import shutil
import tempfile
import urllib.parse
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from .common import path, sha256_bytes
USER_AGENT = "threshing-atlas-collector/1.0 (+https://github.com/ekcSchiano/threshing-atlas)"
RAW_DIR = path("data/raw")
class Skip(Exception):
"""The source cannot be extracted here; not an error."""
def _get(url, headers=None, data=None, timeout=60):
req = urllib.request.Request(url, data=data, headers={"User-Agent": USER_AGENT, **(headers or {})})
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.read()
# ---------------------------------------------------------------- extractors, by `kind`
def http(spec, env):
data = _get(spec["url"])
head = data[:15].lower()
if head.startswith(b"<!doctype html") or head.startswith(b"<html"):
raise RuntimeError(f"{spec['url']} returned a web page, not a file (is the link still shared publicly?)")
return data
def reddit(spec, env):
"""The threads' comments through the official Reddit API (app-only OAuth: a "script" app's id and secret)."""
cid, secret = (env.get(k) for k in spec["secrets"])
if not (cid and secret):
raise Skip(f"set the {' and '.join(spec['secrets'])} secrets to read the threads through the Reddit API")
auth = base64.b64encode(f"{cid}:{secret}".encode()).decode()
token = json.loads(_get("https://www.reddit.com/api/v1/access_token",
headers={"Authorization": f"Basic {auth}"},
data=b"grant_type=client_credentials"))["access_token"]
threads = {}
for tid in spec["threads"]:
threads[tid] = json.loads(_get(f"https://oauth.reddit.com/comments/{tid}?limit=500&depth=10&raw_json=1",
headers={"Authorization": f"Bearer {token}"}))
return json.dumps(threads, ensure_ascii=False, sort_keys=True).encode("utf-8")
def github_issues(spec, env):
"""Every issue with the label (open and closed), through the GitHub REST API."""
headers = {"Accept": "application/vnd.github+json"}
token = env.get("GITHUB_TOKEN") or env.get("GH_TOKEN")
if token:
headers["Authorization"] = f"Bearer {token}"
label = urllib.parse.quote(spec["label"])
issues, page = [], 1
while True:
url = f"https://api.github.com/repos/{spec['repo']}/issues?labels={label}&state=all&per_page=100&page={page}"
batch = json.loads(_get(url, headers=headers))
issues += [i for i in batch if "pull_request" not in i]
if len(batch) < 100:
break
page += 1
issues.sort(key=lambda i: i["number"])
return json.dumps(issues, ensure_ascii=False, sort_keys=True).encode("utf-8")
def local_dir(spec, env):
"""The bot's files, from a local copy of its private repo (DRAGONKIND_BOT_DIR), packed as one JSON."""
root = env.get(spec["env"])
if not root:
raise Skip(f"the bot's repo is private: set {spec['env']} to a local copy to import it")
files = {}
for rel in spec["files"]:
p = os.path.join(root, rel)
files[rel] = Path(p).read_text(encoding="utf-8") if os.path.exists(p) else None
return json.dumps(files, ensure_ascii=False, sort_keys=True).encode("utf-8")
EXTRACTORS = {"http": http, "reddit": reddit, "github_issues": github_issues, "local_dir": local_dir}
# ---------------------------------------------------------------- the raw store
def raw_paths(name, ext):
d = RAW_DIR / name
return d / f"current.{ext}", d / f"previous.{ext}"
def store(name, ext, data, manifest, today=None):
"""Keep `data` as the current raw file. Returns True when it differs from the last one seen."""
current, previous = raw_paths(name, ext)
digest = sha256_bytes(data)
entry = manifest.get(name) or {}
changed = entry.get("sha256") != digest
current.parent.mkdir(parents=True, exist_ok=True)
on_disk = sha256_bytes(current.read_bytes()) if current.exists() else None
if on_disk and on_disk != digest:
shutil.copyfile(current, previous) # keep one previous version; anything older is gone
if on_disk != digest:
fd, tmp = tempfile.mkstemp(dir=current.parent)
with os.fdopen(fd, "wb") as f:
f.write(data)
os.replace(tmp, current)
if changed:
manifest[name] = {"sha256": digest, "bytes": len(data),
"changed": today or datetime.now(timezone.utc).strftime("%Y-%m-%d")}
return changed
def extract(name, spec, manifest, env=None, local_file=None):
"""Fetch one source (or take a local file) into the raw store. Returns (raw path, changed)."""
env = os.environ if env is None else env
ext = spec["extract"].get("ext") or "json"
data = Path(local_file).read_bytes() if local_file else EXTRACTORS[spec["extract"]["kind"]](spec["extract"], env)
changed = store(name, ext, data, manifest)
return raw_paths(name, ext)[0], changed
+54
View File
@@ -0,0 +1,54 @@
"""Load: the only stage that writes public files. Writes are byte-stable (the same table gives the same bytes)
and only happen when the bytes change, so git shows exactly what changed."""
import csv
import io
import json
from .common import json_bytes, jsonl_bytes, path
def public(obj):
"""Drop transient keys (those starting with "_"), which are for the report only."""
if isinstance(obj, dict):
return {k: public(v) for k, v in obj.items() if not str(k).startswith("_")}
if isinstance(obj, list):
return [public(v) for v in obj]
return obj
def read_json(rel):
p = path(rel)
return json.loads(p.read_text(encoding="utf-8")) if p.exists() else None
def write_bytes(rel, data, check=False):
"""Write when different. Returns True when the file changed (or, with check, would change)."""
p = path(rel)
if p.exists() and p.read_bytes() == data:
return False
if not check:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(data)
return True
def json_file(rel, obj, check=False, compact=True):
return write_bytes(rel, json_bytes(public(obj), compact), check)
def jsonl_file(rel, rows, check=False):
return write_bytes(rel, jsonl_bytes(rows), check)
def csv_bytes(rows, columns):
"""True/False for booleans, empty for missing, everything else as text."""
buf = io.StringIO()
w = csv.writer(buf, lineterminator="\n")
w.writerow(columns)
for r in rows:
w.writerow(["" if r.get(c) is None else r[c] for c in columns])
return buf.getvalue().encode("utf-8")
def csv_file(rel, rows, columns, check=False):
return write_bytes(rel, csv_bytes(rows, columns), check)
@@ -3,20 +3,13 @@
Croissant is the MLCommons metadata format for datasets: it says what each file is, its checksum, and what each
column means, so tools (and people) can load and cite the data without reading this repo's code.
python3 collection/publish_metadata.py # write data/croissant.json
python3 collection/publish_metadata.py --check # exit 1 if data/croissant.json is out of date
Written by `python -m collection build` (a test checks it is up to date).
"""
import hashlib
import json
import sys
from pathlib import Path
import yaml
from .common import ROOT, pipeline, read_yaml, sha256_file
ROOT = Path(__file__).resolve().parents[1]
DICTIONARY = ROOT / "data" / "dictionary.yaml"
SOURCES = ROOT / "data" / "sources.json"
OUT = ROOT / "data" / "croissant.json"
OUT = "data/croissant.json"
CONTEXT = {
"@language": "en",
@@ -57,17 +50,11 @@ CONTEXT = {
DATA_TYPES = {"text": "sc:Text", "integer": "sc:Integer", "boolean": "sc:Boolean", "date": "sc:Date"}
def sha256(path):
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
sha256 = sha256_file
def load_dictionary():
with open(DICTIONARY, encoding="utf-8") as f:
return yaml.safe_load(f)
return read_yaml("data/dictionary.yaml")
def field_description(spec):
@@ -80,8 +67,7 @@ def field_description(spec):
def build(dictionary=None):
d = dictionary or load_dictionary()
ds = d["dataset"]
with open(SOURCES, encoding="utf-8") as f:
sources = json.load(f)
sources = pipeline()["sources"]
distribution = []
for key, spec in d["files"].items():
@@ -124,9 +110,10 @@ def build(dictionary=None):
"citeAs": ds["citation"],
"isLiveDataset": True,
"isBasedOn": [
{"@type": "sc:CreativeWork", "name": s["name"], "url": s["view"]}
for key, s in sources.items()
if key != "pdf" # the route PDF is only fingerprinted, never redistributed
{"@type": "sc:CreativeWork", "name": s["label"], "url": next(iter(s["links"].values()))}
for s in sources.values()
# the route PDF is only fingerprinted, never redistributed; the bot's repo is private
if (s.get("privacy") or {}).get("redistribute", True) and s.get("links")
],
"distribution": distribution,
"recordSet": [
@@ -134,7 +121,7 @@ def build(dictionary=None):
"@type": "cr:RecordSet",
"@id": "runs",
"name": "runs",
"description": "One coded threshing run per row (data/derived/runs_clean.csv).",
"description": "One coded threshing run per row, from every source (data/public/runs_clean.csv).",
"key": {"@id": "runs/run_id"},
"field": fields,
}
@@ -144,20 +131,3 @@ def build(dictionary=None):
def render(doc):
return json.dumps(doc, indent=2, ensure_ascii=False) + "\n"
def main(argv):
text = render(build())
if "--check" in argv:
if not OUT.exists() or OUT.read_text(encoding="utf-8") != text:
print("data/croissant.json is out of date: run python3 collection/publish_metadata.py", file=sys.stderr)
return 1
print("data/croissant.json is up to date")
return 0
OUT.write_text(text, encoding="utf-8")
print(f"wrote {OUT.relative_to(ROOT)}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+60
View File
@@ -0,0 +1,60 @@
"""The "Report a run" GitHub issue form, written from collection/specs/report_form.yaml, and the parser for the
issues it creates (GitHub renders each answer as "### <question>" followed by the answer)."""
import re
import yaml
from .common import read_yaml
class _Dumper(yaml.SafeDumper):
pass
def _str(dumper, s):
# multi-line text (the intro) as a readable block
return dumper.represent_scalar("tag:yaml.org,2002:str", s, style="|" if "\n" in s else None)
_Dumper.add_representer(str, _str)
SPEC_PATH = "collection/specs/report_form.yaml"
TEMPLATE = ".github/ISSUE_TEMPLATE/report-a-run.yml"
NOT_SURE = "Not sure"
NO_RESPONSE = {"", "_No response_", "None", NOT_SURE}
def spec():
return read_yaml(SPEC_PATH)
def render(s=None):
s = s or spec()
body = [{"type": "markdown", "attributes": {"value": s["intro"]}}]
for f in s["fields"]:
if f.get("type") == "integer":
item = {"type": "input", "id": f["id"], "attributes": {"label": f["label"], "placeholder": "e.g. 3"}}
else:
opts = list(f["options"])
if not f.get("required"):
opts.append(NOT_SURE)
item = {"type": "dropdown", "id": f["id"], "attributes": {"label": f["label"], "options": opts}}
item["validations"] = {"required": bool(f.get("required"))}
body.append(item)
form = {"name": s["name"], "description": s["description"], "title": s["title"], "labels": [s["label"]], "body": body}
head = f"# Written by collection/etl/report_form.py from {SPEC_PATH}: edit that file, then run `python -m collection build`.\n"
return head + yaml.dump(form, Dumper=_Dumper, sort_keys=False, allow_unicode=True, width=1000)
def parse_body(body, s=None):
"""Issue body -> {field id: chosen option label or text}. Unanswered questions are left out."""
s = s or spec()
by_label = {f["label"].strip(): f["id"] for f in s["fields"]}
out = {}
for block in re.split(r"^### ", body or "", flags=re.M)[1:]:
q, _, a = block.partition("\n")
fid = by_label.get(q.strip())
a = a.strip()
if fid and a not in NO_RESPONSE:
out[fid] = a
return out
+194
View File
@@ -0,0 +1,194 @@
"""Build: every public source file -> one coded run per row, for data/public/runs_clean.csv.
One coder per source (pipeline.yaml `runs:`). Every coder returns rows keyed by the columns of data/dictionary.yaml;
missing answers get the same "not known" values the community form's coding uses, so every source can be filtered
the same way. The `source` column says where a run came from; `entry` keeps the spreadsheet's own row label.
"""
import json
import re
from pathlib import Path
from .common import path, read_jsonl, read_yaml
from . import report_form
DICT = read_yaml("data/dictionary.yaml")
COLUMNS = list(DICT["columns"])
COLOURS = ["Red", "Green", "Orange", "Brown", "Blue", "Black"]
# what "not known" is, per column, for sources that did not ask (the community form's coding uses the same)
NOT_KNOWN = {
"wing": "Unknown", "squad": "Unknown", "attempt_group": "Unknown", "goal": "(not asked/blank)",
"company": "Unknown", "direction": "Unknown", "first_sighting": "Unknown", "sighting_action": "Unknown",
"scream": "Unknown", "resolution": "Unknown", "combat_first": "(not recorded)", "combat_second": "(not recorded)",
"post_path": "Unknown", "hour": "Unknown", "rare_reported": "(not asked)", "malek_intent": "(not asked)",
"outcome": "No outcome recorded", "data_flags": "",
}
def blank_row(**values):
row = {c: None for c in COLUMNS}
row.update(values)
for c, v in NOT_KNOWN.items():
if row[c] is None:
row[c] = v
return row
def attempt_group(n):
return "First attempt" if n == 1 else "Later attempt" if n and n > 1 else "Unknown"
def encounter_columns(row, intentional=False):
"""The bonding-grid column of the dragon that ended the run."""
col = row["outcome_color"]
if col and not intentional:
if row["outcome"] == "Bonded":
row["enc_" + col] = "Bonded!"
elif row["outcome"] == "Incinerated":
row["enc_" + col] = "Incinerated"
# ---------------------------------------------------------------- coders, by pipeline.yaml `runs:`
def spreadsheet(name, spec):
"""The community form: core/spec/form.yaml + core/atlas/coding.py, unchanged."""
from atlas import coding
df, missing = coding.code_frame(coding.read_export(path(spec["load"])))
if missing:
print(f"WARNING: {name}: questions not found in the sheet: {missing}")
df["attempt"] = df["attempt"].astype("Int64")
rows = []
for rec in df.astype(object).where(df.notna(), None).to_dict("records"):
n = rec.pop("run_id")
entry = rec.pop("source")
rows.append({**{c: None for c in COLUMNS}, **rec, "run_id": f"sheet-{n}", "source": spec["run_source"], "entry": entry})
return rows
def reddit_results(name, spec):
"""Dragons read from result pages linked in the megathreads: bonded, colour and tail known, route unknown."""
rows = []
for r in read_yaml(spec["transform"]["manual"])["results"]:
row = blank_row(run_id=f"thread-{r['id']}", source=spec["run_source"], outcome="Bonded",
outcome_color=r["colour"], tail=r["tail"])
encounter_columns(row)
rows.append(row)
return rows
def reports(name, spec):
"""Runs reported through the "Report a run" issue form (collection/specs/report_form.yaml)."""
form = report_form.spec()
fields = {f["id"]: f for f in form["fields"]}
p = path(spec["load"])
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"reports": []}
rows = []
for rep in data["reports"]:
vals, flags = {}, []
for fid, answer in rep["answers"].items():
f = fields.get(fid)
if not f:
continue
if f.get("type") == "integer":
m = re.search(r"\d+", answer)
vals[f["column"]] = int(m.group()) if m else None
elif answer in f["options"]:
vals[f["column"]] = f["options"][answer]
else:
flags.append(f"unrecognised answer to {fid}")
row = blank_row(run_id=f"reported-{rep['number']}", source=spec["run_source"], date=rep.get("opened"), **vals)
row["attempt_group"] = attempt_group(row["attempt"])
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
row["hour"] = "DIED before the hour"
branch = row["branch"]
if branch:
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
if row["hour"] == "Unknown":
row["hour"] = "Tree" if tree else "Valley"
elif row["hour"] != ("Tree" if tree else "Valley"):
flags.append("encounter does not match the tree/valley answer")
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
row["saw_blue_meadow"] = True
if row["outcome"] == "Bonded" and not row["outcome_color"]:
flags.append("bonded, colour not given")
encounter_columns(row)
row["data_flags"] = "; ".join(flags)
rows.append(row)
return rows
def _norm(text):
from .transform.bot import normalize
return normalize(text)
def bot_trajectories(spec):
"""Every published trajectory (data/bot/trajectories-*.jsonl), each attempt once."""
seen, out = set(), []
for f in sorted(Path(path(spec["load"])).glob("trajectories-*.jsonl")):
for t in read_jsonl(f):
if t["id"] not in seen:
seen.add(t["id"])
out.append(t)
return out
def bot_variables(traj, crosswalk):
"""A bot attempt -> columns, through the crosswalk in collection/specs/bot.yaml. First match wins."""
v, unmapped = {}, []
for s in traj["steps"]:
t = _norm(s["optionText"])
hit = next((c for c in crosswalk if c[0].search(t)), None)
if hit:
v.setdefault(hit[1], hit[2])
else:
unmapped.append(s["optionText"])
# the first dragon is the first tinted scene once the direction is chosen
if v.get("direction") and not v.get("first_sighting") and s.get("tint"):
v["first_sighting"] = s["tint"].capitalize()
return v, unmapped
def bot(name, spec):
cw = [(re.compile(rx), col, val) for rx, col, val in read_yaml(spec["transform"]["spec"])["crosswalk"]]
rows = []
for traj in bot_trajectories(spec):
v, _ = bot_variables(traj, cw)
red = v.pop("red", None)
v.pop("scream_investigate", None)
if red == "Let it pass (two Greens)":
v["branch"] = "Descend quickly: let the Red pass"
o = traj["outcome"]
meta = traj.get("meta") or {}
flags = []
if o["type"] == "BONDED":
outcome, colour = "Bonded", (o.get("colour") or "").capitalize() or None
elif o["type"] == "BURNED":
last_tint = traj["steps"][-1].get("tint")
outcome = "Incinerated" if v.get("hour") else "Died before the hour"
colour = last_tint.capitalize() if last_tint and v.get("hour") else None
if o.get("intentional"):
flags.append("deliberate burn (the bot chose to die)")
else:
outcome, colour = "No outcome recorded", None
target = (meta.get("target") or "").capitalize() or None
row = blank_row(run_id=traj["id"], source=spec["run_source"], date=(meta.get("startedAt") or "")[:10] or None,
attempt=meta.get("attempt"), goal=target if target in COLOURS else None,
outcome=outcome, outcome_color=colour, data_flags="; ".join(flags), **v)
row["attempt_group"] = attempt_group(row["attempt"])
encounter_columns(row, intentional=bool(o.get("intentional")))
rows.append(row)
return rows
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reports": reports, "bot": bot}
def build_runs(pipe):
rows = []
for name in pipe["runs"]["order"]:
spec = pipe["sources"][name]
if spec.get("runs"):
rows += CODERS[spec["runs"]](name, spec)
return rows
+7
View File
@@ -0,0 +1,7 @@
"""Named transforms (pipeline.yaml `transform: {name: ...}`). Each module has
transform(raw_path, spec, ctx) -> the public source file's content (pure: no network, no clock)
describe(old, new) -> report lines comparing the previous public file with the new one
"""
from . import bot, pdf, reddit, reports, sheet
TRANSFORMS = {"sheet": sheet, "pdf_fingerprint": pdf, "bot": bot, "reddit_results": reddit, "reports": reports}
+239
View File
@@ -0,0 +1,239 @@
"""The DragonKind bot's private data -> anonymised trajectories and a scene catalogue (Python port of bot.js).
Raw input (from the private bot repo, packed by extract.local_dir): data/knowledge.json (scene atlas: prose,
choices, tint, backdrop), data/runs.jsonl (one record per attempt) and logs/encounters.jsonl (every decision the
bot made, with the options on screen). Nothing of the prose or the account ids leaves this module: the output
holds scene fingerprints, short labels (collection/specs/bot.yaml), choice text and outcomes, and check_public()
refuses anything else.
Output: {"scenes": {sceneId: {stage, tint, backdrop, label, options: [{id, text}]}},
"trajectories": [{id, source, meta, steps, outcome}], "unlabeled": [...]}
"""
import hashlib
import json
import re
from datetime import datetime
from ..common import read_yaml
SPEC = read_yaml("collection/specs/bot.yaml")
_NORMALIZE = [(re.compile(a), b) for a, b in SPEC["normalize"]]
CHOICE_EVENTS = set(SPEC["choice_events"])
def sha1(s, n=12):
return hashlib.sha1(str(s).encode("utf-8")).hexdigest()[:n]
def normalize(text):
"""Must match normalize() in the bot's src/strategy.js, so fingerprints line up."""
t = str(text or "").lower()
for rx, rep in _NORMALIZE:
t = rx.sub(rep, t)
return t.strip()
def scene_id(prose, choices):
"""Must match computeSceneFingerprint() in the bot's src/learner.js."""
prose_norm = normalize(prose)[:500]
opts = sorted(c for c in (normalize(c) for c in choices) if c)
return sha1(f"{prose_norm} ### {' || '.join(opts)}")
def option_id(text):
return sha1(normalize(text), 10)
def label_for(sid, node):
if sid in SPEC["labels"]:
return SPEC["labels"][sid]
bits = [node.get("stage") or "Scene"]
if node.get("tint"):
bits.append(node["tint"])
return " · ".join(bits)
def _jsonl(text):
out = []
for line in (text or "").splitlines():
if line.strip():
try:
out.append(json.loads(line))
except json.JSONDecodeError:
pass
return out
def build_scenes(knowledge, encounters):
scenes = {}
for sid, n in ((knowledge or {}).get("nodes") or {}).items():
scenes[sid] = {
"stage": n.get("stage") or "UNKNOWN", "tint": n.get("tint"), "backdrop": n.get("backdrop"),
"label": label_for(sid, n),
"options": [{"id": option_id(c.get("label") or c.get("choiceNorm")), "text": c.get("label") or c.get("choiceNorm")}
for c in (n.get("choices") or {}).values()],
}
for e in encounters:
if not isinstance(e.get("choices"), list) or not e.get("prose"):
continue
sid = e.get("nodeId") or scene_id(e["prose"], e["choices"])
if sid in scenes:
continue
stage = (e.get("decision") or {}).get("stage") or e.get("stage") or "UNKNOWN"
scenes[sid] = {"stage": stage, "tint": None, "backdrop": None, "label": label_for(sid, {"stage": stage}),
"options": [{"id": option_id(t), "text": t} for t in e["choices"]]}
return scenes
def trajectories_from_encounters(encounters, scenes):
"""Group the log by attempt, keep the decisions acted on, and close each attempt at its first BURNED or
DRAGON_BONDED event. Repeated BURNED events with no decision between them are ignored."""
by_attempt = {}
for e in encounters:
key = f"{e['account']}:{e.get('attempt')}" if e.get("account") else str(e.get("attempt"))
by_attempt.setdefault(key, []).append(e)
out = []
for key, events in by_attempt.items():
events = sorted(events, key=lambda e: str(e.get("ts")))
steps, outcome = [], None
for e in events:
if outcome:
break
if e.get("event") in CHOICE_EVENTS:
d = e.get("decision") or {}
text = d.get("choiceText") or e.get("choiceText")
if d.get("action") and d["action"] != "choose":
continue # paused, nothing happened
if not text or not isinstance(e.get("choices"), list):
continue
scene = e.get("nodeId") or scene_id(e.get("prose") or "", e["choices"])
manual = e["event"] != "DECISION"
sc = scenes.get(scene) or {}
steps.append({
"scene": scene, "stage": d.get("stage") or sc.get("stage") or "UNKNOWN", "tint": sc.get("tint"),
"option": option_id(text), "optionText": text, "offered": [option_id(c) for c in e["choices"]],
"rule": d.get("rule"), "decisionMode": "manual" if manual else d.get("source"),
"manual": manual, "at": e.get("ts"),
})
elif e.get("event") == "BURNED" and steps:
outcome = {"type": "BURNED", "colour": None, "intentional": bool(e.get("intentional"))}
elif e.get("event") == "DRAGON_BONDED" and steps:
outcome = {"type": "BONDED", "colour": str(e["dragon"]).lower() if e.get("dragon") else None, "intentional": False}
# a decision repeated on the same scene (the bot re-detecting the screen) counts once
dedup = [s for i, s in enumerate(steps)
if not (i > 0 and steps[i - 1]["scene"] == s["scene"] and steps[i - 1]["option"] == s["option"])]
for i, s in enumerate(dedup):
s["to"] = dedup[i + 1]["scene"] if i + 1 < len(dedup) else None
if not dedup:
continue
if outcome:
dedup[-1]["to"] = "BURNED" if outcome["type"] == "BURNED" else f"BONDED:{outcome['colour'] or 'unknown'}"
try:
attempt = int(float(key.split(":")[-1])) or None
except ValueError:
attempt = None
started = dedup[0]["at"]
out.append({
"id": "bot-" + sha1(f"{key}:{started}", 10),
"source": "bot",
"meta": {"attempt": attempt, "startedAt": dedup[0]["at"], "accountKey": None, "target": None, "decisionMode": None},
"steps": dedup,
"outcome": outcome or {"type": "ABORTED", "colour": None, "intentional": False},
})
return out
def _ms(ts):
try:
return datetime.fromisoformat(str(ts).replace("Z", "+00:00")).timestamp() * 1000
except ValueError:
return float("nan")
def attach_run_meta(trajs, runs, salt):
"""Run-level metadata (target colour, mode) from runs.jsonl, where an attempt is in both."""
window = SPEC["match_hours"] * 3600e3
for r in runs:
t = next((x for x in trajs if x["meta"]["attempt"] == r.get("attempt")
and abs(_ms(x["meta"]["startedAt"]) - _ms(r.get("startedAt"))) < window), None)
if not t:
continue
m = t["meta"]
m["target"] = r.get("targetDragon")
m["decisionMode"] = r.get("decisionMode")
m["policyId"] = r.get("policyId")
m["experimentId"] = r.get("experimentId")
m["arm"] = r.get("arm")
if r.get("accountKey"):
m["accountKey"] = r["accountKey"]
elif r.get("accountId") and salt:
m["accountKey"] = sha1(f"{salt}:{r['accountId']}", 10)
if r.get("outcome") == "BONDED" and r.get("dragonColor"):
t["outcome"] = {"type": "BONDED", "colour": str(r["dragonColor"]).lower(), "intentional": False}
if r.get("intentionalBurn"):
t["outcome"]["intentional"] = True
return trajs
def import_bot(knowledge, encounters, runs, salt=None):
scenes = build_scenes(knowledge, encounters)
trajectories = attach_run_meta(trajectories_from_encounters(encounters, scenes), runs, salt)
unlabeled = [f"{sid} ({s['stage']}{', ' + s['tint'] if s['tint'] else ''})" for sid, s in scenes.items() if sid not in SPEC["labels"]]
return scenes, trajectories, unlabeled
def check_public(obj, refuse, max_string):
"""Nothing that looks like game prose or an account identifier may leave this module."""
def walk(v, p):
if isinstance(v, list):
for i, x in enumerate(v):
walk(x, f"{p}[{i}]")
elif isinstance(v, dict):
for k, x in v.items():
if k in refuse:
raise ValueError(f'refusing to export field "{k}" at {p}')
walk(x, f"{p}.{k}")
elif isinstance(v, str) and len(v) > max_string:
raise ValueError(f"refusing to export a {len(v)}-character string at {p} (looks like prose)")
walk(obj, "$")
def merge_scenes(existing, scenes):
"""Add new scenes and new options to the published catalogue; never drop what is already there."""
out = json.loads(json.dumps(existing or {}))
for sid, s in scenes.items():
old = out.get(sid)
if not old:
out[sid] = s
continue
ids = {o["id"] for o in old["options"]}
old["options"] += [o for o in s["options"] if o["id"] not in ids]
old["tint"] = old.get("tint") or s["tint"]
old["backdrop"] = old.get("backdrop") or s["backdrop"]
if "·" in old["label"] and "·" not in s["label"]:
old["label"] = s["label"]
return dict(sorted(out.items()))
def transform(raw_path, spec, ctx):
with open(raw_path, encoding="utf-8") as f:
files = json.load(f)
k = files.get("data/knowledge.json")
knowledge = json.loads(k) if k else {"nodes": {}}
encounters = _jsonl(files.get("logs/encounters.jsonl"))
runs = _jsonl(files.get("data/runs.jsonl"))
privacy = spec.get("privacy") or {}
scenes, trajectories, unlabeled = import_bot(knowledge, encounters, runs, ctx.get("env", {}).get(privacy.get("account_salt_env", "")))
for obj in (scenes, trajectories):
check_public(obj, set(privacy.get("refuse_fields", [])), privacy.get("max_string", 200))
return {"scenes": scenes, "trajectories": trajectories, "_unlabeled": unlabeled}
def describe(old, new):
outcomes = {}
for t in new["trajectories"]:
outcomes[t["outcome"]["type"]] = outcomes.get(t["outcome"]["type"], 0) + 1
lines = [f"{len(new['scenes'])} scenes, {len(new['trajectories'])} attempts {json.dumps(outcomes)}"]
if new.get("_unlabeled"):
lines.append(" - Scenes without a short label (add them to collection/specs/bot.yaml): " + ", ".join(new["_unlabeled"]))
return lines
+51
View File
@@ -0,0 +1,51 @@
"""The route PDF -> its fingerprint only. The PDF is never redistributed: the public file holds hashes of the file,
of each page's text and of each line, so a change can be detected and located without keeping the text."""
import hashlib
def _sha(text, n):
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:n]
def pdf_lines(path):
from pypdf import PdfReader
pages = []
for page in PdfReader(str(path)).pages:
text = page.extract_text() or ""
pages.append([" ".join(line.split()) for line in text.splitlines() if line.strip()])
return pages
def transform(raw_path, spec, ctx):
with open(raw_path, "rb") as f:
data = f.read()
pages = pdf_lines(raw_path)
return {
"sha256": hashlib.sha256(data).hexdigest(),
"bytes": len(data),
"pages": len(pages),
"page_hashes": [_sha("\n".join(p), 16) for p in pages],
"line_hashes": sorted({_sha(line.lower(), 12) for p in pages for line in p}),
"checked": ctx.get("changed"),
# transient: the new lines' text, for the report only (load drops every key starting with "_")
"_lines": [line for p in pages for line in p],
}
def describe(old, new):
if old is None:
return [f"fingerprint recorded for the first time ({new['pages']} pages)"]
if old.get("page_hashes") == new["page_hashes"]:
return ["the file changed but its text did not (layout or images only). Worth a look."]
changed = [i + 1 for i, h in enumerate(new["page_hashes"])
if i >= len(old["page_hashes"]) or old["page_hashes"][i] != h]
old_lines = set(old["line_hashes"])
new_lines = [line for line in new.get("_lines", []) if _sha(line.lower(), 12) not in old_lines]
lines = [f"changed. {old['pages']} → {new['pages']} pages; pages with new text: {', '.join(map(str, changed)) or 'none'}; "
f"{len(new_lines)} new lines, {len(old_lines - set(new['line_hashes']))} lines gone."]
if new_lines:
lines.append(" - New lines (update core/spec/game.yaml or the crosswalks if they add options):")
lines += [f" - {line}" for line in new_lines[:60]]
if len(new_lines) > 60:
lines.append(f" - … and {len(new_lines) - 60} more")
return lines
+45
View File
@@ -0,0 +1,45 @@
"""Reddit megathreads (raw: the threads' comment trees from the official API) -> dragonkind.com result links that
are not yet in collection/manual/reddit_results.yaml. Usernames and comment text are never kept: only the result
ids and the comment permalinks, for the report, so a person can read the result page and add it by hand."""
import json
import re
from pathlib import Path
from ..common import read_yaml
RESULT_RE = re.compile(r"dragonkind\.com/results/(THRESH-[A-Z0-9]+)", re.I)
def _comments(node):
"""Every comment in a listing tree."""
if isinstance(node, list):
for x in node:
yield from _comments(x)
elif isinstance(node, dict):
data = node.get("data") or {}
if node.get("kind") in ("t1", "t3"):
yield data
for key in ("children", "replies"):
if isinstance(data.get(key), (dict, list)):
yield from _comments(data[key])
def transform(raw_path, spec, ctx):
threads = json.loads(Path(raw_path).read_text(encoding="utf-8"))
known = {r["id"].upper() for r in read_yaml(spec["transform"]["manual"])["results"]}
found = {}
for tid, tree in sorted(threads.items()):
for c in _comments(tree):
for m in RESULT_RE.finditer((c.get("body") or "") + " " + (c.get("selftext") or "") + " " + (c.get("url") or "")):
rid = m.group(1).upper()
if rid not in known and rid not in found:
found[rid] = "https://www.reddit.com" + (c.get("permalink") or f"/comments/{tid}/")
return {"new_results": [{"id": k, "comment": v} for k, v in sorted(found.items())]}
def describe(old, new):
rs = new["new_results"]
if not rs:
return ["no new result links in the threads"]
return [f"{len(rs)} result link(s) not yet in collection/manual/reddit_results.yaml (read each page, then add it):"] + \
[f" - https://dragonkind.com/results/{r['id']} (posted at {r['comment']})" for r in rs[:40]]
+31
View File
@@ -0,0 +1,31 @@
"""Run reports (raw: the "run report" issues from the GitHub API) -> data/public/reports.json.
Kept per report: the issue number, the date it was opened, and the answers (the option labels chosen). GitHub
usernames and any other text are dropped. Reports closed as "not planned" or carrying a reject label (e.g.
invalid, duplicate) are left out, so a maintainer removes a bad report by closing it that way."""
import json
from pathlib import Path
from .. import report_form
def transform(raw_path, spec, ctx):
issues = json.loads(Path(raw_path).read_text(encoding="utf-8"))
reject = set(spec["transform"].get("reject_labels", []))
form = report_form.spec()
reports = []
for i in sorted(issues, key=lambda i: i["number"]):
labels = {lb["name"] if isinstance(lb, dict) else lb for lb in i.get("labels") or []}
if labels & reject or i.get("state_reason") == "not_planned":
continue
answers = report_form.parse_body(i.get("body"), form)
if not answers:
continue
reports.append({"number": i["number"], "opened": (i.get("created_at") or "")[:10] or None, "answers": answers})
return {"source": spec["label"], "reports": reports}
def describe(old, new):
o = {r["number"] for r in (old or {}).get("reports", [])}
n = {r["number"] for r in new["reports"]}
return [f"{len(n)} report(s) ({len(n - o)} new, {len(o - n)} removed)"]
+75
View File
@@ -0,0 +1,75 @@
"""A Google Sheets export (.xlsx or .csv) -> the public source file: the sheet's own questions and answers, with the
source's privacy rules applied. No recoding happens here (that is core/atlas/coding.py, at build time)."""
import re
from datetime import datetime
import pandas as pd
from ..common import read_yaml
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
def load_sheet(path, sheet_contains="form responses"):
if str(path).lower().endswith(".csv"):
return pd.read_csv(path)
xl = pd.ExcelFile(path)
sheet = next((n for n in xl.sheet_names if sheet_contains in n.lower()), xl.sheet_names[0])
return pd.read_excel(xl, sheet)
def cell(v, redact=True):
if v is None or v is pd.NaT or (not isinstance(v, str) and pd.isna(v)):
return None
if isinstance(v, (datetime, pd.Timestamp)):
return pd.Timestamp(v).strftime("%Y-%m-%d %H:%M:%S")
if isinstance(v, float) and v.is_integer():
return int(v)
if isinstance(v, str) and redact:
return EMAIL_RE.sub("[email removed]", v) # respondents sometimes leave an email in free text
return v
def extra_results(manual):
"""Dragons read from result pages linked on Reddit (collection/manual/reddit_results.yaml)."""
return [{"source": "dragonkind.com result page linked on Reddit", "url": f"https://dragonkind.com/results/{r['id']}",
"name": r["name"], "colour": r["colour"], "tail": r["tail"], "sex": r["sex"], "age": r["age"],
"lineage": r["lineage"]} for r in read_yaml(manual)["results"]]
def transform(raw_path, spec, ctx):
t, privacy = spec["transform"], spec.get("privacy") or {}
raw = load_sheet(raw_path, t.get("sheet_contains", "form responses"))
drop = [d.lower() for d in privacy.get("drop_columns_matching", [])]
raw = raw[[c for c in raw.columns if not any(d in str(c).lower() for d in drop)]]
redact = privacy.get("redact_emails", True)
rows = []
for _, r in raw.iterrows():
row = {}
for k, v in r.items():
c = cell(v, redact)
if c is not None:
row[str(k)] = c
rows.append(row)
out = {"source": spec["label"], "exported": ctx.get("changed"),
# every header, even one blank in every row, so the coding rules see every question
"headers": [str(c) for c in raw.columns], "rows": rows}
if t.get("extra"):
out["extra"] = extra_results(t["extra"])
return out
def describe(old, new):
"""One report line (plus details) comparing the previous public file with the new one."""
if old is None:
return [f"first export: {len(new['rows'])} rows"]
key = lambda r: repr(sorted(r.items())) # noqa: E731
o, n = {key(r) for r in old["rows"]}, {key(r) for r in new["rows"]}
lines = [f"{len(old['rows'])} → {len(new['rows'])} rows ({len(n - o)} new or edited, {len(o - n)} removed or edited)"]
new_q = [h for h in new["headers"] if h not in old["headers"]]
gone_q = [h for h in old["headers"] if h not in new["headers"]]
if new_q:
lines.append(f" - New questions (add them to `core/spec/form.yaml`): {', '.join(new_q)}")
if gone_q:
lines.append(f" - Questions no longer in the sheet: {', '.join(gone_q)}")
return lines
+44
View File
@@ -0,0 +1,44 @@
"""Validate: a table is checked against data/dictionary.yaml before it is loaded. A failure stops the load."""
import re
CHECKS = {
"integer": lambda v: isinstance(v, int) and not isinstance(v, bool) or bool(re.fullmatch(r"-?\d+", str(v))),
"boolean": lambda v: isinstance(v, bool) or v in ("True", "False"),
"date": lambda v: bool(re.fullmatch(r"\d{4}-\d{2}-\d{2}", str(v))),
"text": lambda v: isinstance(v, str),
}
class Invalid(Exception):
pass
def problems(rows, dictionary, header=None):
"""Every way the rows break the dictionary, as readable lines (empty when the table is valid)."""
cols = dictionary["columns"]
out = []
if header is not None and list(header) != list(cols):
out.append(f"columns differ from the dictionary: {list(header)} vs {list(cols)}")
ids = [r.get("run_id") for r in rows]
if len(set(ids)) != len(ids):
out.append("run_id is not unique")
for name, col in cols.items():
vals = [r.get(name) for r in rows]
present = [v for v in vals if v is not None and v != ""]
bad_type = sorted({str(v) for v in present if not CHECKS[col["type"]](v)})
if bad_type:
out.append(f"{name}: not {col['type']}: {bad_type[:5]}")
if "values" in col:
allowed = {str(v) for v in col["values"]}
bad = sorted({str(v) for v in present if str(v) not in allowed})
if bad:
out.append(f"{name}: values not in the dictionary: {bad[:10]}")
if col.get("required") and len(present) != len(vals):
out.append(f"{name}: required, but {len(vals) - len(present)} rows are empty")
return out
def check(rows, dictionary, what="table"):
errs = problems(rows, dictionary)
if errs:
raise Invalid(f"{what} does not match data/dictionary.yaml:\n " + "\n ".join(errs))
+19
View File
@@ -0,0 +1,19 @@
# Dragons from dragonkind.com result pages that riders linked in the r/fourthwing megathreads, read by hand.
# Each is one run in runs_clean.csv (source "reddit thread": bonded, colour and tail known, route not recorded) and
# appears under "extra" in data/public/register.json. The reddit_thread source lists result links found in the
# threads that are not here yet; add them after reading the result page. No usernames are kept.
version: 1
read: "2026-10-19"
results:
- { id: THRESH-VFRQVM5K3B0CP, name: Thiduach, colour: Blue, tail: Swordtail, sex: Female, age: 51, lineage: Gormfaileas }
- { id: THRESH-3M4P9MNFQGBS8, name: Caenuis, colour: Blue, tail: Swordtail, sex: Female, age: 38, lineage: Gormfaileas }
- { id: THRESH-5JQDJ21G1BYKR, name: Oavuagh, colour: Green, tail: Morningstartail, sex: Female, age: 43, lineage: Uaineloidsig }
- { id: THRESH-5KRAAVFB3D1MG, name: Geagios, colour: Blue, tail: Clubtail, sex: Male, age: 99, lineage: Gormfaileas }
- { id: THRESH-B5HSM00J4A2Z4, name: Fiedheil, colour: Black, tail: Clubtail, sex: Male, age: 32, lineage: Moerlaighen }
- { id: THRESH-BMPM2GAPK58JT, name: Teirfua, colour: Green, tail: Morningstartail, sex: Female, age: 68, lineage: Eifyrielle }
- { id: THRESH-MCQ8D6GAGF4K8, name: Caollo, colour: Blue, tail: Morningstartail, sex: Female, age: 93, lineage: Nyghmuiron }
- { id: THRESH-Q0XDBFMNBEB86, name: Aenminn, colour: Red, tail: Swordtail, sex: Male, age: 59, lineage: Berradhyr }
- { id: THRESH-W99DB29MG16QA, name: Meinso, colour: Green, tail: Clubtail, sex: Female, age: 63, lineage: Eifyrielle }
- { id: THRESH-XCJCY7HW7QSSP, name: Einmial, colour: Blue, tail: Clubtail, sex: Male, age: 58, lineage: Gormfaileas }
- { id: THRESH-XR6G68VPGT48M, name: Recuagh, colour: Brown, tail: Morningstartail, sex: Female, age: 55, lineage: Vormalgh }
- { id: THRESH-YQC97TK34214E, name: Dannidh, colour: Orange, tail: Clubtail, sex: Male, age: 61, lineage: Cariaslog }
+89
View File
@@ -0,0 +1,89 @@
# Data collection as extract -> transform -> load, one block per source. collection/run.py reads this file.
#
# extract fetches the source as it is now. The raw file goes to data/raw/<source>/current.<ext> (git-ignored,
# never published); the version it replaces is kept as previous.<ext>, and the daily workflow keeps
# that previous copy in an encrypted, private cache. Nothing older is kept.
# transform turns the raw file into the public source file: a pure function of the raw file (no network, no
# clock), with the source's privacy rules applied. Named transforms live in collection/etl/transform/.
# load writes the public source file (byte-stable: same input, same bytes) and reports what changed.
#
# Then `build` codes every public source file into one run per row (runs: the coder) and writes
# data/public/runs_clean.csv: every run from every source, with a `source` column. Each table is checked against
# data/dictionary.yaml before it is written. Every input to `build` is committed, so CI can rebuild it exactly.
version: 1
raw_dir: data/raw
manifest: data/manifest.json # for each source: the sha256 of the current raw file and when it last changed
runs:
path: data/public/runs_clean.csv
# sources in this order; within a source, the order the coder returns
order: [reddit_spreadsheet, reddit_thread, reported, bot, reddit_pdf]
sources:
reddit_spreadsheet:
label: Threshing Routes form and spreadsheet (u/ClumsyBadger, linked from the r/fourthwing megathreads)
run_source: reddit spreadsheet
links:
sheet: https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0
form: null # the megathread's Google Form (responder link); shown on "Report a run" when set
extract: { kind: http, url: "https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0/export?format=xlsx", ext: xlsx }
transform: { name: sheet, sheet_contains: form responses }
privacy: { redact_emails: true }
load: data/public/form_responses.json
runs: spreadsheet # core/spec/form.yaml + core/atlas/coding.py
register:
label: Unofficial Dragon Register (which dragon each rider bonded)
links: { sheet: https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ }
extract: { kind: http, url: "https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ/export?format=xlsx", ext: xlsx }
transform: { name: sheet, sheet_contains: form responses, extra: collection/manual/reddit_results.yaml }
privacy: { redact_emails: true, drop_columns_matching: [photo, cadet name] } # no rider names, no photos
load: data/public/register.json
runs: null # bonded dragons without the route that led there
reddit_pdf:
label: Community route PDF (u/ClumsyBadger)
run_source: reddit pdf
links: { view: https://drive.google.com/file/d/17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg/view }
extract: { kind: http, url: "https://drive.google.com/uc?export=download&id=17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg", ext: pdf }
transform: { name: pdf_fingerprint }
privacy: { redistribute: false } # only hashes of the file, its pages and its lines are kept
load: data/derived/pdf_fingerprint.json
runs: null # a flowchart of the routes, not a log of runs (see collection/README.md)
reddit_thread:
label: Runs posted in the r/fourthwing Dragonkind megathreads
run_source: reddit thread
links:
part3: https://www.reddit.com/r/fourthwing/comments/1wvwfp9/
part4: https://www.reddit.com/r/fourthwing/comments/1wz5cnj/
# the official Reddit API (app-only OAuth); skipped with a note when the secrets are not set
extract: { kind: reddit, threads: [1wvwfp9, 1wz5cnj], secrets: [REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET], ext: json }
# lists dragonkind.com result links that are not in the manual file yet; usernames are never kept
transform: { name: reddit_results, manual: collection/manual/reddit_results.yaml }
load: null # the public source file is the hand-checked collection/manual/reddit_results.yaml
runs: reddit_results
reported:
label: Runs reported on the Atlas ("Report a run", a GitHub issue form)
run_source: reported
links: { report: "https://github.com/ekcSchiano/threshing-atlas/issues/new?template=report-a-run.yml" }
extract: { kind: github_issues, repo: ekcSchiano/threshing-atlas, label: run report, ext: json }
# GitHub usernames are dropped; issues closed as "not planned" or labelled "invalid" are left out
transform: { name: reports, form: collection/specs/report_form.yaml, reject_labels: [invalid, duplicate] }
load: data/public/reports.json
runs: reports
bot:
label: DragonKind bot runs (Ken's private bot, imported locally)
run_source: bot
# the bot's repo is private: this step runs only on a machine with a copy of it
extract: { kind: local_dir, env: DRAGONKIND_BOT_DIR, files: [data/knowledge.json, data/runs.jsonl, logs/encounters.jsonl] }
transform: { name: bot, spec: collection/specs/bot.yaml }
privacy:
account_salt_env: DK_ACCOUNT_SALT # account ids are hashed with this salt, which is never committed
refuse_fields: [prose, proseNorm, proseExcerpt, email, password, accountId, accountName]
max_string: 200 # anything longer looks like game prose
load: data/bot # scenes.json (merged) + trajectories-<date>.jsonl
runs: bot
+127
View File
@@ -0,0 +1,127 @@
"""Data collection: extract -> transform -> load for every source, then build the combined runs.
python -m collection update [--source NAME ...] [--file NAME=PATH ...] [--report FILE]
For each source (all by default): extract it (or take a local file), and when the raw file changed,
transform it and load the public source file. Then build. Exit code 0 even when a source could not be
reached: the report says what happened (and the daily workflow opens an issue).
python -m collection build [--check]
Code every public source file into data/public/runs_clean.csv (validated against data/dictionary.yaml),
and write the files generated from the specs: the "Report a run" issue form, the bot crosswalk in
core/graph.js, and data/croissant.json. With --check nothing is written; exit 1
if any of them is out of date (CI runs this).
Sources and their rules: collection/pipeline.yaml. What each column means: data/dictionary.yaml.
"""
import argparse
import os
import sys
from datetime import datetime, timezone
from .etl import extract as E
from .etl import load as L
from .etl import codegen, metadata, report_form, runs, validate
from .etl.common import Report, path, pipeline
from .etl.transform import TRANSFORMS
def update_source(name, spec, manifest, report, local_file=None, env=None):
env = os.environ if env is None else env
try:
raw, changed = E.extract(name, spec, manifest, env, local_file)
except E.Skip as e:
report.add(f"- **{spec['label']}:** skipped ({e}).")
return False
except Exception as e: # one broken source must not block the others
report.failed.append(name)
report.add(f"- **{spec['label']}:** could not extract ({type(e).__name__}: {e}).")
return False
if not changed:
report.add(f"- **{spec['label']}:** no change.")
return False
mod = TRANSFORMS[spec["transform"]["name"]]
ctx = {"changed": manifest[name]["changed"], "env": env}
new = mod.transform(raw, spec, ctx)
target = spec.get("load")
if name == "bot":
old = {"scenes": L.read_json(f"{target}/scenes.json") or {}}
lines = mod.describe(old, new)
wrote = L.json_file(f"{target}/scenes.json", mod.merge_scenes(old["scenes"], new["scenes"]), compact=False)
wrote |= L.jsonl_file(f"{target}/trajectories-{ctx['changed']}.jsonl", new["trajectories"])
elif target:
old = L.read_json(target)
lines = mod.describe(old, new)
wrote = L.json_file(target, new, compact=not target.startswith("data/derived"))
else: # nothing to load: the report is the output (e.g. Reddit result links to add by hand)
lines, wrote = mod.describe(None, new), False
report.add(f"- **{spec['label']}:** {lines[0]}")
for line in lines[1:]:
report.add(line)
return wrote
def build(check=False, report=None):
"""Combined runs, issue form and metadata. Returns the list of files that changed (or would change)."""
pipe = pipeline()
rows = runs.build_runs(pipe)
validate.check(rows, runs.DICT, "runs_clean.csv")
out = []
if L.csv_file(pipe["runs"]["path"], rows, runs.COLUMNS, check):
out.append(pipe["runs"]["path"])
if L.write_bytes(report_form.TEMPLATE, report_form.render().encode("utf-8"), check):
out.append(report_form.TEMPLATE)
if L.write_bytes(codegen.GRAPH_JS, codegen.graph_js().encode("utf-8"), check):
out.append(codegen.GRAPH_JS)
# the metadata hashes the files above, so it is built from what is (or would be) on disk
if not check or not out:
if L.write_bytes(metadata.OUT, metadata.render(metadata.build()).encode("utf-8"), check):
out.append(metadata.OUT)
counts = {}
for r in rows:
counts[r["source"]] = counts.get(r["source"], 0) + 1
msg = f"runs_clean.csv: {len(rows)} runs ({', '.join(f'{k} {v}' for k, v in counts.items())})"
(report.add if report else print)(f"- **Combined runs:** {msg}")
return out
def main(argv=None):
ap = argparse.ArgumentParser(prog="python -m collection", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
sub = ap.add_subparsers(dest="cmd", required=True)
up = sub.add_parser("update")
up.add_argument("--source", action="append", help="only these sources (repeatable)")
up.add_argument("--file", action="append", default=[], help="NAME=PATH: use a local file instead of extracting")
up.add_argument("--report")
b = sub.add_parser("build")
b.add_argument("--check", action="store_true")
a = ap.parse_args(argv)
if a.cmd == "build":
changed = build(check=a.check)
if a.check and changed:
print("out of date (run `python -m collection build`): " + ", ".join(changed), file=sys.stderr)
return 1
return 0
pipe = pipeline()
manifest = L.read_json(pipe["manifest"]) or {}
files = dict(f.split("=", 1) for f in a.file)
report = Report()
for name, spec in pipe["sources"].items():
if a.source and name not in a.source:
continue
report.changed |= update_source(name, spec, manifest, report, files.get(name))
L.json_file(pipe["manifest"], dict(sorted(manifest.items())), compact=False)
try:
report.changed |= bool(build(report=report))
except validate.Invalid as e: # new answers the coding rules do not know yet: report them, load nothing
report.failed.append("build")
report.add(f"- **Combined runs not rebuilt:** {e}")
report.add(" - Add the new answers to `core/spec/form.yaml` (or `data/dictionary.yaml`), then run `python -m collection build`.")
text = report.text(f"Source check {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}")
if a.report:
with open(a.report, "w", encoding="utf-8") as f:
f.write(text)
if os.environ.get("GITHUB_OUTPUT"):
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"changed={'true' if report.changed else 'false'}\nfailed={','.join(report.failed)}\n")
return 0
-49
View File
@@ -1,49 +0,0 @@
"""export_raw.py — convert the Google Sheets .xlsx (or .csv) export into the raw JSON the web page loads.
python export_raw.py ../../data/raw/form/Threshing_Routes_Responses.xlsx ../../data/public/form_responses.json
The JSON is a list of objects keyed by the original question text (no recoding happens here, so the page
applies exactly the same cleaning rules to it as it would to a freshly uploaded export).
"""
import json
import sys
from datetime import datetime
import re
import pandas as pd
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
def load_raw(path):
if str(path).lower().endswith(".csv"):
return pd.read_csv(path)
xl = pd.ExcelFile(path)
sheet = next((n for n in xl.sheet_names if "form responses" in n.lower()), xl.sheet_names[0])
return pd.read_excel(xl, sheet)
def cell(v):
if v is None or v is pd.NaT or (not isinstance(v, str) and pd.isna(v)):
return None
if isinstance(v, (datetime, pd.Timestamp)):
return pd.Timestamp(v).strftime("%Y-%m-%d %H:%M:%S")
if isinstance(v, float) and v.is_integer():
return int(v)
if isinstance(v, str):
return EMAIL_RE.sub("[email removed]", v) # respondents sometimes leave an email in free text
return v
if __name__ == "__main__":
src = sys.argv[1] if len(sys.argv) > 1 else "../../data/raw/form/Threshing_Routes_Responses.xlsx"
out = sys.argv[2] if len(sys.argv) > 2 else "../../data/public/form_responses.json"
raw = load_raw(src)
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
# keep every header present even if a row is blank there, so column detection sees all questions
headers = [str(c) for c in raw.columns]
with open(out, "w", encoding="utf-8") as f:
json.dump({"source_file": str(src).split("/")[-1], "exported": datetime.now().strftime("%Y-%m-%d"),
"headers": headers, "rows": rows}, f, ensure_ascii=False, separators=(",", ":"))
print(f"{len(rows)} rows -> {out}")
-43
View File
@@ -1,43 +0,0 @@
"""export_register.py — convert the "Unofficial Dragon Register" .xlsx/.csv export into web/data/register.json.
python export_register.py ../../data/raw/register/Unofficial_Dragon_Register_Responses.xlsx ../../data/public/register.json
Raw rows are kept as-is, except that photo links and rider names are dropped. Dragons read from public dragonkind.com result pages
linked in the Reddit megathreads are stored separately under "extra" with their source URL.
"""
import json
import sys
from datetime import datetime
import pandas as pd
from export_raw import cell, load_raw
# Read from dragonkind.com/results/<id> pages linked in the Part 3 and Part 4 megathreads (2026-10-19).
REDDIT_RESULTS = [
("THRESH-VFRQVM5K3B0CP", "Thiduach", "Blue", "Swordtail", "Female", 51, "Gormfaileas"),
("THRESH-3M4P9MNFQGBS8", "Caenuis", "Blue", "Swordtail", "Female", 38, "Gormfaileas"),
("THRESH-5JQDJ21G1BYKR", "Oavuagh", "Green", "Morningstartail", "Female", 43, "Uaineloidsig"),
("THRESH-5KRAAVFB3D1MG", "Geagios", "Blue", "Clubtail", "Male", 99, "Gormfaileas"),
("THRESH-B5HSM00J4A2Z4", "Fiedheil", "Black", "Clubtail", "Male", 32, "Moerlaighen"),
("THRESH-BMPM2GAPK58JT", "Teirfua", "Green", "Morningstartail", "Female", 68, "Eifyrielle"),
("THRESH-MCQ8D6GAGF4K8", "Caollo", "Blue", "Morningstartail", "Female", 93, "Nyghmuiron"),
("THRESH-Q0XDBFMNBEB86", "Aenminn", "Red", "Swordtail", "Male", 59, "Berradhyr"),
("THRESH-W99DB29MG16QA", "Meinso", "Green", "Clubtail", "Female", 63, "Eifyrielle"),
("THRESH-XCJCY7HW7QSSP", "Einmial", "Blue", "Clubtail", "Male", 58, "Gormfaileas"),
("THRESH-XR6G68VPGT48M", "Recuagh", "Brown", "Morningstartail", "Female", 55, "Vormalgh"),
("THRESH-YQC97TK34214E", "Dannidh", "Orange", "Clubtail", "Male", 61, "Cariaslog"),
]
if __name__ == "__main__":
src = sys.argv[1] if len(sys.argv) > 1 else "../../data/raw/register/Unofficial_Dragon_Register_Responses.xlsx"
out = sys.argv[2] if len(sys.argv) > 2 else "../../data/public/register.json"
raw = load_raw(src)
raw = raw[[c for c in raw.columns if "photo" not in str(c).lower() and "cadet name" not in str(c).lower()]]
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
extra = [{"source": "dragonkind.com result page linked on Reddit", "url": f"https://dragonkind.com/results/{i}",
"name": n, "colour": c, "tail": t, "sex": s, "age": a, "lineage": l} for i, n, c, t, s, a, l in REDDIT_RESULTS]
json.dump({"source_file": str(src).split("/")[-1], "exported": datetime.now().strftime("%Y-%m-%d"),
"headers": [str(c) for c in raw.columns], "rows": rows, "extra": extra},
open(out, "w", encoding="utf-8"), ensure_ascii=False, separators=(",", ":"))
print(f"{len(rows)} register rows + {len(extra)} Reddit result pages -> {out}")
-185
View File
@@ -1,185 +0,0 @@
"""update_sources.py: check the community spreadsheets and the route PDF for new information.
python collection/sources/update_sources.py [--report report.md] [--form FILE] [--register FILE] [--pdf FILE]
Run from the repo root. Without file arguments it downloads the current versions from the URLs in
data/sources.json (the GitHub Action does this every day). With file arguments it uses local copies,
which is handy for testing or when a download link changes.
What it does:
* Threshing Routes form: re-exports data/public/form_responses.json and data/derived/runs_clean.csv
when the responses changed, and reports how many rows were added or edited.
* Dragon Register: re-exports data/public/register.json (rider names and photos dropped) when it changed.
The raw register export is never written into the repo.
* Route PDF: the PDF is not redistributed. Only a fingerprint is kept (data/derived/pdf_fingerprint.json):
the file hash, per-page text hashes and per-line hashes. When it changes, the report lists the pages that
changed and the lines that are new, so the coding rules and the scene crosswalk can be updated by hand.
Exit code is 0 either way; "changed" files are visible to git, and the report says what happened.
"""
import argparse
import hashlib
import json
import os
import shutil
import subprocess
import sys
import tempfile
import urllib.request
from datetime import datetime, timezone
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
sys.path.insert(0, HERE)
sys.path.insert(0, os.path.join(ROOT, "core"))
from export_raw import cell, load_raw # noqa: E402
P = lambda *a: os.path.join(ROOT, *a) # noqa: E731
def sha(data, n=None):
h = hashlib.sha256(data if isinstance(data, bytes) else data.encode("utf-8")).hexdigest()
return h[:n] if n else h
def download(url, dest):
req = urllib.request.Request(url, headers={"User-Agent": "threshing-atlas-updater"})
with urllib.request.urlopen(req, timeout=60) as r, open(dest, "wb") as f:
data = r.read()
f.write(data)
head = data[:15].lower()
if head.startswith(b"<!doctype html") or head.startswith(b"<html"):
raise RuntimeError(f"{url} returned a web page, not a file (is the link still shared publicly?)")
return dest
def sheet_rows(path, drop=()):
raw = load_raw(path)
raw = raw[[c for c in raw.columns if not any(d in str(c).lower() for d in drop)]]
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
return [str(c) for c in raw.columns], rows
def diff_rows(old, new):
key = lambda r: json.dumps(r, sort_keys=True, ensure_ascii=False) # noqa: E731
o, n = {key(r) for r in old}, {key(r) for r in new}
return len(n - o), len(o - n)
def update_form(src, report):
out = P("data", "public", "form_responses.json")
current = json.load(open(out, encoding="utf-8"))
headers, rows = sheet_rows(src)
added, removed = diff_rows(current["rows"], rows)
if headers == current["headers"] and not added and not removed:
report.append(f"- **Threshing Routes form:** no change ({len(rows)} responses).")
return False
subprocess.run([sys.executable, P("collection", "sources", "export_raw.py"), src, out], check=True, cwd=HERE)
from atlas import coding # noqa: E402
coding.main([out, P("data", "derived", "runs_clean.csv")])
raw_copy = P("data", "raw", "form", "Threshing_Routes_Responses.xlsx")
if os.path.abspath(src) != raw_copy: # local copy only; data/raw/form/*.xlsx is git-ignored
os.makedirs(os.path.dirname(raw_copy), exist_ok=True)
shutil.copyfile(src, raw_copy)
new_q = [h for h in headers if h not in current["headers"]]
gone_q = [h for h in current["headers"] if h not in headers]
report.append(f"- **Threshing Routes form:** {len(current['rows'])} → {len(rows)} responses "
f"({added} new or edited, {removed} removed or edited).")
if new_q:
report.append(f" - New questions (add them to `core/spec/form.yaml`): {', '.join(new_q)}")
if gone_q:
report.append(f" - Questions no longer in the sheet: {', '.join(gone_q)}")
return True
def update_register(src, report):
out = P("data", "public", "register.json")
current = json.load(open(out, encoding="utf-8"))
headers, rows = sheet_rows(src, drop=("photo", "cadet name"))
added, removed = diff_rows(current["rows"], rows)
if headers == current["headers"] and not added and not removed:
report.append(f"- **Dragon Register:** no change ({len(rows)} dragons).")
return False
subprocess.run([sys.executable, P("collection", "sources", "export_register.py"), src, out], check=True, cwd=HERE)
report.append(f"- **Dragon Register:** {len(current['rows'])} → {len(rows)} entries ({added} new or edited, {removed} removed or edited).")
return True
def pdf_lines(path):
try:
from pypdf import PdfReader
except ImportError:
sys.exit("pypdf is required: pip install -r requirements.txt")
pages = []
for page in PdfReader(path).pages:
text = page.extract_text() or ""
pages.append([" ".join(l.split()) for l in text.splitlines() if l.strip()])
return pages
def update_pdf(src, report):
out = P("data", "derived", "pdf_fingerprint.json")
old = json.load(open(out)) if os.path.exists(out) else None
data = open(src, "rb").read()
pages = pdf_lines(src)
fp = {
"sha256": sha(data),
"bytes": len(data),
"pages": len(pages),
"page_hashes": [sha("\n".join(p), 16) for p in pages],
"line_hashes": sorted({sha(l.lower(), 12) for p in pages for l in p}),
"checked": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
}
if old and old["sha256"] == fp["sha256"]:
report.append(f"- **Route PDF:** no change ({fp['pages']} pages).")
return False
if old and old.get("page_hashes") == fp["page_hashes"]:
report.append("- **Route PDF:** the file changed but its text did not (layout or images only). Worth a look.")
elif old:
changed = [i + 1 for i, h in enumerate(fp["page_hashes"]) if i >= len(old["page_hashes"]) or old["page_hashes"][i] != h]
old_lines = set(old["line_hashes"])
new_lines = [l for p in pages for l in p if sha(l.lower(), 12) not in old_lines]
removed = len(old_lines - set(fp["line_hashes"]))
report.append(f"- **Route PDF:** changed. {old['pages']} → {fp['pages']} pages; pages with new text: "
f"{', '.join(map(str, changed)) or 'none'}; {len(new_lines)} new lines, {removed} lines gone.")
if new_lines:
report.append(" - New lines (update the coding rules or the scene crosswalk if they add options):")
report.extend(f" - {l}" for l in new_lines[:60])
if len(new_lines) > 60:
report.append(f" - … and {len(new_lines) - 60} more")
else:
report.append(f"- **Route PDF:** fingerprint recorded for the first time ({fp['pages']} pages).")
json.dump(fp, open(out, "w"), indent=1)
return True
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--report", default=None)
ap.add_argument("--form")
ap.add_argument("--register")
ap.add_argument("--pdf")
args = ap.parse_args()
sources = json.load(open(P("data", "sources.json")))
report, changed, failed = [], False, []
tmp = tempfile.mkdtemp(prefix="sources-")
steps = [("form", args.form, update_form), ("register", args.register, update_register), ("pdf", args.pdf, update_pdf)]
for key, local, fn in steps:
try:
path = local or download(sources[key]["download"], os.path.join(tmp, sources[key]["filename"]))
changed |= fn(os.path.abspath(path), report)
except Exception as e: # keep going: one broken link should not block the others
failed.append(key)
report.append(f"- **{sources[key]['name']}:** could not check ({e}).")
head = f"Source check {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}"
text = "\n".join([f"## {head}", ""] + report) + "\n"
print(text)
if args.report:
open(args.report, "w", encoding="utf-8").write(text)
if os.environ.get("GITHUB_OUTPUT"):
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"changed={'true' if changed else 'false'}\nfailed={','.join(failed)}\n")
if __name__ == "__main__":
main()
+81
View File
@@ -0,0 +1,81 @@
# The DragonKind bot: how its screens and choices become public data and runs. collection/etl/transform/bot.py.
#
# The bot's raw data (game prose, account ids) stays on the machine that runs it. What is published:
# scene fingerprints, the short labels below (our own paraphrases, not game prose), choice text and outcomes.
version: 1
# Must match normalize() in the bot's src/strategy.js, so fingerprints line up (applied in this order).
normalize:
- ["[\u2018\u2019\u201b`']", "'"]
- ["[\u201c\u201d\"]", "\""]
- ["[\u2013\u2014-]", " "]
- ["\u2026", " "]
- ["[^a-z0-9'\\s]", " "]
- ["\\s+", " "]
# Events in the bot's log that record a choice it acted on
choice_events: [DECISION, MANUAL_DECISION, MANUAL_EDGE_PICK]
# runs.jsonl and the log are matched when the same attempt starts within this many hours
match_hours: 6
# Short public labels, by scene fingerprint. A scene without one is labelled "<stage> · <tint>" and listed by the
# importer so a label can be added here.
labels:
"db7786392626": "Start: choose company"
"a68f681076c6": "Valley edge: choose direction"
"bc4ee4754b9c": "First sighting: Red (forest centre)"
"abe3011378ac": "First sighting: Brown (forest centre)"
"84780dab01e3": "Survived the first sighting"
"d25f2759ace0": "Deeper in the forest: a scream"
"f72199f24b91": "Clearing: squadmate vs two bullies"
"b209bacacec3": "Bully fight: first move"
"fbe0f0bea700": "Bully fight: smaller cadet has a dagger"
"a2368b0dc212": "Blocked the bigger cadet: bullies’ fate"
"f4b4944d3f3b": "Beat the smaller cadet: bullies’ fate"
"0cfd06cf8939": "After the fight: choose a direction"
"0a561d53ad19": "An hour later: valley or tree"
"96b68612eaf4": "Top of the tree (shadow check)"
"10bc7353b8eb": "Descended quickly: a Red below"
"c62f1f1a2b3a": "Let the Red pass: two Greens in the clearing"
"07e24681e3b3": "Descended slowly: two Greens in the clearing"
# Crosswalk: a bot choice (option text, normalised as in core/graph.js norm()) -> a column of the coded runs.
# First match wins; a column keeps its first value. Unmatched choices are listed, never guessed.
# Two keys are not columns: scream_investigate (the scream step on its own) and red (the Red below the tree).
crosswalk:
- ["^enter the valley alone", company, "Alone"]
- ["^enter the valley with one of your squadmates", company, "1 Squadmate"]
- ["^enter the valley with two of your squadmates", company, "2 Squadmates"]
- ["^head east toward the roar", direction, "East"]
- ["^head west toward the scent", direction, "West"]
- ["^head toward the center of the forest", direction, "Center"]
- ["^keep walking the cadets are just as dangerous", scream, "Ignore the scream"]
- ["^run and hope", sighting_action, "Retreat / run"]
- ["hold still", sighting_action, "Hold still"]
- ["lower your eyes in respect|lower your eyes and turn to face it", sighting_action, "Respect / face it"]
- ["meeting its gaze", sighting_action, "Meet its gaze"]
- ["^hide behind a tree", sighting_action, "Hide"]
- ["^investigate the scream", scream_investigate, true]
- ["^help your squadmate", scream, "Investigate & help"]
- ["^play it safe and leave them", scream, "Investigate & leave"]
- ["^pause to assess the situation", combat_first, "Assess the situation"]
- ["^charge the other cadet", combat_first, "Even the numbers (smaller cadet)"]
- ["^unsheathe your weapon and charge between", combat_first, "Put yourself in the middle (bigger cadet)"]
- ["^engage the smaller cadet", combat_second, "Engage the smaller cadet"]
- ["^daggers or not", combat_second, "Daggers or not (protect squadmate)"]
- ["^end the lives of the bullies", resolution, "Killed the bullies yourself"]
- ["^allow your squadmate to exact", resolution, "Squadmate's revenge"]
- ["^leave the bullies alive", resolution, "Left bullies to the dragons"]
- ["tourniquet", resolution, "Tourniquet"]
# the wounded cadet's other two choices (the bot has not logged this screen yet, so these are kept loose)
- ["leave (them|him|her) to die", resolution, "Left injured cadet"]
- ["take (their|his|her) weapons?", resolution, "Took their weapons"]
- ["^walk toward the east", post_path, "East"]
- ["^walk toward the west", post_path, "West"]
- ["^walk farther into the forest", post_path, "Forest"]
- ["^continue toward the center of the valley", hour, "Valley"]
- ["^find the tallest tree", hour, "Tree"]
- ["^descend quickly", branch, "Descend quickly: Red"]
- ["^climb down slowly to pursue the green", branch, "Descend slowly: two Greens"]
- ["^reds are too hot tempered", red, "Let it pass (two Greens)"]
- ["^jump (now|in a few seconds)", red, "Jump onto the Red"]
+118
View File
@@ -0,0 +1,118 @@
# "Report a run": the questions of the GitHub issue form, and how each answer becomes a column of runs_clean.csv.
#
# collection/etl/report_form.py writes .github/ISSUE_TEMPLATE/report-a-run.yml from this file (a test checks it is
# up to date), and collection/etl/runs.py codes every report with it. Each option is "label shown: column value";
# an option with value null (or no answer) means "not known". Values must be listed in data/dictionary.yaml.
version: 1
name: Report a run
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
title: "Run report"
label: run report
intro: |
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
Please don't include your DragonKind account name or anything personal.
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
fields:
- id: goal
label: Which dragon were you trying to bond?
column: goal
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black, "Any, just for fun": Any (for fun) }
- id: attempt
label: Which attempt was this for your account? (a number; 1 = your first)
column: attempt
type: integer
- id: company
label: Who entered the valley with you?
column: company
required: true
options: { Alone: Alone, One squadmate: 1 Squadmate, Two squadmates: 2 Squadmates }
- id: left_squadmates
label: If you went with squadmates, did you leave them?
column: left_squadmates
options: { "Yes, I left them": true, "No, we stayed together": false, "I went alone": null }
- id: direction
label: Which way did you go first?
column: direction
required: true
options: { East (toward the roar): East, West (toward the scent): West, Toward the forest centre: Center }
- id: first_sighting
label: Which dragon found you first?
column: first_sighting
options: { Red: Red, Brown: Brown, Green: Green, Orange: Orange }
- id: sighting_action
label: What did you do when it found you?
column: sighting_action
options:
Hold still: Hold still
Lower your eyes in respect / turn to face it: Respect / face it
Run: Retreat / run
Hide: Hide
Draw a weapon: Weapon
Meet its gaze: Meet its gaze
Protect your squadmate: Protect squadmate
- id: scream
label: The scream
column: scream
required: true
options:
I died at the first dragon: DIED at first sighting
Investigated, and helped my squadmate: Investigate & help
Investigated, and walked away: Investigate & leave
Kept walking (ignored it): Ignore the scream
- id: combat_first
label: "If you helped: your first move in the fight"
column: combat_first
options:
Pause to assess: Assess the situation
Charge the smaller cadet: Even the numbers (smaller cadet)
Charge between the bigger cadet and your squadmate: Put yourself in the middle (bigger cadet)
- id: combat_second
label: "If you paused to assess: the smaller cadet's dagger"
column: combat_second
options: { Engage the smaller cadet: Engage the smaller cadet, Charge the bigger cadet anyway: Daggers or not (protect squadmate) }
- id: resolution
label: How did the cadet encounter end?
column: resolution
options:
I died in the fight: DIED in combat
"Bullies: killed them myself": Killed the bullies yourself
"Bullies: my squadmate's revenge": Squadmate's revenge
"Bullies: left them to the dragons": Left bullies to the dragons
"Wounded cadet: tourniquet": Tourniquet
"Wounded cadet: left them": Left injured cadet
"Wounded cadet: took their weapons": Took their weapons
"I walked away from the clearing": Left them to it
- id: post_path
label: "After the fight: which way did you walk?"
column: post_path
options: { Deeper into the forest: Forest, West: West, East: East, There was no fork: (no fork) }
- id: hour
label: After an hour's walk
column: hour
options: { Climbed the tallest tree: Tree, Went to the valley centre: Valley, I died before the hour: DIED before the hour }
- id: branch
label: Which bonding encounter did you reach?
column: branch
options:
"Tree: the shadow (Blue or Black)": Shadow (rare)
"Tree: climbed down quickly, jumped on the Red": "Descend quickly: Red"
"Tree: climbed down quickly, let the Red pass": "Descend quickly: let the Red pass"
"Tree: climbed down slowly to the two Greens": "Descend slowly: two Greens"
"Valley: faced the Red": Valley Red
"Valley: passed the Red, the meadow": "Passed the Red: meadow"
- id: outcome
label: How did the run end?
column: outcome
required: true
options: { Bonded: Bonded, Incinerated at a bonding encounter: Incinerated, Died before the hour: Died before the hour }
- id: outcome_color
label: Which dragon bonded you (or incinerated you)?
column: outcome_color
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black }
- id: tail
label: "If you bonded: the dragon's tail"
column: tail
options: { Clubtail: Clubtail, Daggertail: Daggertail, Morningstartail: Morningstartail, Scorpiontail: Scorpiontail, Swordtail: Swordtail }