Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+50
-13
@@ -1,19 +1,56 @@
|
||||
# collection
|
||||
|
||||
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
|
||||
under `data/`.
|
||||
Getting data in, as a strict **extract → transform → load** pipeline. Every source is declared in
|
||||
[`pipeline.yaml`](pipeline.yaml); the code only carries those declarations out.
|
||||
|
||||
| Source | Code | Output |
|
||||
```
|
||||
python -m collection update # every source: extract, then transform + load if it changed; then build
|
||||
python -m collection update --source reported --source register
|
||||
python -m collection update --file reddit_spreadsheet=~/Downloads/export.xlsx # a local file instead of the download
|
||||
python -m collection build # code every public source file into data/public/runs_clean.csv
|
||||
python -m collection build --check # nothing written; exit 1 if anything is out of date (CI runs this)
|
||||
```
|
||||
|
||||
## The stages
|
||||
|
||||
| Stage | Does | Never does |
|
||||
|---|---|---|
|
||||
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/atlas/coding.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
|
||||
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
|
||||
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
|
||||
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
|
||||
| Reddit megathreads | not collected yet | — |
|
||||
| **extract** (`etl/extract.py`) | Fetches the source as it is now into `data/raw/<source>/current.<ext>`. The version it replaces becomes `previous.<ext>`; nothing older is kept. `data/manifest.json` records the current raw file's sha256 and the day it last changed. | Change the data. |
|
||||
| **transform** (`etl/transform/`) | Turns the raw file into the public source file, applying the source's privacy rules. A pure function of the raw file: no network, no clock. | Write files. |
|
||||
| **validate** (`etl/validate.py`) | Checks the combined runs against `data/dictionary.yaml` (types, every value listed, unique ids). A failure stops the load and is reported. | Fix anything silently. |
|
||||
| **load** (`etl/load.py`) | The only stage that writes public files. Byte-stable: the same input always gives the same bytes, and a file is only rewritten when its bytes change. | Publish anything raw. |
|
||||
|
||||
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
|
||||
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
|
||||
merges itself when CI passes. A broken link opens an issue.
|
||||
`build` then codes every public source file into one run per row (`etl/runs.py`, one coder per source) and writes
|
||||
`data/public/runs_clean.csv`, plus the files generated from the specs: the "Report a run" issue form, the bot
|
||||
crosswalk inside `core/graph.js`, and `data/croissant.json`. Every input to `build` is committed, so CI rebuilds
|
||||
it exactly.
|
||||
|
||||
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
|
||||
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.
|
||||
## The sources
|
||||
|
||||
| `source` in runs_clean.csv | Pipeline source | Extract | Public source file |
|
||||
|---|---|---|---|
|
||||
| reddit spreadsheet | `reddit_spreadsheet`: u/ClumsyBadger's Threshing Routes form and spreadsheet, linked from the megathreads | the sheet's .xlsx export | `data/public/form_responses.json` (emails redacted), coded by `core/spec/form.yaml` |
|
||||
| reddit thread | `reddit_thread`: dragonkind.com result pages posted in the r/fourthwing megathreads | the threads, through the official Reddit API (needs `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`) | `manual/reddit_results.yaml`, checked by hand; the extract lists result links not in it yet |
|
||||
| reported | `reported`: the Atlas's "Report a run" form (a GitHub issue form, `specs/report_form.yaml`) | issues labelled `run report`, through the GitHub API | `data/public/reports.json` (answers only, no usernames) |
|
||||
| bot | `bot`: Ken's DragonKind bot | a local copy of the bot's private repo (`DRAGONKIND_BOT_DIR`) | `data/bot/scenes.json`, `data/bot/trajectories-<date>.jsonl` (no prose, accounts hashed with `DK_ACCOUNT_SALT`; `specs/bot.yaml`) |
|
||||
| reddit pdf | `reddit_pdf`: the community route PDF | the PDF | `data/derived/pdf_fingerprint.json` only: hashes, never the text |
|
||||
| — | `register`: the Unofficial Dragon Register | the sheet's .xlsx export | `data/public/register.json` (rider names and photos dropped) |
|
||||
|
||||
The route PDF is a flowchart of the choices, not a log of runs, so it adds no rows to `runs_clean.csv`; its
|
||||
fingerprint tells us when the routes it documents change. The register lists bonded dragons without the route that
|
||||
led there, so it adds no rows either.
|
||||
|
||||
## Raw files and privacy
|
||||
|
||||
`data/raw/` is git-ignored and never published. It holds the current raw file of each source and the one before
|
||||
it; older versions are not kept. The daily workflow keeps that store between runs in the Actions cache, encrypted
|
||||
with the `RAW_ARCHIVE_KEY` secret (see `docs/REPO_SETTINGS.md`). Privacy rules sit with each source in
|
||||
`pipeline.yaml`: emails redacted, register names and photos dropped, the bot's fields refused if they look like
|
||||
prose or account ids, GitHub and Reddit usernames never kept.
|
||||
|
||||
## Adding a source
|
||||
|
||||
1. Add a block to `pipeline.yaml` (extract kind, transform, privacy, load target, and `run_source` / `runs` if it
|
||||
has runs).
|
||||
2. If it needs a new transform or coder, add one module to `etl/transform/` and one function to `etl/runs.py`.
|
||||
3. Add its `source` value to `data/dictionary.yaml`, then `python -m collection build` and `npm test`.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
from collection.run import main
|
||||
|
||||
raise SystemExit(main())
|
||||
@@ -1,211 +0,0 @@
|
||||
'use strict';
|
||||
/**
|
||||
* Import DragonKind bot data into anonymised trajectories and a scene catalogue.
|
||||
*
|
||||
* Inputs (all from the PRIVATE bot repo, read locally, never committed):
|
||||
* data/knowledge.json scene atlas: nodes (prose, choices, tint, backdrop) and edges
|
||||
* data/runs.jsonl one record per attempt (newer bot versions)
|
||||
* logs/encounters.jsonl every decision the bot made, with the options on screen
|
||||
*
|
||||
* Outputs (public, CC BY 4.0): no game prose, only fingerprints, short labels and choice text.
|
||||
* scenes.json { sceneId: { stage, tint, backdrop, label, options: [{ id, text }] } }
|
||||
* trajectories.jsonl one line per attempt: { id, source, meta, steps: [...], outcome }
|
||||
*/
|
||||
const crypto = require('node:crypto');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const sha1 = (s, n = 12) => crypto.createHash('sha1').update(String(s)).digest('hex').slice(0, n);
|
||||
|
||||
// Must match normalize() in the bot's src/strategy.js, so fingerprints line up.
|
||||
function normalize(text) {
|
||||
return String(text || '')
|
||||
.toLowerCase()
|
||||
.replace(/[‘’‛`']/g, "'")
|
||||
.replace(/[“”"]/g, '"')
|
||||
.replace(/[–—-]/g, ' ')
|
||||
.replace(/…/g, ' ')
|
||||
.replace(/[^a-z0-9'\s]/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
// Must match computeSceneFingerprint() in the bot's src/learner.js.
|
||||
function sceneId(prose, choices) {
|
||||
const proseNorm = normalize(prose).slice(0, 500);
|
||||
const sorted = choices.map(normalize).filter(Boolean).sort();
|
||||
return sha1(`${proseNorm} ### ${sorted.join(' || ')}`);
|
||||
}
|
||||
|
||||
const optionId = (text) => sha1(normalize(text), 10);
|
||||
|
||||
const readJsonl = (file) => {
|
||||
if (!file || !fs.existsSync(file)) return [];
|
||||
return fs.readFileSync(file, 'utf8').split(/\r?\n/).filter((l) => l.trim()).map((l) => {
|
||||
try { return JSON.parse(l); } catch { return null; }
|
||||
}).filter(Boolean);
|
||||
};
|
||||
|
||||
/**
|
||||
* Short public labels. Keys are scene fingerprints; values are our own paraphrases, not game prose.
|
||||
* Unknown scenes fall back to "<stage> · <tint>" and are listed by the importer so labels can be added.
|
||||
*/
|
||||
const LABELS = {
|
||||
db7786392626: 'Start: choose company',
|
||||
a68f681076c6: 'Valley edge: choose direction',
|
||||
bc4ee4754b9c: 'First sighting: Red (forest centre)',
|
||||
abe3011378ac: 'First sighting: Brown (forest centre)',
|
||||
'84780dab01e3': 'Survived the first sighting',
|
||||
d25f2759ace0: 'Deeper in the forest: a scream',
|
||||
f72199f24b91: 'Clearing: squadmate vs two bullies',
|
||||
b209bacacec3: 'Bully fight: first move',
|
||||
fbe0f0bea700: 'Bully fight: smaller cadet has a dagger',
|
||||
a2368b0dc212: 'Blocked the bigger cadet: bullies’ fate',
|
||||
f4b4944d3f3b: 'Beat the smaller cadet: bullies’ fate',
|
||||
'0cfd06cf8939': 'After the fight: choose a direction',
|
||||
'0a561d53ad19': 'An hour later: valley or tree',
|
||||
'96b68612eaf4': 'Top of the tree (shadow check)',
|
||||
'10bc7353b8eb': 'Descended quickly: a Red below',
|
||||
c62f1f1a2b3a: 'Let the Red pass: two Greens in the clearing',
|
||||
'07e24681e3b3': 'Descended slowly: two Greens in the clearing'
|
||||
};
|
||||
|
||||
function labelFor(id, node) {
|
||||
if (LABELS[id]) return LABELS[id];
|
||||
const bits = [node.stage || 'Scene'];
|
||||
if (node.tint) bits.push(node.tint);
|
||||
return bits.join(' · ');
|
||||
}
|
||||
|
||||
/** Scene catalogue from knowledge.json (and, for scenes only in the log, from the log). */
|
||||
function buildScenes(knowledge, encounters) {
|
||||
const scenes = {};
|
||||
for (const [id, n] of Object.entries((knowledge && knowledge.nodes) || {})) {
|
||||
scenes[id] = {
|
||||
stage: n.stage || 'UNKNOWN',
|
||||
tint: n.tint || null,
|
||||
backdrop: n.backdrop || null,
|
||||
label: labelFor(id, n),
|
||||
options: Object.values(n.choices || {}).map((c) => ({ id: optionId(c.label || c.choiceNorm), text: c.label || c.choiceNorm }))
|
||||
};
|
||||
}
|
||||
for (const e of encounters) {
|
||||
if (!Array.isArray(e.choices) || !e.prose) continue;
|
||||
const id = e.nodeId || sceneId(e.prose, e.choices);
|
||||
if (scenes[id]) continue;
|
||||
const stage = (e.decision && e.decision.stage) || e.stage || 'UNKNOWN';
|
||||
scenes[id] = { stage, tint: null, backdrop: null, label: labelFor(id, { stage }), options: e.choices.map((t) => ({ id: optionId(t), text: t })) };
|
||||
}
|
||||
return scenes;
|
||||
}
|
||||
|
||||
const CHOICE_EVENTS = new Set(['DECISION', 'MANUAL_DECISION', 'MANUAL_EDGE_PICK']);
|
||||
|
||||
/**
|
||||
* Trajectories from the encounter log: group events by attempt, keep the decisions that were acted on,
|
||||
* and close each attempt at its first BURNED / DRAGON_BONDED event. Repeated BURNED events with no
|
||||
* decisions in between (the cooldown screen being re-detected) are ignored.
|
||||
*/
|
||||
function trajectoriesFromEncounters(encounters, scenes) {
|
||||
const byAttempt = new Map();
|
||||
for (const e of encounters) {
|
||||
const key = e.account ? `${e.account}:${e.attempt}` : String(e.attempt);
|
||||
if (!byAttempt.has(key)) byAttempt.set(key, []);
|
||||
byAttempt.get(key).push(e);
|
||||
}
|
||||
const out = [];
|
||||
for (const [key, events] of byAttempt) {
|
||||
events.sort((a, b) => String(a.ts).localeCompare(String(b.ts)));
|
||||
const steps = [];
|
||||
let outcome = null;
|
||||
for (const e of events) {
|
||||
if (outcome) break;
|
||||
if (CHOICE_EVENTS.has(e.event)) {
|
||||
const d = e.decision || {};
|
||||
const text = d.choiceText || e.choiceText;
|
||||
if (d.action && d.action !== 'choose') continue; // paused, nothing happened
|
||||
if (!text || !Array.isArray(e.choices)) continue;
|
||||
const scene = e.nodeId || sceneId(e.prose || '', e.choices);
|
||||
const manual = e.event !== 'DECISION';
|
||||
steps.push({
|
||||
scene,
|
||||
stage: d.stage || (scenes[scene] && scenes[scene].stage) || 'UNKNOWN',
|
||||
tint: (scenes[scene] && scenes[scene].tint) || null,
|
||||
option: optionId(text),
|
||||
optionText: text,
|
||||
offered: e.choices.map(optionId),
|
||||
rule: d.rule || null,
|
||||
decisionMode: manual ? 'manual' : d.source || null,
|
||||
manual,
|
||||
at: e.ts
|
||||
});
|
||||
} else if (e.event === 'BURNED' && steps.length) {
|
||||
outcome = { type: 'BURNED', colour: null, intentional: Boolean(e.intentional) };
|
||||
} else if (e.event === 'DRAGON_BONDED' && steps.length) {
|
||||
outcome = { type: 'BONDED', colour: e.dragon ? String(e.dragon).toLowerCase() : null, intentional: false };
|
||||
}
|
||||
}
|
||||
// A decision repeated on the same scene (the bot re-detecting the screen) counts once.
|
||||
const dedup = steps.filter((s, i) => !(i > 0 && steps[i - 1].scene === s.scene && steps[i - 1].option === s.option));
|
||||
for (let i = 0; i < dedup.length; i++) dedup[i].to = i + 1 < dedup.length ? dedup[i + 1].scene : null;
|
||||
if (!dedup.length) continue;
|
||||
const last = dedup[dedup.length - 1];
|
||||
if (outcome) last.to = outcome.type === 'BURNED' ? 'BURNED' : `BONDED:${outcome.colour || 'unknown'}`;
|
||||
out.push({
|
||||
id: `bot-${sha1(`${key}:${dedup[0].at}`, 10)}`,
|
||||
source: 'bot',
|
||||
meta: { attempt: Number(String(key).split(':').pop()) || null, startedAt: dedup[0].at, accountKey: null, target: null, decisionMode: null },
|
||||
steps: dedup,
|
||||
outcome: outcome || { type: 'ABORTED', colour: null, intentional: false }
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Fill run-level metadata (target colour, mode) from runs.jsonl where an attempt is covered by both. */
|
||||
function attachRunMeta(trajs, runs, salt) {
|
||||
for (const r of runs) {
|
||||
const t = trajs.find((x) => x.meta.attempt === r.attempt && Math.abs(Date.parse(x.meta.startedAt) - Date.parse(r.startedAt)) < 6 * 3600e3);
|
||||
if (!t) continue;
|
||||
t.meta.target = r.targetDragon || null;
|
||||
t.meta.decisionMode = r.decisionMode || null;
|
||||
t.meta.policyId = r.policyId || null;
|
||||
t.meta.experimentId = r.experimentId || null;
|
||||
t.meta.arm = r.arm || null;
|
||||
if (r.accountKey) t.meta.accountKey = r.accountKey;
|
||||
else if (r.accountId && salt) t.meta.accountKey = sha1(`${salt}:${r.accountId}`, 10);
|
||||
if (r.outcome === 'BONDED' && r.dragonColor) t.outcome = { type: 'BONDED', colour: String(r.dragonColor).toLowerCase(), intentional: false };
|
||||
if (r.intentionalBurn) t.outcome.intentional = true;
|
||||
}
|
||||
return trajs;
|
||||
}
|
||||
|
||||
function importBot({ dataDir, logsDir, salt = process.env.DK_ACCOUNT_SALT || null }) {
|
||||
const knowledgeFile = path.join(dataDir, 'knowledge.json');
|
||||
const knowledge = fs.existsSync(knowledgeFile) ? JSON.parse(fs.readFileSync(knowledgeFile, 'utf8')) : { nodes: {} };
|
||||
const encounters = readJsonl(path.join(logsDir, 'encounters.jsonl'));
|
||||
const runs = readJsonl(path.join(dataDir, 'runs.jsonl'));
|
||||
const scenes = buildScenes(knowledge, encounters);
|
||||
const trajectories = attachRunMeta(trajectoriesFromEncounters(encounters, scenes), runs, salt);
|
||||
const unlabeled = Object.entries(scenes).filter(([id]) => !LABELS[id]).map(([id, s]) => `${id} (${s.stage}${s.tint ? `, ${s.tint}` : ''})`);
|
||||
return { scenes, trajectories, unlabeled };
|
||||
}
|
||||
|
||||
/** Guard: nothing that looks like game prose or an account identifier may leave this module. */
|
||||
function assertPublicSafe(obj) {
|
||||
const banned = ['prose', 'proseNorm', 'proseExcerpt', 'email', 'password', 'accountId', 'accountName'];
|
||||
const walk = (v, p) => {
|
||||
if (Array.isArray(v)) return v.forEach((x, i) => walk(x, `${p}[${i}]`));
|
||||
if (v && typeof v === 'object') {
|
||||
for (const [k, x] of Object.entries(v)) {
|
||||
if (banned.includes(k)) throw new Error(`refusing to export field "${k}" at ${p}`);
|
||||
walk(x, `${p}.${k}`);
|
||||
}
|
||||
} else if (typeof v === 'string' && v.length > 200) {
|
||||
throw new Error(`refusing to export a ${v.length}-character string at ${p} (looks like prose)`);
|
||||
}
|
||||
};
|
||||
walk(obj, '$');
|
||||
}
|
||||
|
||||
module.exports = { normalize, sceneId, optionId, buildScenes, trajectoriesFromEncounters, attachRunMeta, importBot, assertPublicSafe, LABELS };
|
||||
@@ -1,54 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
/**
|
||||
* Convert a local copy of the DragonKind bot's data into the public, anonymised format.
|
||||
*
|
||||
* node collection/bot/import-bot.js --bot /path/to/DragonKind [--out data/bot] [--tag 2026-11-01]
|
||||
*
|
||||
* Reads <bot>/data/knowledge.json, <bot>/data/runs.jsonl and <bot>/logs/encounters.jsonl.
|
||||
* Writes <out>/scenes.json (merged with any existing catalogue) and <out>/trajectories-<tag>.jsonl.
|
||||
* Set DK_ACCOUNT_SALT in the environment to hash account ids; never commit the salt.
|
||||
*/
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { importBot, assertPublicSafe } = require('./bot');
|
||||
|
||||
const args = Object.fromEntries(process.argv.slice(2).reduce((acc, a, i, all) => {
|
||||
if (a.startsWith('--')) acc.push([a.slice(2), all[i + 1]]);
|
||||
return acc;
|
||||
}, []));
|
||||
if (!args.bot) {
|
||||
console.error('usage: import-bot.js --bot /path/to/DragonKind [--out data/bot] [--tag YYYY-MM-DD]');
|
||||
process.exit(2);
|
||||
}
|
||||
const out = path.resolve(args.out || 'data/bot');
|
||||
const tag = args.tag || new Date().toISOString().slice(0, 10);
|
||||
|
||||
const { scenes, trajectories, unlabeled } = importBot({
|
||||
dataDir: path.join(args.bot, 'data'),
|
||||
logsDir: path.join(args.bot, 'logs')
|
||||
});
|
||||
assertPublicSafe(scenes);
|
||||
assertPublicSafe(trajectories);
|
||||
|
||||
fs.mkdirSync(out, { recursive: true });
|
||||
const sceneFile = path.join(out, 'scenes.json');
|
||||
const existing = fs.existsSync(sceneFile) ? JSON.parse(fs.readFileSync(sceneFile, 'utf8')) : {};
|
||||
for (const [id, s] of Object.entries(scenes)) {
|
||||
const old = existing[id];
|
||||
if (!old) { existing[id] = s; continue; }
|
||||
const ids = new Set(old.options.map((o) => o.id));
|
||||
for (const o of s.options) if (!ids.has(o.id)) old.options.push(o);
|
||||
old.tint = old.tint || s.tint;
|
||||
old.backdrop = old.backdrop || s.backdrop;
|
||||
if (old.label.includes('·') && !s.label.includes('·')) old.label = s.label;
|
||||
}
|
||||
const sorted = Object.fromEntries(Object.entries(existing).sort(([a], [b]) => a.localeCompare(b)));
|
||||
fs.writeFileSync(sceneFile, `${JSON.stringify(sorted, null, 1)}\n`);
|
||||
const trajFile = path.join(out, `trajectories-${tag}.jsonl`);
|
||||
fs.writeFileSync(trajFile, trajectories.map((t) => JSON.stringify(t)).join('\n') + '\n');
|
||||
|
||||
const outcomes = trajectories.reduce((m, t) => ((m[t.outcome.type] = (m[t.outcome.type] || 0) + 1), m), {});
|
||||
console.log(`${Object.keys(scenes).length} scenes, ${trajectories.length} attempts ${JSON.stringify(outcomes)}`);
|
||||
console.log(`wrote ${path.relative(process.cwd(), sceneFile)} and ${path.relative(process.cwd(), trajFile)}`);
|
||||
if (unlabeled.length) console.log(`scenes without a short label (add them to LABELS in collection/bot/bot.js):\n ${unlabeled.join('\n ')}`);
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Files written from the declarative specs, so each rule has one source of truth:
|
||||
.github/ISSUE_TEMPLATE/report-a-run.yml from collection/specs/report_form.yaml (report_form.render)
|
||||
the CROSSWALK block of core/graph.js from collection/specs/bot.yaml (Ken's bot uses it at run time)
|
||||
"""
|
||||
import json
|
||||
|
||||
from .common import path, read_yaml
|
||||
|
||||
GRAPH_JS = "core/graph.js"
|
||||
BEGIN = " // BEGIN CROSSWALK"
|
||||
END = " // END CROSSWALK\n"
|
||||
|
||||
|
||||
def graph_js():
|
||||
"""core/graph.js with its CROSSWALK block rewritten from the bot spec."""
|
||||
text = path(GRAPH_JS).read_text(encoding="utf-8")
|
||||
a = text.index(BEGIN)
|
||||
a = text.index("\n", a) + 1
|
||||
b = text.index(END, a)
|
||||
rules = read_yaml("collection/specs/bot.yaml")["crosswalk"]
|
||||
lines = [" const CROSSWALK = ["]
|
||||
for i, (rx, col, val) in enumerate(rules):
|
||||
sep = "," if i < len(rules) - 1 else ""
|
||||
lines.append(f" [/{rx.replace('/', chr(92) + '/')}/, {json.dumps(col)}, {json.dumps(val, ensure_ascii=False)}]{sep}")
|
||||
lines.append(" ];")
|
||||
return text[:a] + "\n".join(lines) + "\n" + text[b:]
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Paths, the pipeline spec, and byte-stable writers shared by every stage."""
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(ROOT / "core") not in sys.path:
|
||||
sys.path.insert(0, str(ROOT / "core"))
|
||||
|
||||
|
||||
def path(p):
|
||||
return ROOT / p
|
||||
|
||||
|
||||
def read_yaml(p):
|
||||
with open(path(p), encoding="utf-8") as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
|
||||
def pipeline():
|
||||
return read_yaml("collection/pipeline.yaml")
|
||||
|
||||
|
||||
def sha256_bytes(data):
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def sha256_file(p):
|
||||
h = hashlib.sha256()
|
||||
with open(p, "rb") as f:
|
||||
for block in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def json_bytes(obj, compact=True):
|
||||
if compact:
|
||||
return (json.dumps(obj, ensure_ascii=False, separators=(",", ":"))).encode("utf-8")
|
||||
return (json.dumps(obj, ensure_ascii=False, indent=1) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def jsonl_bytes(rows):
|
||||
return "".join(json.dumps(r, ensure_ascii=False, separators=(",", ":")) + "\n" for r in rows).encode("utf-8")
|
||||
|
||||
|
||||
def read_jsonl(p):
|
||||
p = Path(p)
|
||||
if not p.exists():
|
||||
return []
|
||||
out = []
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
try:
|
||||
out.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
class Report:
|
||||
"""What a pipeline run did, as Markdown lines (the daily pull request's body)."""
|
||||
|
||||
def __init__(self):
|
||||
self.lines = []
|
||||
self.changed = False
|
||||
self.failed = []
|
||||
|
||||
def add(self, line):
|
||||
self.lines.append(line)
|
||||
print(line)
|
||||
|
||||
def text(self, title):
|
||||
return "\n".join([f"## {title}", ""] + self.lines) + "\n"
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Extract: fetch each source as it is now into the raw store. No transformation happens here.
|
||||
|
||||
The raw store is data/raw/<source>/ (git-ignored, never published):
|
||||
current.<ext> the latest version
|
||||
previous.<ext> the version before it (kept only in case it is needed; the daily workflow keeps it in an
|
||||
encrypted, private cache). Nothing older is kept and old versions are not hashed.
|
||||
data/manifest.json records, for each source, the sha256 of the current raw file and the date it last changed.
|
||||
|
||||
Each extractor returns the raw bytes, or raises Skip when the source cannot be reached from here (a missing
|
||||
secret, the bot's private repo not on this machine).
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from .common import path, sha256_bytes
|
||||
|
||||
USER_AGENT = "threshing-atlas-collector/1.0 (+https://github.com/ekcSchiano/threshing-atlas)"
|
||||
RAW_DIR = path("data/raw")
|
||||
|
||||
|
||||
class Skip(Exception):
|
||||
"""The source cannot be extracted here; not an error."""
|
||||
|
||||
|
||||
def _get(url, headers=None, data=None, timeout=60):
|
||||
req = urllib.request.Request(url, data=data, headers={"User-Agent": USER_AGENT, **(headers or {})})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return r.read()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- extractors, by `kind`
|
||||
|
||||
def http(spec, env):
|
||||
data = _get(spec["url"])
|
||||
head = data[:15].lower()
|
||||
if head.startswith(b"<!doctype html") or head.startswith(b"<html"):
|
||||
raise RuntimeError(f"{spec['url']} returned a web page, not a file (is the link still shared publicly?)")
|
||||
return data
|
||||
|
||||
|
||||
def reddit(spec, env):
|
||||
"""The threads' comments through the official Reddit API (app-only OAuth: a "script" app's id and secret)."""
|
||||
cid, secret = (env.get(k) for k in spec["secrets"])
|
||||
if not (cid and secret):
|
||||
raise Skip(f"set the {' and '.join(spec['secrets'])} secrets to read the threads through the Reddit API")
|
||||
auth = base64.b64encode(f"{cid}:{secret}".encode()).decode()
|
||||
token = json.loads(_get("https://www.reddit.com/api/v1/access_token",
|
||||
headers={"Authorization": f"Basic {auth}"},
|
||||
data=b"grant_type=client_credentials"))["access_token"]
|
||||
threads = {}
|
||||
for tid in spec["threads"]:
|
||||
threads[tid] = json.loads(_get(f"https://oauth.reddit.com/comments/{tid}?limit=500&depth=10&raw_json=1",
|
||||
headers={"Authorization": f"Bearer {token}"}))
|
||||
return json.dumps(threads, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
def github_issues(spec, env):
|
||||
"""Every issue with the label (open and closed), through the GitHub REST API."""
|
||||
headers = {"Accept": "application/vnd.github+json"}
|
||||
token = env.get("GITHUB_TOKEN") or env.get("GH_TOKEN")
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
label = urllib.parse.quote(spec["label"])
|
||||
issues, page = [], 1
|
||||
while True:
|
||||
url = f"https://api.github.com/repos/{spec['repo']}/issues?labels={label}&state=all&per_page=100&page={page}"
|
||||
batch = json.loads(_get(url, headers=headers))
|
||||
issues += [i for i in batch if "pull_request" not in i]
|
||||
if len(batch) < 100:
|
||||
break
|
||||
page += 1
|
||||
issues.sort(key=lambda i: i["number"])
|
||||
return json.dumps(issues, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
def local_dir(spec, env):
|
||||
"""The bot's files, from a local copy of its private repo (DRAGONKIND_BOT_DIR), packed as one JSON."""
|
||||
root = env.get(spec["env"])
|
||||
if not root:
|
||||
raise Skip(f"the bot's repo is private: set {spec['env']} to a local copy to import it")
|
||||
files = {}
|
||||
for rel in spec["files"]:
|
||||
p = os.path.join(root, rel)
|
||||
files[rel] = Path(p).read_text(encoding="utf-8") if os.path.exists(p) else None
|
||||
return json.dumps(files, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
EXTRACTORS = {"http": http, "reddit": reddit, "github_issues": github_issues, "local_dir": local_dir}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the raw store
|
||||
|
||||
def raw_paths(name, ext):
|
||||
d = RAW_DIR / name
|
||||
return d / f"current.{ext}", d / f"previous.{ext}"
|
||||
|
||||
|
||||
def store(name, ext, data, manifest, today=None):
|
||||
"""Keep `data` as the current raw file. Returns True when it differs from the last one seen."""
|
||||
current, previous = raw_paths(name, ext)
|
||||
digest = sha256_bytes(data)
|
||||
entry = manifest.get(name) or {}
|
||||
changed = entry.get("sha256") != digest
|
||||
current.parent.mkdir(parents=True, exist_ok=True)
|
||||
on_disk = sha256_bytes(current.read_bytes()) if current.exists() else None
|
||||
if on_disk and on_disk != digest:
|
||||
shutil.copyfile(current, previous) # keep one previous version; anything older is gone
|
||||
if on_disk != digest:
|
||||
fd, tmp = tempfile.mkstemp(dir=current.parent)
|
||||
with os.fdopen(fd, "wb") as f:
|
||||
f.write(data)
|
||||
os.replace(tmp, current)
|
||||
if changed:
|
||||
manifest[name] = {"sha256": digest, "bytes": len(data),
|
||||
"changed": today or datetime.now(timezone.utc).strftime("%Y-%m-%d")}
|
||||
return changed
|
||||
|
||||
|
||||
def extract(name, spec, manifest, env=None, local_file=None):
|
||||
"""Fetch one source (or take a local file) into the raw store. Returns (raw path, changed)."""
|
||||
env = os.environ if env is None else env
|
||||
ext = spec["extract"].get("ext") or "json"
|
||||
data = Path(local_file).read_bytes() if local_file else EXTRACTORS[spec["extract"]["kind"]](spec["extract"], env)
|
||||
changed = store(name, ext, data, manifest)
|
||||
return raw_paths(name, ext)[0], changed
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Load: the only stage that writes public files. Writes are byte-stable (the same table gives the same bytes)
|
||||
and only happen when the bytes change, so git shows exactly what changed."""
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
|
||||
from .common import json_bytes, jsonl_bytes, path
|
||||
|
||||
|
||||
def public(obj):
|
||||
"""Drop transient keys (those starting with "_"), which are for the report only."""
|
||||
if isinstance(obj, dict):
|
||||
return {k: public(v) for k, v in obj.items() if not str(k).startswith("_")}
|
||||
if isinstance(obj, list):
|
||||
return [public(v) for v in obj]
|
||||
return obj
|
||||
|
||||
|
||||
def read_json(rel):
|
||||
p = path(rel)
|
||||
return json.loads(p.read_text(encoding="utf-8")) if p.exists() else None
|
||||
|
||||
|
||||
def write_bytes(rel, data, check=False):
|
||||
"""Write when different. Returns True when the file changed (or, with check, would change)."""
|
||||
p = path(rel)
|
||||
if p.exists() and p.read_bytes() == data:
|
||||
return False
|
||||
if not check:
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_bytes(data)
|
||||
return True
|
||||
|
||||
|
||||
def json_file(rel, obj, check=False, compact=True):
|
||||
return write_bytes(rel, json_bytes(public(obj), compact), check)
|
||||
|
||||
|
||||
def jsonl_file(rel, rows, check=False):
|
||||
return write_bytes(rel, jsonl_bytes(rows), check)
|
||||
|
||||
|
||||
def csv_bytes(rows, columns):
|
||||
"""True/False for booleans, empty for missing, everything else as text."""
|
||||
buf = io.StringIO()
|
||||
w = csv.writer(buf, lineterminator="\n")
|
||||
w.writerow(columns)
|
||||
for r in rows:
|
||||
w.writerow(["" if r.get(c) is None else r[c] for c in columns])
|
||||
return buf.getvalue().encode("utf-8")
|
||||
|
||||
|
||||
def csv_file(rel, rows, columns, check=False):
|
||||
return write_bytes(rel, csv_bytes(rows, columns), check)
|
||||
@@ -3,20 +3,13 @@
|
||||
Croissant is the MLCommons metadata format for datasets: it says what each file is, its checksum, and what each
|
||||
column means, so tools (and people) can load and cite the data without reading this repo's code.
|
||||
|
||||
python3 collection/publish_metadata.py # write data/croissant.json
|
||||
python3 collection/publish_metadata.py --check # exit 1 if data/croissant.json is out of date
|
||||
Written by `python -m collection build` (a test checks it is up to date).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from .common import ROOT, pipeline, read_yaml, sha256_file
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DICTIONARY = ROOT / "data" / "dictionary.yaml"
|
||||
SOURCES = ROOT / "data" / "sources.json"
|
||||
OUT = ROOT / "data" / "croissant.json"
|
||||
OUT = "data/croissant.json"
|
||||
|
||||
CONTEXT = {
|
||||
"@language": "en",
|
||||
@@ -57,17 +50,11 @@ CONTEXT = {
|
||||
DATA_TYPES = {"text": "sc:Text", "integer": "sc:Integer", "boolean": "sc:Boolean", "date": "sc:Date"}
|
||||
|
||||
|
||||
def sha256(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
for block in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
sha256 = sha256_file
|
||||
|
||||
|
||||
def load_dictionary():
|
||||
with open(DICTIONARY, encoding="utf-8") as f:
|
||||
return yaml.safe_load(f)
|
||||
return read_yaml("data/dictionary.yaml")
|
||||
|
||||
|
||||
def field_description(spec):
|
||||
@@ -80,8 +67,7 @@ def field_description(spec):
|
||||
def build(dictionary=None):
|
||||
d = dictionary or load_dictionary()
|
||||
ds = d["dataset"]
|
||||
with open(SOURCES, encoding="utf-8") as f:
|
||||
sources = json.load(f)
|
||||
sources = pipeline()["sources"]
|
||||
|
||||
distribution = []
|
||||
for key, spec in d["files"].items():
|
||||
@@ -124,9 +110,10 @@ def build(dictionary=None):
|
||||
"citeAs": ds["citation"],
|
||||
"isLiveDataset": True,
|
||||
"isBasedOn": [
|
||||
{"@type": "sc:CreativeWork", "name": s["name"], "url": s["view"]}
|
||||
for key, s in sources.items()
|
||||
if key != "pdf" # the route PDF is only fingerprinted, never redistributed
|
||||
{"@type": "sc:CreativeWork", "name": s["label"], "url": next(iter(s["links"].values()))}
|
||||
for s in sources.values()
|
||||
# the route PDF is only fingerprinted, never redistributed; the bot's repo is private
|
||||
if (s.get("privacy") or {}).get("redistribute", True) and s.get("links")
|
||||
],
|
||||
"distribution": distribution,
|
||||
"recordSet": [
|
||||
@@ -134,7 +121,7 @@ def build(dictionary=None):
|
||||
"@type": "cr:RecordSet",
|
||||
"@id": "runs",
|
||||
"name": "runs",
|
||||
"description": "One coded threshing run per row (data/derived/runs_clean.csv).",
|
||||
"description": "One coded threshing run per row, from every source (data/public/runs_clean.csv).",
|
||||
"key": {"@id": "runs/run_id"},
|
||||
"field": fields,
|
||||
}
|
||||
@@ -144,20 +131,3 @@ def build(dictionary=None):
|
||||
|
||||
def render(doc):
|
||||
return json.dumps(doc, indent=2, ensure_ascii=False) + "\n"
|
||||
|
||||
|
||||
def main(argv):
|
||||
text = render(build())
|
||||
if "--check" in argv:
|
||||
if not OUT.exists() or OUT.read_text(encoding="utf-8") != text:
|
||||
print("data/croissant.json is out of date: run python3 collection/publish_metadata.py", file=sys.stderr)
|
||||
return 1
|
||||
print("data/croissant.json is up to date")
|
||||
return 0
|
||||
OUT.write_text(text, encoding="utf-8")
|
||||
print(f"wrote {OUT.relative_to(ROOT)}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,60 @@
|
||||
"""The "Report a run" GitHub issue form, written from collection/specs/report_form.yaml, and the parser for the
|
||||
issues it creates (GitHub renders each answer as "### <question>" followed by the answer)."""
|
||||
import re
|
||||
|
||||
import yaml
|
||||
|
||||
from .common import read_yaml
|
||||
|
||||
|
||||
class _Dumper(yaml.SafeDumper):
|
||||
pass
|
||||
|
||||
|
||||
def _str(dumper, s):
|
||||
# multi-line text (the intro) as a readable block
|
||||
return dumper.represent_scalar("tag:yaml.org,2002:str", s, style="|" if "\n" in s else None)
|
||||
|
||||
|
||||
_Dumper.add_representer(str, _str)
|
||||
|
||||
SPEC_PATH = "collection/specs/report_form.yaml"
|
||||
TEMPLATE = ".github/ISSUE_TEMPLATE/report-a-run.yml"
|
||||
NOT_SURE = "Not sure"
|
||||
NO_RESPONSE = {"", "_No response_", "None", NOT_SURE}
|
||||
|
||||
|
||||
def spec():
|
||||
return read_yaml(SPEC_PATH)
|
||||
|
||||
|
||||
def render(s=None):
|
||||
s = s or spec()
|
||||
body = [{"type": "markdown", "attributes": {"value": s["intro"]}}]
|
||||
for f in s["fields"]:
|
||||
if f.get("type") == "integer":
|
||||
item = {"type": "input", "id": f["id"], "attributes": {"label": f["label"], "placeholder": "e.g. 3"}}
|
||||
else:
|
||||
opts = list(f["options"])
|
||||
if not f.get("required"):
|
||||
opts.append(NOT_SURE)
|
||||
item = {"type": "dropdown", "id": f["id"], "attributes": {"label": f["label"], "options": opts}}
|
||||
item["validations"] = {"required": bool(f.get("required"))}
|
||||
body.append(item)
|
||||
form = {"name": s["name"], "description": s["description"], "title": s["title"], "labels": [s["label"]], "body": body}
|
||||
head = f"# Written by collection/etl/report_form.py from {SPEC_PATH}: edit that file, then run `python -m collection build`.\n"
|
||||
return head + yaml.dump(form, Dumper=_Dumper, sort_keys=False, allow_unicode=True, width=1000)
|
||||
|
||||
|
||||
def parse_body(body, s=None):
|
||||
"""Issue body -> {field id: chosen option label or text}. Unanswered questions are left out."""
|
||||
s = s or spec()
|
||||
by_label = {f["label"].strip(): f["id"] for f in s["fields"]}
|
||||
out = {}
|
||||
for block in re.split(r"^### ", body or "", flags=re.M)[1:]:
|
||||
q, _, a = block.partition("\n")
|
||||
fid = by_label.get(q.strip())
|
||||
a = a.strip()
|
||||
if fid and a not in NO_RESPONSE:
|
||||
out[fid] = a
|
||||
return out
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Build: every public source file -> one coded run per row, for data/public/runs_clean.csv.
|
||||
|
||||
One coder per source (pipeline.yaml `runs:`). Every coder returns rows keyed by the columns of data/dictionary.yaml;
|
||||
missing answers get the same "not known" values the community form's coding uses, so every source can be filtered
|
||||
the same way. The `source` column says where a run came from; `entry` keeps the spreadsheet's own row label.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from .common import path, read_jsonl, read_yaml
|
||||
from . import report_form
|
||||
|
||||
DICT = read_yaml("data/dictionary.yaml")
|
||||
COLUMNS = list(DICT["columns"])
|
||||
COLOURS = ["Red", "Green", "Orange", "Brown", "Blue", "Black"]
|
||||
|
||||
# what "not known" is, per column, for sources that did not ask (the community form's coding uses the same)
|
||||
NOT_KNOWN = {
|
||||
"wing": "Unknown", "squad": "Unknown", "attempt_group": "Unknown", "goal": "(not asked/blank)",
|
||||
"company": "Unknown", "direction": "Unknown", "first_sighting": "Unknown", "sighting_action": "Unknown",
|
||||
"scream": "Unknown", "resolution": "Unknown", "combat_first": "(not recorded)", "combat_second": "(not recorded)",
|
||||
"post_path": "Unknown", "hour": "Unknown", "rare_reported": "(not asked)", "malek_intent": "(not asked)",
|
||||
"outcome": "No outcome recorded", "data_flags": "",
|
||||
}
|
||||
|
||||
|
||||
def blank_row(**values):
|
||||
row = {c: None for c in COLUMNS}
|
||||
row.update(values)
|
||||
for c, v in NOT_KNOWN.items():
|
||||
if row[c] is None:
|
||||
row[c] = v
|
||||
return row
|
||||
|
||||
|
||||
def attempt_group(n):
|
||||
return "First attempt" if n == 1 else "Later attempt" if n and n > 1 else "Unknown"
|
||||
|
||||
|
||||
def encounter_columns(row, intentional=False):
|
||||
"""The bonding-grid column of the dragon that ended the run."""
|
||||
col = row["outcome_color"]
|
||||
if col and not intentional:
|
||||
if row["outcome"] == "Bonded":
|
||||
row["enc_" + col] = "Bonded!"
|
||||
elif row["outcome"] == "Incinerated":
|
||||
row["enc_" + col] = "Incinerated"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- coders, by pipeline.yaml `runs:`
|
||||
|
||||
def spreadsheet(name, spec):
|
||||
"""The community form: core/spec/form.yaml + core/atlas/coding.py, unchanged."""
|
||||
from atlas import coding
|
||||
df, missing = coding.code_frame(coding.read_export(path(spec["load"])))
|
||||
if missing:
|
||||
print(f"WARNING: {name}: questions not found in the sheet: {missing}")
|
||||
df["attempt"] = df["attempt"].astype("Int64")
|
||||
rows = []
|
||||
for rec in df.astype(object).where(df.notna(), None).to_dict("records"):
|
||||
n = rec.pop("run_id")
|
||||
entry = rec.pop("source")
|
||||
rows.append({**{c: None for c in COLUMNS}, **rec, "run_id": f"sheet-{n}", "source": spec["run_source"], "entry": entry})
|
||||
return rows
|
||||
|
||||
|
||||
def reddit_results(name, spec):
|
||||
"""Dragons read from result pages linked in the megathreads: bonded, colour and tail known, route unknown."""
|
||||
rows = []
|
||||
for r in read_yaml(spec["transform"]["manual"])["results"]:
|
||||
row = blank_row(run_id=f"thread-{r['id']}", source=spec["run_source"], outcome="Bonded",
|
||||
outcome_color=r["colour"], tail=r["tail"])
|
||||
encounter_columns(row)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def reports(name, spec):
|
||||
"""Runs reported through the "Report a run" issue form (collection/specs/report_form.yaml)."""
|
||||
form = report_form.spec()
|
||||
fields = {f["id"]: f for f in form["fields"]}
|
||||
p = path(spec["load"])
|
||||
data = json.loads(p.read_text(encoding="utf-8")) if p.exists() else {"reports": []}
|
||||
rows = []
|
||||
for rep in data["reports"]:
|
||||
vals, flags = {}, []
|
||||
for fid, answer in rep["answers"].items():
|
||||
f = fields.get(fid)
|
||||
if not f:
|
||||
continue
|
||||
if f.get("type") == "integer":
|
||||
m = re.search(r"\d+", answer)
|
||||
vals[f["column"]] = int(m.group()) if m else None
|
||||
elif answer in f["options"]:
|
||||
vals[f["column"]] = f["options"][answer]
|
||||
else:
|
||||
flags.append(f"unrecognised answer to {fid}")
|
||||
row = blank_row(run_id=f"reported-{rep['number']}", source=spec["run_source"], date=rep.get("opened"), **vals)
|
||||
row["attempt_group"] = attempt_group(row["attempt"])
|
||||
if row["outcome"] == "Died before the hour" and row["hour"] == "Unknown":
|
||||
row["hour"] = "DIED before the hour"
|
||||
branch = row["branch"]
|
||||
if branch:
|
||||
tree = branch in ("Shadow (rare)", "Descend quickly: Red", "Descend quickly: let the Red pass", "Descend slowly: two Greens")
|
||||
if row["hour"] == "Unknown":
|
||||
row["hour"] = "Tree" if tree else "Valley"
|
||||
elif row["hour"] != ("Tree" if tree else "Valley"):
|
||||
flags.append("encounter does not match the tree/valley answer")
|
||||
row["saw_shadow"] = branch == "Shadow (rare)" if tree else None
|
||||
if branch == "Passed the Red: meadow" and row["outcome_color"] == "Blue":
|
||||
row["saw_blue_meadow"] = True
|
||||
if row["outcome"] == "Bonded" and not row["outcome_color"]:
|
||||
flags.append("bonded, colour not given")
|
||||
encounter_columns(row)
|
||||
row["data_flags"] = "; ".join(flags)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def _norm(text):
|
||||
from .transform.bot import normalize
|
||||
return normalize(text)
|
||||
|
||||
|
||||
def bot_trajectories(spec):
|
||||
"""Every published trajectory (data/bot/trajectories-*.jsonl), each attempt once."""
|
||||
seen, out = set(), []
|
||||
for f in sorted(Path(path(spec["load"])).glob("trajectories-*.jsonl")):
|
||||
for t in read_jsonl(f):
|
||||
if t["id"] not in seen:
|
||||
seen.add(t["id"])
|
||||
out.append(t)
|
||||
return out
|
||||
|
||||
|
||||
def bot_variables(traj, crosswalk):
|
||||
"""A bot attempt -> columns, through the crosswalk in collection/specs/bot.yaml. First match wins."""
|
||||
v, unmapped = {}, []
|
||||
for s in traj["steps"]:
|
||||
t = _norm(s["optionText"])
|
||||
hit = next((c for c in crosswalk if c[0].search(t)), None)
|
||||
if hit:
|
||||
v.setdefault(hit[1], hit[2])
|
||||
else:
|
||||
unmapped.append(s["optionText"])
|
||||
# the first dragon is the first tinted scene once the direction is chosen
|
||||
if v.get("direction") and not v.get("first_sighting") and s.get("tint"):
|
||||
v["first_sighting"] = s["tint"].capitalize()
|
||||
return v, unmapped
|
||||
|
||||
|
||||
def bot(name, spec):
|
||||
cw = [(re.compile(rx), col, val) for rx, col, val in read_yaml(spec["transform"]["spec"])["crosswalk"]]
|
||||
rows = []
|
||||
for traj in bot_trajectories(spec):
|
||||
v, _ = bot_variables(traj, cw)
|
||||
red = v.pop("red", None)
|
||||
v.pop("scream_investigate", None)
|
||||
if red == "Let it pass (two Greens)":
|
||||
v["branch"] = "Descend quickly: let the Red pass"
|
||||
o = traj["outcome"]
|
||||
meta = traj.get("meta") or {}
|
||||
flags = []
|
||||
if o["type"] == "BONDED":
|
||||
outcome, colour = "Bonded", (o.get("colour") or "").capitalize() or None
|
||||
elif o["type"] == "BURNED":
|
||||
last_tint = traj["steps"][-1].get("tint")
|
||||
outcome = "Incinerated" if v.get("hour") else "Died before the hour"
|
||||
colour = last_tint.capitalize() if last_tint and v.get("hour") else None
|
||||
if o.get("intentional"):
|
||||
flags.append("deliberate burn (the bot chose to die)")
|
||||
else:
|
||||
outcome, colour = "No outcome recorded", None
|
||||
target = (meta.get("target") or "").capitalize() or None
|
||||
row = blank_row(run_id=traj["id"], source=spec["run_source"], date=(meta.get("startedAt") or "")[:10] or None,
|
||||
attempt=meta.get("attempt"), goal=target if target in COLOURS else None,
|
||||
outcome=outcome, outcome_color=colour, data_flags="; ".join(flags), **v)
|
||||
row["attempt_group"] = attempt_group(row["attempt"])
|
||||
encounter_columns(row, intentional=bool(o.get("intentional")))
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
CODERS = {"spreadsheet": spreadsheet, "reddit_results": reddit_results, "reports": reports, "bot": bot}
|
||||
|
||||
|
||||
def build_runs(pipe):
|
||||
rows = []
|
||||
for name in pipe["runs"]["order"]:
|
||||
spec = pipe["sources"][name]
|
||||
if spec.get("runs"):
|
||||
rows += CODERS[spec["runs"]](name, spec)
|
||||
return rows
|
||||
@@ -0,0 +1,7 @@
|
||||
"""Named transforms (pipeline.yaml `transform: {name: ...}`). Each module has
|
||||
transform(raw_path, spec, ctx) -> the public source file's content (pure: no network, no clock)
|
||||
describe(old, new) -> report lines comparing the previous public file with the new one
|
||||
"""
|
||||
from . import bot, pdf, reddit, reports, sheet
|
||||
|
||||
TRANSFORMS = {"sheet": sheet, "pdf_fingerprint": pdf, "bot": bot, "reddit_results": reddit, "reports": reports}
|
||||
@@ -0,0 +1,239 @@
|
||||
"""The DragonKind bot's private data -> anonymised trajectories and a scene catalogue (Python port of bot.js).
|
||||
|
||||
Raw input (from the private bot repo, packed by extract.local_dir): data/knowledge.json (scene atlas: prose,
|
||||
choices, tint, backdrop), data/runs.jsonl (one record per attempt) and logs/encounters.jsonl (every decision the
|
||||
bot made, with the options on screen). Nothing of the prose or the account ids leaves this module: the output
|
||||
holds scene fingerprints, short labels (collection/specs/bot.yaml), choice text and outcomes, and check_public()
|
||||
refuses anything else.
|
||||
|
||||
Output: {"scenes": {sceneId: {stage, tint, backdrop, label, options: [{id, text}]}},
|
||||
"trajectories": [{id, source, meta, steps, outcome}], "unlabeled": [...]}
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
from ..common import read_yaml
|
||||
|
||||
SPEC = read_yaml("collection/specs/bot.yaml")
|
||||
_NORMALIZE = [(re.compile(a), b) for a, b in SPEC["normalize"]]
|
||||
CHOICE_EVENTS = set(SPEC["choice_events"])
|
||||
|
||||
|
||||
def sha1(s, n=12):
|
||||
return hashlib.sha1(str(s).encode("utf-8")).hexdigest()[:n]
|
||||
|
||||
|
||||
def normalize(text):
|
||||
"""Must match normalize() in the bot's src/strategy.js, so fingerprints line up."""
|
||||
t = str(text or "").lower()
|
||||
for rx, rep in _NORMALIZE:
|
||||
t = rx.sub(rep, t)
|
||||
return t.strip()
|
||||
|
||||
|
||||
def scene_id(prose, choices):
|
||||
"""Must match computeSceneFingerprint() in the bot's src/learner.js."""
|
||||
prose_norm = normalize(prose)[:500]
|
||||
opts = sorted(c for c in (normalize(c) for c in choices) if c)
|
||||
return sha1(f"{prose_norm} ### {' || '.join(opts)}")
|
||||
|
||||
|
||||
def option_id(text):
|
||||
return sha1(normalize(text), 10)
|
||||
|
||||
|
||||
def label_for(sid, node):
|
||||
if sid in SPEC["labels"]:
|
||||
return SPEC["labels"][sid]
|
||||
bits = [node.get("stage") or "Scene"]
|
||||
if node.get("tint"):
|
||||
bits.append(node["tint"])
|
||||
return " · ".join(bits)
|
||||
|
||||
|
||||
def _jsonl(text):
|
||||
out = []
|
||||
for line in (text or "").splitlines():
|
||||
if line.strip():
|
||||
try:
|
||||
out.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def build_scenes(knowledge, encounters):
|
||||
scenes = {}
|
||||
for sid, n in ((knowledge or {}).get("nodes") or {}).items():
|
||||
scenes[sid] = {
|
||||
"stage": n.get("stage") or "UNKNOWN", "tint": n.get("tint"), "backdrop": n.get("backdrop"),
|
||||
"label": label_for(sid, n),
|
||||
"options": [{"id": option_id(c.get("label") or c.get("choiceNorm")), "text": c.get("label") or c.get("choiceNorm")}
|
||||
for c in (n.get("choices") or {}).values()],
|
||||
}
|
||||
for e in encounters:
|
||||
if not isinstance(e.get("choices"), list) or not e.get("prose"):
|
||||
continue
|
||||
sid = e.get("nodeId") or scene_id(e["prose"], e["choices"])
|
||||
if sid in scenes:
|
||||
continue
|
||||
stage = (e.get("decision") or {}).get("stage") or e.get("stage") or "UNKNOWN"
|
||||
scenes[sid] = {"stage": stage, "tint": None, "backdrop": None, "label": label_for(sid, {"stage": stage}),
|
||||
"options": [{"id": option_id(t), "text": t} for t in e["choices"]]}
|
||||
return scenes
|
||||
|
||||
|
||||
def trajectories_from_encounters(encounters, scenes):
|
||||
"""Group the log by attempt, keep the decisions acted on, and close each attempt at its first BURNED or
|
||||
DRAGON_BONDED event. Repeated BURNED events with no decision between them are ignored."""
|
||||
by_attempt = {}
|
||||
for e in encounters:
|
||||
key = f"{e['account']}:{e.get('attempt')}" if e.get("account") else str(e.get("attempt"))
|
||||
by_attempt.setdefault(key, []).append(e)
|
||||
out = []
|
||||
for key, events in by_attempt.items():
|
||||
events = sorted(events, key=lambda e: str(e.get("ts")))
|
||||
steps, outcome = [], None
|
||||
for e in events:
|
||||
if outcome:
|
||||
break
|
||||
if e.get("event") in CHOICE_EVENTS:
|
||||
d = e.get("decision") or {}
|
||||
text = d.get("choiceText") or e.get("choiceText")
|
||||
if d.get("action") and d["action"] != "choose":
|
||||
continue # paused, nothing happened
|
||||
if not text or not isinstance(e.get("choices"), list):
|
||||
continue
|
||||
scene = e.get("nodeId") or scene_id(e.get("prose") or "", e["choices"])
|
||||
manual = e["event"] != "DECISION"
|
||||
sc = scenes.get(scene) or {}
|
||||
steps.append({
|
||||
"scene": scene, "stage": d.get("stage") or sc.get("stage") or "UNKNOWN", "tint": sc.get("tint"),
|
||||
"option": option_id(text), "optionText": text, "offered": [option_id(c) for c in e["choices"]],
|
||||
"rule": d.get("rule"), "decisionMode": "manual" if manual else d.get("source"),
|
||||
"manual": manual, "at": e.get("ts"),
|
||||
})
|
||||
elif e.get("event") == "BURNED" and steps:
|
||||
outcome = {"type": "BURNED", "colour": None, "intentional": bool(e.get("intentional"))}
|
||||
elif e.get("event") == "DRAGON_BONDED" and steps:
|
||||
outcome = {"type": "BONDED", "colour": str(e["dragon"]).lower() if e.get("dragon") else None, "intentional": False}
|
||||
# a decision repeated on the same scene (the bot re-detecting the screen) counts once
|
||||
dedup = [s for i, s in enumerate(steps)
|
||||
if not (i > 0 and steps[i - 1]["scene"] == s["scene"] and steps[i - 1]["option"] == s["option"])]
|
||||
for i, s in enumerate(dedup):
|
||||
s["to"] = dedup[i + 1]["scene"] if i + 1 < len(dedup) else None
|
||||
if not dedup:
|
||||
continue
|
||||
if outcome:
|
||||
dedup[-1]["to"] = "BURNED" if outcome["type"] == "BURNED" else f"BONDED:{outcome['colour'] or 'unknown'}"
|
||||
try:
|
||||
attempt = int(float(key.split(":")[-1])) or None
|
||||
except ValueError:
|
||||
attempt = None
|
||||
started = dedup[0]["at"]
|
||||
out.append({
|
||||
"id": "bot-" + sha1(f"{key}:{started}", 10),
|
||||
"source": "bot",
|
||||
"meta": {"attempt": attempt, "startedAt": dedup[0]["at"], "accountKey": None, "target": None, "decisionMode": None},
|
||||
"steps": dedup,
|
||||
"outcome": outcome or {"type": "ABORTED", "colour": None, "intentional": False},
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _ms(ts):
|
||||
try:
|
||||
return datetime.fromisoformat(str(ts).replace("Z", "+00:00")).timestamp() * 1000
|
||||
except ValueError:
|
||||
return float("nan")
|
||||
|
||||
|
||||
def attach_run_meta(trajs, runs, salt):
|
||||
"""Run-level metadata (target colour, mode) from runs.jsonl, where an attempt is in both."""
|
||||
window = SPEC["match_hours"] * 3600e3
|
||||
for r in runs:
|
||||
t = next((x for x in trajs if x["meta"]["attempt"] == r.get("attempt")
|
||||
and abs(_ms(x["meta"]["startedAt"]) - _ms(r.get("startedAt"))) < window), None)
|
||||
if not t:
|
||||
continue
|
||||
m = t["meta"]
|
||||
m["target"] = r.get("targetDragon")
|
||||
m["decisionMode"] = r.get("decisionMode")
|
||||
m["policyId"] = r.get("policyId")
|
||||
m["experimentId"] = r.get("experimentId")
|
||||
m["arm"] = r.get("arm")
|
||||
if r.get("accountKey"):
|
||||
m["accountKey"] = r["accountKey"]
|
||||
elif r.get("accountId") and salt:
|
||||
m["accountKey"] = sha1(f"{salt}:{r['accountId']}", 10)
|
||||
if r.get("outcome") == "BONDED" and r.get("dragonColor"):
|
||||
t["outcome"] = {"type": "BONDED", "colour": str(r["dragonColor"]).lower(), "intentional": False}
|
||||
if r.get("intentionalBurn"):
|
||||
t["outcome"]["intentional"] = True
|
||||
return trajs
|
||||
|
||||
|
||||
def import_bot(knowledge, encounters, runs, salt=None):
|
||||
scenes = build_scenes(knowledge, encounters)
|
||||
trajectories = attach_run_meta(trajectories_from_encounters(encounters, scenes), runs, salt)
|
||||
unlabeled = [f"{sid} ({s['stage']}{', ' + s['tint'] if s['tint'] else ''})" for sid, s in scenes.items() if sid not in SPEC["labels"]]
|
||||
return scenes, trajectories, unlabeled
|
||||
|
||||
|
||||
def check_public(obj, refuse, max_string):
|
||||
"""Nothing that looks like game prose or an account identifier may leave this module."""
|
||||
def walk(v, p):
|
||||
if isinstance(v, list):
|
||||
for i, x in enumerate(v):
|
||||
walk(x, f"{p}[{i}]")
|
||||
elif isinstance(v, dict):
|
||||
for k, x in v.items():
|
||||
if k in refuse:
|
||||
raise ValueError(f'refusing to export field "{k}" at {p}')
|
||||
walk(x, f"{p}.{k}")
|
||||
elif isinstance(v, str) and len(v) > max_string:
|
||||
raise ValueError(f"refusing to export a {len(v)}-character string at {p} (looks like prose)")
|
||||
walk(obj, "$")
|
||||
|
||||
|
||||
def merge_scenes(existing, scenes):
|
||||
"""Add new scenes and new options to the published catalogue; never drop what is already there."""
|
||||
out = json.loads(json.dumps(existing or {}))
|
||||
for sid, s in scenes.items():
|
||||
old = out.get(sid)
|
||||
if not old:
|
||||
out[sid] = s
|
||||
continue
|
||||
ids = {o["id"] for o in old["options"]}
|
||||
old["options"] += [o for o in s["options"] if o["id"] not in ids]
|
||||
old["tint"] = old.get("tint") or s["tint"]
|
||||
old["backdrop"] = old.get("backdrop") or s["backdrop"]
|
||||
if "·" in old["label"] and "·" not in s["label"]:
|
||||
old["label"] = s["label"]
|
||||
return dict(sorted(out.items()))
|
||||
|
||||
|
||||
def transform(raw_path, spec, ctx):
|
||||
with open(raw_path, encoding="utf-8") as f:
|
||||
files = json.load(f)
|
||||
k = files.get("data/knowledge.json")
|
||||
knowledge = json.loads(k) if k else {"nodes": {}}
|
||||
encounters = _jsonl(files.get("logs/encounters.jsonl"))
|
||||
runs = _jsonl(files.get("data/runs.jsonl"))
|
||||
privacy = spec.get("privacy") or {}
|
||||
scenes, trajectories, unlabeled = import_bot(knowledge, encounters, runs, ctx.get("env", {}).get(privacy.get("account_salt_env", "")))
|
||||
for obj in (scenes, trajectories):
|
||||
check_public(obj, set(privacy.get("refuse_fields", [])), privacy.get("max_string", 200))
|
||||
return {"scenes": scenes, "trajectories": trajectories, "_unlabeled": unlabeled}
|
||||
|
||||
|
||||
def describe(old, new):
|
||||
outcomes = {}
|
||||
for t in new["trajectories"]:
|
||||
outcomes[t["outcome"]["type"]] = outcomes.get(t["outcome"]["type"], 0) + 1
|
||||
lines = [f"{len(new['scenes'])} scenes, {len(new['trajectories'])} attempts {json.dumps(outcomes)}"]
|
||||
if new.get("_unlabeled"):
|
||||
lines.append(" - Scenes without a short label (add them to collection/specs/bot.yaml): " + ", ".join(new["_unlabeled"]))
|
||||
return lines
|
||||
@@ -0,0 +1,51 @@
|
||||
"""The route PDF -> its fingerprint only. The PDF is never redistributed: the public file holds hashes of the file,
|
||||
of each page's text and of each line, so a change can be detected and located without keeping the text."""
|
||||
import hashlib
|
||||
|
||||
|
||||
def _sha(text, n):
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:n]
|
||||
|
||||
|
||||
def pdf_lines(path):
|
||||
from pypdf import PdfReader
|
||||
pages = []
|
||||
for page in PdfReader(str(path)).pages:
|
||||
text = page.extract_text() or ""
|
||||
pages.append([" ".join(line.split()) for line in text.splitlines() if line.strip()])
|
||||
return pages
|
||||
|
||||
|
||||
def transform(raw_path, spec, ctx):
|
||||
with open(raw_path, "rb") as f:
|
||||
data = f.read()
|
||||
pages = pdf_lines(raw_path)
|
||||
return {
|
||||
"sha256": hashlib.sha256(data).hexdigest(),
|
||||
"bytes": len(data),
|
||||
"pages": len(pages),
|
||||
"page_hashes": [_sha("\n".join(p), 16) for p in pages],
|
||||
"line_hashes": sorted({_sha(line.lower(), 12) for p in pages for line in p}),
|
||||
"checked": ctx.get("changed"),
|
||||
# transient: the new lines' text, for the report only (load drops every key starting with "_")
|
||||
"_lines": [line for p in pages for line in p],
|
||||
}
|
||||
|
||||
|
||||
def describe(old, new):
|
||||
if old is None:
|
||||
return [f"fingerprint recorded for the first time ({new['pages']} pages)"]
|
||||
if old.get("page_hashes") == new["page_hashes"]:
|
||||
return ["the file changed but its text did not (layout or images only). Worth a look."]
|
||||
changed = [i + 1 for i, h in enumerate(new["page_hashes"])
|
||||
if i >= len(old["page_hashes"]) or old["page_hashes"][i] != h]
|
||||
old_lines = set(old["line_hashes"])
|
||||
new_lines = [line for line in new.get("_lines", []) if _sha(line.lower(), 12) not in old_lines]
|
||||
lines = [f"changed. {old['pages']} → {new['pages']} pages; pages with new text: {', '.join(map(str, changed)) or 'none'}; "
|
||||
f"{len(new_lines)} new lines, {len(old_lines - set(new['line_hashes']))} lines gone."]
|
||||
if new_lines:
|
||||
lines.append(" - New lines (update core/spec/game.yaml or the crosswalks if they add options):")
|
||||
lines += [f" - {line}" for line in new_lines[:60]]
|
||||
if len(new_lines) > 60:
|
||||
lines.append(f" - … and {len(new_lines) - 60} more")
|
||||
return lines
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Reddit megathreads (raw: the threads' comment trees from the official API) -> dragonkind.com result links that
|
||||
are not yet in collection/manual/reddit_results.yaml. Usernames and comment text are never kept: only the result
|
||||
ids and the comment permalinks, for the report, so a person can read the result page and add it by hand."""
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from ..common import read_yaml
|
||||
|
||||
RESULT_RE = re.compile(r"dragonkind\.com/results/(THRESH-[A-Z0-9]+)", re.I)
|
||||
|
||||
|
||||
def _comments(node):
|
||||
"""Every comment in a listing tree."""
|
||||
if isinstance(node, list):
|
||||
for x in node:
|
||||
yield from _comments(x)
|
||||
elif isinstance(node, dict):
|
||||
data = node.get("data") or {}
|
||||
if node.get("kind") in ("t1", "t3"):
|
||||
yield data
|
||||
for key in ("children", "replies"):
|
||||
if isinstance(data.get(key), (dict, list)):
|
||||
yield from _comments(data[key])
|
||||
|
||||
|
||||
def transform(raw_path, spec, ctx):
|
||||
threads = json.loads(Path(raw_path).read_text(encoding="utf-8"))
|
||||
known = {r["id"].upper() for r in read_yaml(spec["transform"]["manual"])["results"]}
|
||||
found = {}
|
||||
for tid, tree in sorted(threads.items()):
|
||||
for c in _comments(tree):
|
||||
for m in RESULT_RE.finditer((c.get("body") or "") + " " + (c.get("selftext") or "") + " " + (c.get("url") or "")):
|
||||
rid = m.group(1).upper()
|
||||
if rid not in known and rid not in found:
|
||||
found[rid] = "https://www.reddit.com" + (c.get("permalink") or f"/comments/{tid}/")
|
||||
return {"new_results": [{"id": k, "comment": v} for k, v in sorted(found.items())]}
|
||||
|
||||
|
||||
def describe(old, new):
|
||||
rs = new["new_results"]
|
||||
if not rs:
|
||||
return ["no new result links in the threads"]
|
||||
return [f"{len(rs)} result link(s) not yet in collection/manual/reddit_results.yaml (read each page, then add it):"] + \
|
||||
[f" - https://dragonkind.com/results/{r['id']} (posted at {r['comment']})" for r in rs[:40]]
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Run reports (raw: the "run report" issues from the GitHub API) -> data/public/reports.json.
|
||||
|
||||
Kept per report: the issue number, the date it was opened, and the answers (the option labels chosen). GitHub
|
||||
usernames and any other text are dropped. Reports closed as "not planned" or carrying a reject label (e.g.
|
||||
invalid, duplicate) are left out, so a maintainer removes a bad report by closing it that way."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from .. import report_form
|
||||
|
||||
|
||||
def transform(raw_path, spec, ctx):
|
||||
issues = json.loads(Path(raw_path).read_text(encoding="utf-8"))
|
||||
reject = set(spec["transform"].get("reject_labels", []))
|
||||
form = report_form.spec()
|
||||
reports = []
|
||||
for i in sorted(issues, key=lambda i: i["number"]):
|
||||
labels = {lb["name"] if isinstance(lb, dict) else lb for lb in i.get("labels") or []}
|
||||
if labels & reject or i.get("state_reason") == "not_planned":
|
||||
continue
|
||||
answers = report_form.parse_body(i.get("body"), form)
|
||||
if not answers:
|
||||
continue
|
||||
reports.append({"number": i["number"], "opened": (i.get("created_at") or "")[:10] or None, "answers": answers})
|
||||
return {"source": spec["label"], "reports": reports}
|
||||
|
||||
|
||||
def describe(old, new):
|
||||
o = {r["number"] for r in (old or {}).get("reports", [])}
|
||||
n = {r["number"] for r in new["reports"]}
|
||||
return [f"{len(n)} report(s) ({len(n - o)} new, {len(o - n)} removed)"]
|
||||
@@ -0,0 +1,75 @@
|
||||
"""A Google Sheets export (.xlsx or .csv) -> the public source file: the sheet's own questions and answers, with the
|
||||
source's privacy rules applied. No recoding happens here (that is core/atlas/coding.py, at build time)."""
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from ..common import read_yaml
|
||||
|
||||
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
|
||||
|
||||
|
||||
def load_sheet(path, sheet_contains="form responses"):
|
||||
if str(path).lower().endswith(".csv"):
|
||||
return pd.read_csv(path)
|
||||
xl = pd.ExcelFile(path)
|
||||
sheet = next((n for n in xl.sheet_names if sheet_contains in n.lower()), xl.sheet_names[0])
|
||||
return pd.read_excel(xl, sheet)
|
||||
|
||||
|
||||
def cell(v, redact=True):
|
||||
if v is None or v is pd.NaT or (not isinstance(v, str) and pd.isna(v)):
|
||||
return None
|
||||
if isinstance(v, (datetime, pd.Timestamp)):
|
||||
return pd.Timestamp(v).strftime("%Y-%m-%d %H:%M:%S")
|
||||
if isinstance(v, float) and v.is_integer():
|
||||
return int(v)
|
||||
if isinstance(v, str) and redact:
|
||||
return EMAIL_RE.sub("[email removed]", v) # respondents sometimes leave an email in free text
|
||||
return v
|
||||
|
||||
|
||||
def extra_results(manual):
|
||||
"""Dragons read from result pages linked on Reddit (collection/manual/reddit_results.yaml)."""
|
||||
return [{"source": "dragonkind.com result page linked on Reddit", "url": f"https://dragonkind.com/results/{r['id']}",
|
||||
"name": r["name"], "colour": r["colour"], "tail": r["tail"], "sex": r["sex"], "age": r["age"],
|
||||
"lineage": r["lineage"]} for r in read_yaml(manual)["results"]]
|
||||
|
||||
|
||||
def transform(raw_path, spec, ctx):
|
||||
t, privacy = spec["transform"], spec.get("privacy") or {}
|
||||
raw = load_sheet(raw_path, t.get("sheet_contains", "form responses"))
|
||||
drop = [d.lower() for d in privacy.get("drop_columns_matching", [])]
|
||||
raw = raw[[c for c in raw.columns if not any(d in str(c).lower() for d in drop)]]
|
||||
redact = privacy.get("redact_emails", True)
|
||||
rows = []
|
||||
for _, r in raw.iterrows():
|
||||
row = {}
|
||||
for k, v in r.items():
|
||||
c = cell(v, redact)
|
||||
if c is not None:
|
||||
row[str(k)] = c
|
||||
rows.append(row)
|
||||
out = {"source": spec["label"], "exported": ctx.get("changed"),
|
||||
# every header, even one blank in every row, so the coding rules see every question
|
||||
"headers": [str(c) for c in raw.columns], "rows": rows}
|
||||
if t.get("extra"):
|
||||
out["extra"] = extra_results(t["extra"])
|
||||
return out
|
||||
|
||||
|
||||
def describe(old, new):
|
||||
"""One report line (plus details) comparing the previous public file with the new one."""
|
||||
if old is None:
|
||||
return [f"first export: {len(new['rows'])} rows"]
|
||||
key = lambda r: repr(sorted(r.items())) # noqa: E731
|
||||
o, n = {key(r) for r in old["rows"]}, {key(r) for r in new["rows"]}
|
||||
lines = [f"{len(old['rows'])} → {len(new['rows'])} rows ({len(n - o)} new or edited, {len(o - n)} removed or edited)"]
|
||||
new_q = [h for h in new["headers"] if h not in old["headers"]]
|
||||
gone_q = [h for h in old["headers"] if h not in new["headers"]]
|
||||
if new_q:
|
||||
lines.append(f" - New questions (add them to `core/spec/form.yaml`): {', '.join(new_q)}")
|
||||
if gone_q:
|
||||
lines.append(f" - Questions no longer in the sheet: {', '.join(gone_q)}")
|
||||
return lines
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Validate: a table is checked against data/dictionary.yaml before it is loaded. A failure stops the load."""
|
||||
import re
|
||||
|
||||
CHECKS = {
|
||||
"integer": lambda v: isinstance(v, int) and not isinstance(v, bool) or bool(re.fullmatch(r"-?\d+", str(v))),
|
||||
"boolean": lambda v: isinstance(v, bool) or v in ("True", "False"),
|
||||
"date": lambda v: bool(re.fullmatch(r"\d{4}-\d{2}-\d{2}", str(v))),
|
||||
"text": lambda v: isinstance(v, str),
|
||||
}
|
||||
|
||||
|
||||
class Invalid(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def problems(rows, dictionary, header=None):
|
||||
"""Every way the rows break the dictionary, as readable lines (empty when the table is valid)."""
|
||||
cols = dictionary["columns"]
|
||||
out = []
|
||||
if header is not None and list(header) != list(cols):
|
||||
out.append(f"columns differ from the dictionary: {list(header)} vs {list(cols)}")
|
||||
ids = [r.get("run_id") for r in rows]
|
||||
if len(set(ids)) != len(ids):
|
||||
out.append("run_id is not unique")
|
||||
for name, col in cols.items():
|
||||
vals = [r.get(name) for r in rows]
|
||||
present = [v for v in vals if v is not None and v != ""]
|
||||
bad_type = sorted({str(v) for v in present if not CHECKS[col["type"]](v)})
|
||||
if bad_type:
|
||||
out.append(f"{name}: not {col['type']}: {bad_type[:5]}")
|
||||
if "values" in col:
|
||||
allowed = {str(v) for v in col["values"]}
|
||||
bad = sorted({str(v) for v in present if str(v) not in allowed})
|
||||
if bad:
|
||||
out.append(f"{name}: values not in the dictionary: {bad[:10]}")
|
||||
if col.get("required") and len(present) != len(vals):
|
||||
out.append(f"{name}: required, but {len(vals) - len(present)} rows are empty")
|
||||
return out
|
||||
|
||||
|
||||
def check(rows, dictionary, what="table"):
|
||||
errs = problems(rows, dictionary)
|
||||
if errs:
|
||||
raise Invalid(f"{what} does not match data/dictionary.yaml:\n " + "\n ".join(errs))
|
||||
@@ -0,0 +1,19 @@
|
||||
# Dragons from dragonkind.com result pages that riders linked in the r/fourthwing megathreads, read by hand.
|
||||
# Each is one run in runs_clean.csv (source "reddit thread": bonded, colour and tail known, route not recorded) and
|
||||
# appears under "extra" in data/public/register.json. The reddit_thread source lists result links found in the
|
||||
# threads that are not here yet; add them after reading the result page. No usernames are kept.
|
||||
version: 1
|
||||
read: "2026-10-19"
|
||||
results:
|
||||
- { id: THRESH-VFRQVM5K3B0CP, name: Thiduach, colour: Blue, tail: Swordtail, sex: Female, age: 51, lineage: Gormfaileas }
|
||||
- { id: THRESH-3M4P9MNFQGBS8, name: Caenuis, colour: Blue, tail: Swordtail, sex: Female, age: 38, lineage: Gormfaileas }
|
||||
- { id: THRESH-5JQDJ21G1BYKR, name: Oavuagh, colour: Green, tail: Morningstartail, sex: Female, age: 43, lineage: Uaineloidsig }
|
||||
- { id: THRESH-5KRAAVFB3D1MG, name: Geagios, colour: Blue, tail: Clubtail, sex: Male, age: 99, lineage: Gormfaileas }
|
||||
- { id: THRESH-B5HSM00J4A2Z4, name: Fiedheil, colour: Black, tail: Clubtail, sex: Male, age: 32, lineage: Moerlaighen }
|
||||
- { id: THRESH-BMPM2GAPK58JT, name: Teirfua, colour: Green, tail: Morningstartail, sex: Female, age: 68, lineage: Eifyrielle }
|
||||
- { id: THRESH-MCQ8D6GAGF4K8, name: Caollo, colour: Blue, tail: Morningstartail, sex: Female, age: 93, lineage: Nyghmuiron }
|
||||
- { id: THRESH-Q0XDBFMNBEB86, name: Aenminn, colour: Red, tail: Swordtail, sex: Male, age: 59, lineage: Berradhyr }
|
||||
- { id: THRESH-W99DB29MG16QA, name: Meinso, colour: Green, tail: Clubtail, sex: Female, age: 63, lineage: Eifyrielle }
|
||||
- { id: THRESH-XCJCY7HW7QSSP, name: Einmial, colour: Blue, tail: Clubtail, sex: Male, age: 58, lineage: Gormfaileas }
|
||||
- { id: THRESH-XR6G68VPGT48M, name: Recuagh, colour: Brown, tail: Morningstartail, sex: Female, age: 55, lineage: Vormalgh }
|
||||
- { id: THRESH-YQC97TK34214E, name: Dannidh, colour: Orange, tail: Clubtail, sex: Male, age: 61, lineage: Cariaslog }
|
||||
@@ -0,0 +1,89 @@
|
||||
# Data collection as extract -> transform -> load, one block per source. collection/run.py reads this file.
|
||||
#
|
||||
# extract fetches the source as it is now. The raw file goes to data/raw/<source>/current.<ext> (git-ignored,
|
||||
# never published); the version it replaces is kept as previous.<ext>, and the daily workflow keeps
|
||||
# that previous copy in an encrypted, private cache. Nothing older is kept.
|
||||
# transform turns the raw file into the public source file: a pure function of the raw file (no network, no
|
||||
# clock), with the source's privacy rules applied. Named transforms live in collection/etl/transform/.
|
||||
# load writes the public source file (byte-stable: same input, same bytes) and reports what changed.
|
||||
#
|
||||
# Then `build` codes every public source file into one run per row (runs: the coder) and writes
|
||||
# data/public/runs_clean.csv: every run from every source, with a `source` column. Each table is checked against
|
||||
# data/dictionary.yaml before it is written. Every input to `build` is committed, so CI can rebuild it exactly.
|
||||
version: 1
|
||||
|
||||
raw_dir: data/raw
|
||||
manifest: data/manifest.json # for each source: the sha256 of the current raw file and when it last changed
|
||||
|
||||
runs:
|
||||
path: data/public/runs_clean.csv
|
||||
# sources in this order; within a source, the order the coder returns
|
||||
order: [reddit_spreadsheet, reddit_thread, reported, bot, reddit_pdf]
|
||||
|
||||
sources:
|
||||
reddit_spreadsheet:
|
||||
label: Threshing Routes form and spreadsheet (u/ClumsyBadger, linked from the r/fourthwing megathreads)
|
||||
run_source: reddit spreadsheet
|
||||
links:
|
||||
sheet: https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0
|
||||
form: null # the megathread's Google Form (responder link); shown on "Report a run" when set
|
||||
extract: { kind: http, url: "https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0/export?format=xlsx", ext: xlsx }
|
||||
transform: { name: sheet, sheet_contains: form responses }
|
||||
privacy: { redact_emails: true }
|
||||
load: data/public/form_responses.json
|
||||
runs: spreadsheet # core/spec/form.yaml + core/atlas/coding.py
|
||||
|
||||
register:
|
||||
label: Unofficial Dragon Register (which dragon each rider bonded)
|
||||
links: { sheet: https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ }
|
||||
extract: { kind: http, url: "https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ/export?format=xlsx", ext: xlsx }
|
||||
transform: { name: sheet, sheet_contains: form responses, extra: collection/manual/reddit_results.yaml }
|
||||
privacy: { redact_emails: true, drop_columns_matching: [photo, cadet name] } # no rider names, no photos
|
||||
load: data/public/register.json
|
||||
runs: null # bonded dragons without the route that led there
|
||||
|
||||
reddit_pdf:
|
||||
label: Community route PDF (u/ClumsyBadger)
|
||||
run_source: reddit pdf
|
||||
links: { view: https://drive.google.com/file/d/17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg/view }
|
||||
extract: { kind: http, url: "https://drive.google.com/uc?export=download&id=17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg", ext: pdf }
|
||||
transform: { name: pdf_fingerprint }
|
||||
privacy: { redistribute: false } # only hashes of the file, its pages and its lines are kept
|
||||
load: data/derived/pdf_fingerprint.json
|
||||
runs: null # a flowchart of the routes, not a log of runs (see collection/README.md)
|
||||
|
||||
reddit_thread:
|
||||
label: Runs posted in the r/fourthwing Dragonkind megathreads
|
||||
run_source: reddit thread
|
||||
links:
|
||||
part3: https://www.reddit.com/r/fourthwing/comments/1wvwfp9/
|
||||
part4: https://www.reddit.com/r/fourthwing/comments/1wz5cnj/
|
||||
# the official Reddit API (app-only OAuth); skipped with a note when the secrets are not set
|
||||
extract: { kind: reddit, threads: [1wvwfp9, 1wz5cnj], secrets: [REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET], ext: json }
|
||||
# lists dragonkind.com result links that are not in the manual file yet; usernames are never kept
|
||||
transform: { name: reddit_results, manual: collection/manual/reddit_results.yaml }
|
||||
load: null # the public source file is the hand-checked collection/manual/reddit_results.yaml
|
||||
runs: reddit_results
|
||||
|
||||
reported:
|
||||
label: Runs reported on the Atlas ("Report a run", a GitHub issue form)
|
||||
run_source: reported
|
||||
links: { report: "https://github.com/ekcSchiano/threshing-atlas/issues/new?template=report-a-run.yml" }
|
||||
extract: { kind: github_issues, repo: ekcSchiano/threshing-atlas, label: run report, ext: json }
|
||||
# GitHub usernames are dropped; issues closed as "not planned" or labelled "invalid" are left out
|
||||
transform: { name: reports, form: collection/specs/report_form.yaml, reject_labels: [invalid, duplicate] }
|
||||
load: data/public/reports.json
|
||||
runs: reports
|
||||
|
||||
bot:
|
||||
label: DragonKind bot runs (Ken's private bot, imported locally)
|
||||
run_source: bot
|
||||
# the bot's repo is private: this step runs only on a machine with a copy of it
|
||||
extract: { kind: local_dir, env: DRAGONKIND_BOT_DIR, files: [data/knowledge.json, data/runs.jsonl, logs/encounters.jsonl] }
|
||||
transform: { name: bot, spec: collection/specs/bot.yaml }
|
||||
privacy:
|
||||
account_salt_env: DK_ACCOUNT_SALT # account ids are hashed with this salt, which is never committed
|
||||
refuse_fields: [prose, proseNorm, proseExcerpt, email, password, accountId, accountName]
|
||||
max_string: 200 # anything longer looks like game prose
|
||||
load: data/bot # scenes.json (merged) + trajectories-<date>.jsonl
|
||||
runs: bot
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Data collection: extract -> transform -> load for every source, then build the combined runs.
|
||||
|
||||
python -m collection update [--source NAME ...] [--file NAME=PATH ...] [--report FILE]
|
||||
For each source (all by default): extract it (or take a local file), and when the raw file changed,
|
||||
transform it and load the public source file. Then build. Exit code 0 even when a source could not be
|
||||
reached: the report says what happened (and the daily workflow opens an issue).
|
||||
python -m collection build [--check]
|
||||
Code every public source file into data/public/runs_clean.csv (validated against data/dictionary.yaml),
|
||||
and write the files generated from the specs: the "Report a run" issue form, the bot crosswalk in
|
||||
core/graph.js, and data/croissant.json. With --check nothing is written; exit 1
|
||||
if any of them is out of date (CI runs this).
|
||||
|
||||
Sources and their rules: collection/pipeline.yaml. What each column means: data/dictionary.yaml.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from .etl import extract as E
|
||||
from .etl import load as L
|
||||
from .etl import codegen, metadata, report_form, runs, validate
|
||||
from .etl.common import Report, path, pipeline
|
||||
from .etl.transform import TRANSFORMS
|
||||
|
||||
|
||||
def update_source(name, spec, manifest, report, local_file=None, env=None):
|
||||
env = os.environ if env is None else env
|
||||
try:
|
||||
raw, changed = E.extract(name, spec, manifest, env, local_file)
|
||||
except E.Skip as e:
|
||||
report.add(f"- **{spec['label']}:** skipped ({e}).")
|
||||
return False
|
||||
except Exception as e: # one broken source must not block the others
|
||||
report.failed.append(name)
|
||||
report.add(f"- **{spec['label']}:** could not extract ({type(e).__name__}: {e}).")
|
||||
return False
|
||||
if not changed:
|
||||
report.add(f"- **{spec['label']}:** no change.")
|
||||
return False
|
||||
mod = TRANSFORMS[spec["transform"]["name"]]
|
||||
ctx = {"changed": manifest[name]["changed"], "env": env}
|
||||
new = mod.transform(raw, spec, ctx)
|
||||
target = spec.get("load")
|
||||
if name == "bot":
|
||||
old = {"scenes": L.read_json(f"{target}/scenes.json") or {}}
|
||||
lines = mod.describe(old, new)
|
||||
wrote = L.json_file(f"{target}/scenes.json", mod.merge_scenes(old["scenes"], new["scenes"]), compact=False)
|
||||
wrote |= L.jsonl_file(f"{target}/trajectories-{ctx['changed']}.jsonl", new["trajectories"])
|
||||
elif target:
|
||||
old = L.read_json(target)
|
||||
lines = mod.describe(old, new)
|
||||
wrote = L.json_file(target, new, compact=not target.startswith("data/derived"))
|
||||
else: # nothing to load: the report is the output (e.g. Reddit result links to add by hand)
|
||||
lines, wrote = mod.describe(None, new), False
|
||||
report.add(f"- **{spec['label']}:** {lines[0]}")
|
||||
for line in lines[1:]:
|
||||
report.add(line)
|
||||
return wrote
|
||||
|
||||
|
||||
def build(check=False, report=None):
|
||||
"""Combined runs, issue form and metadata. Returns the list of files that changed (or would change)."""
|
||||
pipe = pipeline()
|
||||
rows = runs.build_runs(pipe)
|
||||
validate.check(rows, runs.DICT, "runs_clean.csv")
|
||||
out = []
|
||||
if L.csv_file(pipe["runs"]["path"], rows, runs.COLUMNS, check):
|
||||
out.append(pipe["runs"]["path"])
|
||||
if L.write_bytes(report_form.TEMPLATE, report_form.render().encode("utf-8"), check):
|
||||
out.append(report_form.TEMPLATE)
|
||||
if L.write_bytes(codegen.GRAPH_JS, codegen.graph_js().encode("utf-8"), check):
|
||||
out.append(codegen.GRAPH_JS)
|
||||
# the metadata hashes the files above, so it is built from what is (or would be) on disk
|
||||
if not check or not out:
|
||||
if L.write_bytes(metadata.OUT, metadata.render(metadata.build()).encode("utf-8"), check):
|
||||
out.append(metadata.OUT)
|
||||
counts = {}
|
||||
for r in rows:
|
||||
counts[r["source"]] = counts.get(r["source"], 0) + 1
|
||||
msg = f"runs_clean.csv: {len(rows)} runs ({', '.join(f'{k} {v}' for k, v in counts.items())})"
|
||||
(report.add if report else print)(f"- **Combined runs:** {msg}")
|
||||
return out
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(prog="python -m collection", description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
up = sub.add_parser("update")
|
||||
up.add_argument("--source", action="append", help="only these sources (repeatable)")
|
||||
up.add_argument("--file", action="append", default=[], help="NAME=PATH: use a local file instead of extracting")
|
||||
up.add_argument("--report")
|
||||
b = sub.add_parser("build")
|
||||
b.add_argument("--check", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
|
||||
if a.cmd == "build":
|
||||
changed = build(check=a.check)
|
||||
if a.check and changed:
|
||||
print("out of date (run `python -m collection build`): " + ", ".join(changed), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
pipe = pipeline()
|
||||
manifest = L.read_json(pipe["manifest"]) or {}
|
||||
files = dict(f.split("=", 1) for f in a.file)
|
||||
report = Report()
|
||||
for name, spec in pipe["sources"].items():
|
||||
if a.source and name not in a.source:
|
||||
continue
|
||||
report.changed |= update_source(name, spec, manifest, report, files.get(name))
|
||||
L.json_file(pipe["manifest"], dict(sorted(manifest.items())), compact=False)
|
||||
try:
|
||||
report.changed |= bool(build(report=report))
|
||||
except validate.Invalid as e: # new answers the coding rules do not know yet: report them, load nothing
|
||||
report.failed.append("build")
|
||||
report.add(f"- **Combined runs not rebuilt:** {e}")
|
||||
report.add(" - Add the new answers to `core/spec/form.yaml` (or `data/dictionary.yaml`), then run `python -m collection build`.")
|
||||
text = report.text(f"Source check {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}")
|
||||
if a.report:
|
||||
with open(a.report, "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
if os.environ.get("GITHUB_OUTPUT"):
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
|
||||
f.write(f"changed={'true' if report.changed else 'false'}\nfailed={','.join(report.failed)}\n")
|
||||
return 0
|
||||
@@ -1,49 +0,0 @@
|
||||
"""export_raw.py — convert the Google Sheets .xlsx (or .csv) export into the raw JSON the web page loads.
|
||||
|
||||
python export_raw.py ../../data/raw/form/Threshing_Routes_Responses.xlsx ../../data/public/form_responses.json
|
||||
|
||||
The JSON is a list of objects keyed by the original question text (no recoding happens here, so the page
|
||||
applies exactly the same cleaning rules to it as it would to a freshly uploaded export).
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
import re
|
||||
|
||||
import pandas as pd
|
||||
|
||||
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
|
||||
|
||||
|
||||
def load_raw(path):
|
||||
if str(path).lower().endswith(".csv"):
|
||||
return pd.read_csv(path)
|
||||
xl = pd.ExcelFile(path)
|
||||
sheet = next((n for n in xl.sheet_names if "form responses" in n.lower()), xl.sheet_names[0])
|
||||
return pd.read_excel(xl, sheet)
|
||||
|
||||
|
||||
def cell(v):
|
||||
if v is None or v is pd.NaT or (not isinstance(v, str) and pd.isna(v)):
|
||||
return None
|
||||
if isinstance(v, (datetime, pd.Timestamp)):
|
||||
return pd.Timestamp(v).strftime("%Y-%m-%d %H:%M:%S")
|
||||
if isinstance(v, float) and v.is_integer():
|
||||
return int(v)
|
||||
if isinstance(v, str):
|
||||
return EMAIL_RE.sub("[email removed]", v) # respondents sometimes leave an email in free text
|
||||
return v
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
src = sys.argv[1] if len(sys.argv) > 1 else "../../data/raw/form/Threshing_Routes_Responses.xlsx"
|
||||
out = sys.argv[2] if len(sys.argv) > 2 else "../../data/public/form_responses.json"
|
||||
raw = load_raw(src)
|
||||
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
|
||||
# keep every header present even if a row is blank there, so column detection sees all questions
|
||||
headers = [str(c) for c in raw.columns]
|
||||
with open(out, "w", encoding="utf-8") as f:
|
||||
json.dump({"source_file": str(src).split("/")[-1], "exported": datetime.now().strftime("%Y-%m-%d"),
|
||||
"headers": headers, "rows": rows}, f, ensure_ascii=False, separators=(",", ":"))
|
||||
print(f"{len(rows)} rows -> {out}")
|
||||
@@ -1,43 +0,0 @@
|
||||
"""export_register.py — convert the "Unofficial Dragon Register" .xlsx/.csv export into web/data/register.json.
|
||||
|
||||
python export_register.py ../../data/raw/register/Unofficial_Dragon_Register_Responses.xlsx ../../data/public/register.json
|
||||
|
||||
Raw rows are kept as-is, except that photo links and rider names are dropped. Dragons read from public dragonkind.com result pages
|
||||
linked in the Reddit megathreads are stored separately under "extra" with their source URL.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from export_raw import cell, load_raw
|
||||
|
||||
# Read from dragonkind.com/results/<id> pages linked in the Part 3 and Part 4 megathreads (2026-10-19).
|
||||
REDDIT_RESULTS = [
|
||||
("THRESH-VFRQVM5K3B0CP", "Thiduach", "Blue", "Swordtail", "Female", 51, "Gormfaileas"),
|
||||
("THRESH-3M4P9MNFQGBS8", "Caenuis", "Blue", "Swordtail", "Female", 38, "Gormfaileas"),
|
||||
("THRESH-5JQDJ21G1BYKR", "Oavuagh", "Green", "Morningstartail", "Female", 43, "Uaineloidsig"),
|
||||
("THRESH-5KRAAVFB3D1MG", "Geagios", "Blue", "Clubtail", "Male", 99, "Gormfaileas"),
|
||||
("THRESH-B5HSM00J4A2Z4", "Fiedheil", "Black", "Clubtail", "Male", 32, "Moerlaighen"),
|
||||
("THRESH-BMPM2GAPK58JT", "Teirfua", "Green", "Morningstartail", "Female", 68, "Eifyrielle"),
|
||||
("THRESH-MCQ8D6GAGF4K8", "Caollo", "Blue", "Morningstartail", "Female", 93, "Nyghmuiron"),
|
||||
("THRESH-Q0XDBFMNBEB86", "Aenminn", "Red", "Swordtail", "Male", 59, "Berradhyr"),
|
||||
("THRESH-W99DB29MG16QA", "Meinso", "Green", "Clubtail", "Female", 63, "Eifyrielle"),
|
||||
("THRESH-XCJCY7HW7QSSP", "Einmial", "Blue", "Clubtail", "Male", 58, "Gormfaileas"),
|
||||
("THRESH-XR6G68VPGT48M", "Recuagh", "Brown", "Morningstartail", "Female", 55, "Vormalgh"),
|
||||
("THRESH-YQC97TK34214E", "Dannidh", "Orange", "Clubtail", "Male", 61, "Cariaslog"),
|
||||
]
|
||||
|
||||
if __name__ == "__main__":
|
||||
src = sys.argv[1] if len(sys.argv) > 1 else "../../data/raw/register/Unofficial_Dragon_Register_Responses.xlsx"
|
||||
out = sys.argv[2] if len(sys.argv) > 2 else "../../data/public/register.json"
|
||||
raw = load_raw(src)
|
||||
raw = raw[[c for c in raw.columns if "photo" not in str(c).lower() and "cadet name" not in str(c).lower()]]
|
||||
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
|
||||
extra = [{"source": "dragonkind.com result page linked on Reddit", "url": f"https://dragonkind.com/results/{i}",
|
||||
"name": n, "colour": c, "tail": t, "sex": s, "age": a, "lineage": l} for i, n, c, t, s, a, l in REDDIT_RESULTS]
|
||||
json.dump({"source_file": str(src).split("/")[-1], "exported": datetime.now().strftime("%Y-%m-%d"),
|
||||
"headers": [str(c) for c in raw.columns], "rows": rows, "extra": extra},
|
||||
open(out, "w", encoding="utf-8"), ensure_ascii=False, separators=(",", ":"))
|
||||
print(f"{len(rows)} register rows + {len(extra)} Reddit result pages -> {out}")
|
||||
@@ -1,185 +0,0 @@
|
||||
"""update_sources.py: check the community spreadsheets and the route PDF for new information.
|
||||
|
||||
python collection/sources/update_sources.py [--report report.md] [--form FILE] [--register FILE] [--pdf FILE]
|
||||
|
||||
Run from the repo root. Without file arguments it downloads the current versions from the URLs in
|
||||
data/sources.json (the GitHub Action does this every day). With file arguments it uses local copies,
|
||||
which is handy for testing or when a download link changes.
|
||||
|
||||
What it does:
|
||||
* Threshing Routes form: re-exports data/public/form_responses.json and data/derived/runs_clean.csv
|
||||
when the responses changed, and reports how many rows were added or edited.
|
||||
* Dragon Register: re-exports data/public/register.json (rider names and photos dropped) when it changed.
|
||||
The raw register export is never written into the repo.
|
||||
* Route PDF: the PDF is not redistributed. Only a fingerprint is kept (data/derived/pdf_fingerprint.json):
|
||||
the file hash, per-page text hashes and per-line hashes. When it changes, the report lists the pages that
|
||||
changed and the lines that are new, so the coding rules and the scene crosswalk can be updated by hand.
|
||||
|
||||
Exit code is 0 either way; "changed" files are visible to git, and the report says what happened.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
|
||||
sys.path.insert(0, HERE)
|
||||
sys.path.insert(0, os.path.join(ROOT, "core"))
|
||||
from export_raw import cell, load_raw # noqa: E402
|
||||
|
||||
P = lambda *a: os.path.join(ROOT, *a) # noqa: E731
|
||||
|
||||
|
||||
def sha(data, n=None):
|
||||
h = hashlib.sha256(data if isinstance(data, bytes) else data.encode("utf-8")).hexdigest()
|
||||
return h[:n] if n else h
|
||||
|
||||
|
||||
def download(url, dest):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "threshing-atlas-updater"})
|
||||
with urllib.request.urlopen(req, timeout=60) as r, open(dest, "wb") as f:
|
||||
data = r.read()
|
||||
f.write(data)
|
||||
head = data[:15].lower()
|
||||
if head.startswith(b"<!doctype html") or head.startswith(b"<html"):
|
||||
raise RuntimeError(f"{url} returned a web page, not a file (is the link still shared publicly?)")
|
||||
return dest
|
||||
|
||||
|
||||
def sheet_rows(path, drop=()):
|
||||
raw = load_raw(path)
|
||||
raw = raw[[c for c in raw.columns if not any(d in str(c).lower() for d in drop)]]
|
||||
rows = [{str(k): cell(v) for k, v in r.items() if cell(v) is not None} for _, r in raw.iterrows()]
|
||||
return [str(c) for c in raw.columns], rows
|
||||
|
||||
|
||||
def diff_rows(old, new):
|
||||
key = lambda r: json.dumps(r, sort_keys=True, ensure_ascii=False) # noqa: E731
|
||||
o, n = {key(r) for r in old}, {key(r) for r in new}
|
||||
return len(n - o), len(o - n)
|
||||
|
||||
|
||||
def update_form(src, report):
|
||||
out = P("data", "public", "form_responses.json")
|
||||
current = json.load(open(out, encoding="utf-8"))
|
||||
headers, rows = sheet_rows(src)
|
||||
added, removed = diff_rows(current["rows"], rows)
|
||||
if headers == current["headers"] and not added and not removed:
|
||||
report.append(f"- **Threshing Routes form:** no change ({len(rows)} responses).")
|
||||
return False
|
||||
subprocess.run([sys.executable, P("collection", "sources", "export_raw.py"), src, out], check=True, cwd=HERE)
|
||||
from atlas import coding # noqa: E402
|
||||
coding.main([out, P("data", "derived", "runs_clean.csv")])
|
||||
raw_copy = P("data", "raw", "form", "Threshing_Routes_Responses.xlsx")
|
||||
if os.path.abspath(src) != raw_copy: # local copy only; data/raw/form/*.xlsx is git-ignored
|
||||
os.makedirs(os.path.dirname(raw_copy), exist_ok=True)
|
||||
shutil.copyfile(src, raw_copy)
|
||||
new_q = [h for h in headers if h not in current["headers"]]
|
||||
gone_q = [h for h in current["headers"] if h not in headers]
|
||||
report.append(f"- **Threshing Routes form:** {len(current['rows'])} → {len(rows)} responses "
|
||||
f"({added} new or edited, {removed} removed or edited).")
|
||||
if new_q:
|
||||
report.append(f" - New questions (add them to `core/spec/form.yaml`): {', '.join(new_q)}")
|
||||
if gone_q:
|
||||
report.append(f" - Questions no longer in the sheet: {', '.join(gone_q)}")
|
||||
return True
|
||||
|
||||
|
||||
def update_register(src, report):
|
||||
out = P("data", "public", "register.json")
|
||||
current = json.load(open(out, encoding="utf-8"))
|
||||
headers, rows = sheet_rows(src, drop=("photo", "cadet name"))
|
||||
added, removed = diff_rows(current["rows"], rows)
|
||||
if headers == current["headers"] and not added and not removed:
|
||||
report.append(f"- **Dragon Register:** no change ({len(rows)} dragons).")
|
||||
return False
|
||||
subprocess.run([sys.executable, P("collection", "sources", "export_register.py"), src, out], check=True, cwd=HERE)
|
||||
report.append(f"- **Dragon Register:** {len(current['rows'])} → {len(rows)} entries ({added} new or edited, {removed} removed or edited).")
|
||||
return True
|
||||
|
||||
|
||||
def pdf_lines(path):
|
||||
try:
|
||||
from pypdf import PdfReader
|
||||
except ImportError:
|
||||
sys.exit("pypdf is required: pip install -r requirements.txt")
|
||||
pages = []
|
||||
for page in PdfReader(path).pages:
|
||||
text = page.extract_text() or ""
|
||||
pages.append([" ".join(l.split()) for l in text.splitlines() if l.strip()])
|
||||
return pages
|
||||
|
||||
|
||||
def update_pdf(src, report):
|
||||
out = P("data", "derived", "pdf_fingerprint.json")
|
||||
old = json.load(open(out)) if os.path.exists(out) else None
|
||||
data = open(src, "rb").read()
|
||||
pages = pdf_lines(src)
|
||||
fp = {
|
||||
"sha256": sha(data),
|
||||
"bytes": len(data),
|
||||
"pages": len(pages),
|
||||
"page_hashes": [sha("\n".join(p), 16) for p in pages],
|
||||
"line_hashes": sorted({sha(l.lower(), 12) for p in pages for l in p}),
|
||||
"checked": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
|
||||
}
|
||||
if old and old["sha256"] == fp["sha256"]:
|
||||
report.append(f"- **Route PDF:** no change ({fp['pages']} pages).")
|
||||
return False
|
||||
if old and old.get("page_hashes") == fp["page_hashes"]:
|
||||
report.append("- **Route PDF:** the file changed but its text did not (layout or images only). Worth a look.")
|
||||
elif old:
|
||||
changed = [i + 1 for i, h in enumerate(fp["page_hashes"]) if i >= len(old["page_hashes"]) or old["page_hashes"][i] != h]
|
||||
old_lines = set(old["line_hashes"])
|
||||
new_lines = [l for p in pages for l in p if sha(l.lower(), 12) not in old_lines]
|
||||
removed = len(old_lines - set(fp["line_hashes"]))
|
||||
report.append(f"- **Route PDF:** changed. {old['pages']} → {fp['pages']} pages; pages with new text: "
|
||||
f"{', '.join(map(str, changed)) or 'none'}; {len(new_lines)} new lines, {removed} lines gone.")
|
||||
if new_lines:
|
||||
report.append(" - New lines (update the coding rules or the scene crosswalk if they add options):")
|
||||
report.extend(f" - {l}" for l in new_lines[:60])
|
||||
if len(new_lines) > 60:
|
||||
report.append(f" - … and {len(new_lines) - 60} more")
|
||||
else:
|
||||
report.append(f"- **Route PDF:** fingerprint recorded for the first time ({fp['pages']} pages).")
|
||||
json.dump(fp, open(out, "w"), indent=1)
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--report", default=None)
|
||||
ap.add_argument("--form")
|
||||
ap.add_argument("--register")
|
||||
ap.add_argument("--pdf")
|
||||
args = ap.parse_args()
|
||||
sources = json.load(open(P("data", "sources.json")))
|
||||
report, changed, failed = [], False, []
|
||||
tmp = tempfile.mkdtemp(prefix="sources-")
|
||||
steps = [("form", args.form, update_form), ("register", args.register, update_register), ("pdf", args.pdf, update_pdf)]
|
||||
for key, local, fn in steps:
|
||||
try:
|
||||
path = local or download(sources[key]["download"], os.path.join(tmp, sources[key]["filename"]))
|
||||
changed |= fn(os.path.abspath(path), report)
|
||||
except Exception as e: # keep going: one broken link should not block the others
|
||||
failed.append(key)
|
||||
report.append(f"- **{sources[key]['name']}:** could not check ({e}).")
|
||||
head = f"Source check {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}"
|
||||
text = "\n".join([f"## {head}", ""] + report) + "\n"
|
||||
print(text)
|
||||
if args.report:
|
||||
open(args.report, "w", encoding="utf-8").write(text)
|
||||
if os.environ.get("GITHUB_OUTPUT"):
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
|
||||
f.write(f"changed={'true' if changed else 'false'}\nfailed={','.join(failed)}\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,81 @@
|
||||
# The DragonKind bot: how its screens and choices become public data and runs. collection/etl/transform/bot.py.
|
||||
#
|
||||
# The bot's raw data (game prose, account ids) stays on the machine that runs it. What is published:
|
||||
# scene fingerprints, the short labels below (our own paraphrases, not game prose), choice text and outcomes.
|
||||
version: 1
|
||||
|
||||
# Must match normalize() in the bot's src/strategy.js, so fingerprints line up (applied in this order).
|
||||
normalize:
|
||||
- ["[\u2018\u2019\u201b`']", "'"]
|
||||
- ["[\u201c\u201d\"]", "\""]
|
||||
- ["[\u2013\u2014-]", " "]
|
||||
- ["\u2026", " "]
|
||||
- ["[^a-z0-9'\\s]", " "]
|
||||
- ["\\s+", " "]
|
||||
|
||||
# Events in the bot's log that record a choice it acted on
|
||||
choice_events: [DECISION, MANUAL_DECISION, MANUAL_EDGE_PICK]
|
||||
# runs.jsonl and the log are matched when the same attempt starts within this many hours
|
||||
match_hours: 6
|
||||
|
||||
# Short public labels, by scene fingerprint. A scene without one is labelled "<stage> · <tint>" and listed by the
|
||||
# importer so a label can be added here.
|
||||
labels:
|
||||
"db7786392626": "Start: choose company"
|
||||
"a68f681076c6": "Valley edge: choose direction"
|
||||
"bc4ee4754b9c": "First sighting: Red (forest centre)"
|
||||
"abe3011378ac": "First sighting: Brown (forest centre)"
|
||||
"84780dab01e3": "Survived the first sighting"
|
||||
"d25f2759ace0": "Deeper in the forest: a scream"
|
||||
"f72199f24b91": "Clearing: squadmate vs two bullies"
|
||||
"b209bacacec3": "Bully fight: first move"
|
||||
"fbe0f0bea700": "Bully fight: smaller cadet has a dagger"
|
||||
"a2368b0dc212": "Blocked the bigger cadet: bullies’ fate"
|
||||
"f4b4944d3f3b": "Beat the smaller cadet: bullies’ fate"
|
||||
"0cfd06cf8939": "After the fight: choose a direction"
|
||||
"0a561d53ad19": "An hour later: valley or tree"
|
||||
"96b68612eaf4": "Top of the tree (shadow check)"
|
||||
"10bc7353b8eb": "Descended quickly: a Red below"
|
||||
"c62f1f1a2b3a": "Let the Red pass: two Greens in the clearing"
|
||||
"07e24681e3b3": "Descended slowly: two Greens in the clearing"
|
||||
|
||||
# Crosswalk: a bot choice (option text, normalised as in core/graph.js norm()) -> a column of the coded runs.
|
||||
# First match wins; a column keeps its first value. Unmatched choices are listed, never guessed.
|
||||
# Two keys are not columns: scream_investigate (the scream step on its own) and red (the Red below the tree).
|
||||
crosswalk:
|
||||
- ["^enter the valley alone", company, "Alone"]
|
||||
- ["^enter the valley with one of your squadmates", company, "1 Squadmate"]
|
||||
- ["^enter the valley with two of your squadmates", company, "2 Squadmates"]
|
||||
- ["^head east toward the roar", direction, "East"]
|
||||
- ["^head west toward the scent", direction, "West"]
|
||||
- ["^head toward the center of the forest", direction, "Center"]
|
||||
- ["^keep walking the cadets are just as dangerous", scream, "Ignore the scream"]
|
||||
- ["^run and hope", sighting_action, "Retreat / run"]
|
||||
- ["hold still", sighting_action, "Hold still"]
|
||||
- ["lower your eyes in respect|lower your eyes and turn to face it", sighting_action, "Respect / face it"]
|
||||
- ["meeting its gaze", sighting_action, "Meet its gaze"]
|
||||
- ["^hide behind a tree", sighting_action, "Hide"]
|
||||
- ["^investigate the scream", scream_investigate, true]
|
||||
- ["^help your squadmate", scream, "Investigate & help"]
|
||||
- ["^play it safe and leave them", scream, "Investigate & leave"]
|
||||
- ["^pause to assess the situation", combat_first, "Assess the situation"]
|
||||
- ["^charge the other cadet", combat_first, "Even the numbers (smaller cadet)"]
|
||||
- ["^unsheathe your weapon and charge between", combat_first, "Put yourself in the middle (bigger cadet)"]
|
||||
- ["^engage the smaller cadet", combat_second, "Engage the smaller cadet"]
|
||||
- ["^daggers or not", combat_second, "Daggers or not (protect squadmate)"]
|
||||
- ["^end the lives of the bullies", resolution, "Killed the bullies yourself"]
|
||||
- ["^allow your squadmate to exact", resolution, "Squadmate's revenge"]
|
||||
- ["^leave the bullies alive", resolution, "Left bullies to the dragons"]
|
||||
- ["tourniquet", resolution, "Tourniquet"]
|
||||
# the wounded cadet's other two choices (the bot has not logged this screen yet, so these are kept loose)
|
||||
- ["leave (them|him|her) to die", resolution, "Left injured cadet"]
|
||||
- ["take (their|his|her) weapons?", resolution, "Took their weapons"]
|
||||
- ["^walk toward the east", post_path, "East"]
|
||||
- ["^walk toward the west", post_path, "West"]
|
||||
- ["^walk farther into the forest", post_path, "Forest"]
|
||||
- ["^continue toward the center of the valley", hour, "Valley"]
|
||||
- ["^find the tallest tree", hour, "Tree"]
|
||||
- ["^descend quickly", branch, "Descend quickly: Red"]
|
||||
- ["^climb down slowly to pursue the green", branch, "Descend slowly: two Greens"]
|
||||
- ["^reds are too hot tempered", red, "Let it pass (two Greens)"]
|
||||
- ["^jump (now|in a few seconds)", red, "Jump onto the Red"]
|
||||
@@ -0,0 +1,118 @@
|
||||
# "Report a run": the questions of the GitHub issue form, and how each answer becomes a column of runs_clean.csv.
|
||||
#
|
||||
# collection/etl/report_form.py writes .github/ISSUE_TEMPLATE/report-a-run.yml from this file (a test checks it is
|
||||
# up to date), and collection/etl/runs.py codes every report with it. Each option is "label shown: column value";
|
||||
# an option with value null (or no answer) means "not known". Values must be listed in data/dictionary.yaml.
|
||||
version: 1
|
||||
|
||||
name: Report a run
|
||||
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
|
||||
title: "Run report"
|
||||
label: run report
|
||||
intro: |
|
||||
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
|
||||
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
|
||||
Please don't include your DragonKind account name or anything personal.
|
||||
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
|
||||
|
||||
fields:
|
||||
- id: goal
|
||||
label: Which dragon were you trying to bond?
|
||||
column: goal
|
||||
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black, "Any, just for fun": Any (for fun) }
|
||||
- id: attempt
|
||||
label: Which attempt was this for your account? (a number; 1 = your first)
|
||||
column: attempt
|
||||
type: integer
|
||||
- id: company
|
||||
label: Who entered the valley with you?
|
||||
column: company
|
||||
required: true
|
||||
options: { Alone: Alone, One squadmate: 1 Squadmate, Two squadmates: 2 Squadmates }
|
||||
- id: left_squadmates
|
||||
label: If you went with squadmates, did you leave them?
|
||||
column: left_squadmates
|
||||
options: { "Yes, I left them": true, "No, we stayed together": false, "I went alone": null }
|
||||
- id: direction
|
||||
label: Which way did you go first?
|
||||
column: direction
|
||||
required: true
|
||||
options: { East (toward the roar): East, West (toward the scent): West, Toward the forest centre: Center }
|
||||
- id: first_sighting
|
||||
label: Which dragon found you first?
|
||||
column: first_sighting
|
||||
options: { Red: Red, Brown: Brown, Green: Green, Orange: Orange }
|
||||
- id: sighting_action
|
||||
label: What did you do when it found you?
|
||||
column: sighting_action
|
||||
options:
|
||||
Hold still: Hold still
|
||||
Lower your eyes in respect / turn to face it: Respect / face it
|
||||
Run: Retreat / run
|
||||
Hide: Hide
|
||||
Draw a weapon: Weapon
|
||||
Meet its gaze: Meet its gaze
|
||||
Protect your squadmate: Protect squadmate
|
||||
- id: scream
|
||||
label: The scream
|
||||
column: scream
|
||||
required: true
|
||||
options:
|
||||
I died at the first dragon: DIED at first sighting
|
||||
Investigated, and helped my squadmate: Investigate & help
|
||||
Investigated, and walked away: Investigate & leave
|
||||
Kept walking (ignored it): Ignore the scream
|
||||
- id: combat_first
|
||||
label: "If you helped: your first move in the fight"
|
||||
column: combat_first
|
||||
options:
|
||||
Pause to assess: Assess the situation
|
||||
Charge the smaller cadet: Even the numbers (smaller cadet)
|
||||
Charge between the bigger cadet and your squadmate: Put yourself in the middle (bigger cadet)
|
||||
- id: combat_second
|
||||
label: "If you paused to assess: the smaller cadet's dagger"
|
||||
column: combat_second
|
||||
options: { Engage the smaller cadet: Engage the smaller cadet, Charge the bigger cadet anyway: Daggers or not (protect squadmate) }
|
||||
- id: resolution
|
||||
label: How did the cadet encounter end?
|
||||
column: resolution
|
||||
options:
|
||||
I died in the fight: DIED in combat
|
||||
"Bullies: killed them myself": Killed the bullies yourself
|
||||
"Bullies: my squadmate's revenge": Squadmate's revenge
|
||||
"Bullies: left them to the dragons": Left bullies to the dragons
|
||||
"Wounded cadet: tourniquet": Tourniquet
|
||||
"Wounded cadet: left them": Left injured cadet
|
||||
"Wounded cadet: took their weapons": Took their weapons
|
||||
"I walked away from the clearing": Left them to it
|
||||
- id: post_path
|
||||
label: "After the fight: which way did you walk?"
|
||||
column: post_path
|
||||
options: { Deeper into the forest: Forest, West: West, East: East, There was no fork: (no fork) }
|
||||
- id: hour
|
||||
label: After an hour's walk
|
||||
column: hour
|
||||
options: { Climbed the tallest tree: Tree, Went to the valley centre: Valley, I died before the hour: DIED before the hour }
|
||||
- id: branch
|
||||
label: Which bonding encounter did you reach?
|
||||
column: branch
|
||||
options:
|
||||
"Tree: the shadow (Blue or Black)": Shadow (rare)
|
||||
"Tree: climbed down quickly, jumped on the Red": "Descend quickly: Red"
|
||||
"Tree: climbed down quickly, let the Red pass": "Descend quickly: let the Red pass"
|
||||
"Tree: climbed down slowly to the two Greens": "Descend slowly: two Greens"
|
||||
"Valley: faced the Red": Valley Red
|
||||
"Valley: passed the Red, the meadow": "Passed the Red: meadow"
|
||||
- id: outcome
|
||||
label: How did the run end?
|
||||
column: outcome
|
||||
required: true
|
||||
options: { Bonded: Bonded, Incinerated at a bonding encounter: Incinerated, Died before the hour: Died before the hour }
|
||||
- id: outcome_color
|
||||
label: Which dragon bonded you (or incinerated you)?
|
||||
column: outcome_color
|
||||
options: { Red: Red, Green: Green, Orange: Orange, Brown: Brown, Blue: Blue, Black: Black }
|
||||
- id: tail
|
||||
label: "If you bonded: the dragon's tail"
|
||||
column: tail
|
||||
options: { Clubtail: Clubtail, Daggertail: Daggertail, Morningstartail: Morningstartail, Scorpiontail: Scorpiontail, Swordtail: Swordtail }
|
||||
Reference in New Issue
Block a user