Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+216
View File
@@ -0,0 +1,216 @@
# Written by collection/etl/report_form.py from collection/specs/report_form.yaml: edit that file, then run `python -m collection build`.
name: Report a run
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
title: Run report
labels:
- run report
body:
- type: markdown
attributes:
value: |
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
Please don't include your DragonKind account name or anything personal.
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
- type: dropdown
id: goal
attributes:
label: Which dragon were you trying to bond?
options:
- Red
- Green
- Orange
- Brown
- Blue
- Black
- Any, just for fun
- Not sure
validations:
required: false
- type: input
id: attempt
attributes:
label: Which attempt was this for your account? (a number; 1 = your first)
placeholder: e.g. 3
validations:
required: false
- type: dropdown
id: company
attributes:
label: Who entered the valley with you?
options:
- Alone
- One squadmate
- Two squadmates
validations:
required: true
- type: dropdown
id: left_squadmates
attributes:
label: If you went with squadmates, did you leave them?
options:
- Yes, I left them
- No, we stayed together
- I went alone
- Not sure
validations:
required: false
- type: dropdown
id: direction
attributes:
label: Which way did you go first?
options:
- East (toward the roar)
- West (toward the scent)
- Toward the forest centre
validations:
required: true
- type: dropdown
id: first_sighting
attributes:
label: Which dragon found you first?
options:
- Red
- Brown
- Green
- Orange
- Not sure
validations:
required: false
- type: dropdown
id: sighting_action
attributes:
label: What did you do when it found you?
options:
- Hold still
- Lower your eyes in respect / turn to face it
- Run
- Hide
- Draw a weapon
- Meet its gaze
- Protect your squadmate
- Not sure
validations:
required: false
- type: dropdown
id: scream
attributes:
label: The scream
options:
- I died at the first dragon
- Investigated, and helped my squadmate
- Investigated, and walked away
- Kept walking (ignored it)
validations:
required: true
- type: dropdown
id: combat_first
attributes:
label: 'If you helped: your first move in the fight'
options:
- Pause to assess
- Charge the smaller cadet
- Charge between the bigger cadet and your squadmate
- Not sure
validations:
required: false
- type: dropdown
id: combat_second
attributes:
label: 'If you paused to assess: the smaller cadet''s dagger'
options:
- Engage the smaller cadet
- Charge the bigger cadet anyway
- Not sure
validations:
required: false
- type: dropdown
id: resolution
attributes:
label: How did the cadet encounter end?
options:
- I died in the fight
- 'Bullies: killed them myself'
- 'Bullies: my squadmate''s revenge'
- 'Bullies: left them to the dragons'
- 'Wounded cadet: tourniquet'
- 'Wounded cadet: left them'
- 'Wounded cadet: took their weapons'
- I walked away from the clearing
- Not sure
validations:
required: false
- type: dropdown
id: post_path
attributes:
label: 'After the fight: which way did you walk?'
options:
- Deeper into the forest
- West
- East
- There was no fork
- Not sure
validations:
required: false
- type: dropdown
id: hour
attributes:
label: After an hour's walk
options:
- Climbed the tallest tree
- Went to the valley centre
- I died before the hour
- Not sure
validations:
required: false
- type: dropdown
id: branch
attributes:
label: Which bonding encounter did you reach?
options:
- 'Tree: the shadow (Blue or Black)'
- 'Tree: climbed down quickly, jumped on the Red'
- 'Tree: climbed down quickly, let the Red pass'
- 'Tree: climbed down slowly to the two Greens'
- 'Valley: faced the Red'
- 'Valley: passed the Red, the meadow'
- Not sure
validations:
required: false
- type: dropdown
id: outcome
attributes:
label: How did the run end?
options:
- Bonded
- Incinerated at a bonding encounter
- Died before the hour
validations:
required: true
- type: dropdown
id: outcome_color
attributes:
label: Which dragon bonded you (or incinerated you)?
options:
- Red
- Green
- Orange
- Brown
- Blue
- Black
- Not sure
validations:
required: false
- type: dropdown
id: tail
attributes:
label: 'If you bonded: the dragon''s tail'
options:
- Clubtail
- Daggertail
- Morningstartail
- Scorpiontail
- Swordtail
- Not sure
validations:
required: false
+42 -3
View File
@@ -22,16 +22,55 @@ jobs:
with:
python-version: "3.12"
- run: pip install -r requirements.txt
# The raw store (data/raw: the current and previous raw file of each source) is never committed. Between runs
# it lives in the Actions cache, encrypted with the RAW_ARCHIVE_KEY secret, so only the previous version is
# kept and nobody without the key can read it (the register has rider names). Without the secret there is no
# archive: each run starts from the manifest alone.
- name: Restore the private raw archive
uses: actions/cache/restore@v4
with:
path: ${{ runner.temp }}/raw-archive
key: raw-archive-none
restore-keys: raw-archive-
- name: Decrypt it
env:
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
run: |
f="$RUNNER_TEMP/raw-archive/raw.tgz.enc"
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data
fi
# extract -> transform -> load for every source, then the combined runs (collection/pipeline.yaml)
- id: check
run: python collection/sources/update_sources.py --report "$RUNNER_TEMP/report.md"
- name: Rebuild graph and run tests
env:
GITHUB_TOKEN: ${{ github.token }}
REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
run: python -m collection update --report "$RUNNER_TEMP/report.md"
- name: Rebuild the analysis and run tests
if: steps.check.outputs.changed == 'true'
run: |
node analysis/network/build-graph.js
npm run policy
npm run stats
npm run metadata
python -m collection build
npm test
- name: Encrypt the raw store
id: archive
env:
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
run: |
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -d data/raw ]; then
mkdir -p "$RUNNER_TEMP/raw-archive"
tar -czf - -C data raw | openssl enc -aes-256-cbc -pbkdf2 -salt -pass env:RAW_ARCHIVE_KEY -out "$RUNNER_TEMP/raw-archive/raw.tgz.enc"
echo "save=true" >> "$GITHUB_OUTPUT"
fi
- name: Save it (a new copy only when a raw file changed)
if: steps.archive.outputs.save == 'true'
uses: actions/cache/save@v4
with:
path: ${{ runner.temp }}/raw-archive
key: raw-archive-${{ hashFiles('data/manifest.json') }}
# The PR is opened with ATLAS_BOT_TOKEN (a fine-grained token, see docs/REPO_SETTINGS.md) when it is set:
# changes made with the built-in GITHUB_TOKEN do not start other workflows, so its merge would not
# redeploy the site. Without the secret it falls back to GITHUB_TOKEN and the PR waits for a manual merge.