Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
@@ -0,0 +1,216 @@
|
||||
# Written by collection/etl/report_form.py from collection/specs/report_form.yaml: edit that file, then run `python -m collection build`.
|
||||
name: Report a run
|
||||
description: Log one threshing run for the Threshing Atlas. Reports are public and become rows of runs_clean.csv.
|
||||
title: Run report
|
||||
labels:
|
||||
- run report
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for reporting a run! Answer what you remember; leave anything you're unsure of on "Not sure".
|
||||
Every answer here is **public** (this is a GitHub issue) and goes into the Atlas's open data as source "reported".
|
||||
Please don't include your DragonKind account name or anything personal.
|
||||
You can also use the community Google Form linked from the r/fourthwing megathread; please report each run only once.
|
||||
- type: dropdown
|
||||
id: goal
|
||||
attributes:
|
||||
label: Which dragon were you trying to bond?
|
||||
options:
|
||||
- Red
|
||||
- Green
|
||||
- Orange
|
||||
- Brown
|
||||
- Blue
|
||||
- Black
|
||||
- Any, just for fun
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: attempt
|
||||
attributes:
|
||||
label: Which attempt was this for your account? (a number; 1 = your first)
|
||||
placeholder: e.g. 3
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: company
|
||||
attributes:
|
||||
label: Who entered the valley with you?
|
||||
options:
|
||||
- Alone
|
||||
- One squadmate
|
||||
- Two squadmates
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: left_squadmates
|
||||
attributes:
|
||||
label: If you went with squadmates, did you leave them?
|
||||
options:
|
||||
- Yes, I left them
|
||||
- No, we stayed together
|
||||
- I went alone
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: direction
|
||||
attributes:
|
||||
label: Which way did you go first?
|
||||
options:
|
||||
- East (toward the roar)
|
||||
- West (toward the scent)
|
||||
- Toward the forest centre
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: first_sighting
|
||||
attributes:
|
||||
label: Which dragon found you first?
|
||||
options:
|
||||
- Red
|
||||
- Brown
|
||||
- Green
|
||||
- Orange
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: sighting_action
|
||||
attributes:
|
||||
label: What did you do when it found you?
|
||||
options:
|
||||
- Hold still
|
||||
- Lower your eyes in respect / turn to face it
|
||||
- Run
|
||||
- Hide
|
||||
- Draw a weapon
|
||||
- Meet its gaze
|
||||
- Protect your squadmate
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: scream
|
||||
attributes:
|
||||
label: The scream
|
||||
options:
|
||||
- I died at the first dragon
|
||||
- Investigated, and helped my squadmate
|
||||
- Investigated, and walked away
|
||||
- Kept walking (ignored it)
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: combat_first
|
||||
attributes:
|
||||
label: 'If you helped: your first move in the fight'
|
||||
options:
|
||||
- Pause to assess
|
||||
- Charge the smaller cadet
|
||||
- Charge between the bigger cadet and your squadmate
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: combat_second
|
||||
attributes:
|
||||
label: 'If you paused to assess: the smaller cadet''s dagger'
|
||||
options:
|
||||
- Engage the smaller cadet
|
||||
- Charge the bigger cadet anyway
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: resolution
|
||||
attributes:
|
||||
label: How did the cadet encounter end?
|
||||
options:
|
||||
- I died in the fight
|
||||
- 'Bullies: killed them myself'
|
||||
- 'Bullies: my squadmate''s revenge'
|
||||
- 'Bullies: left them to the dragons'
|
||||
- 'Wounded cadet: tourniquet'
|
||||
- 'Wounded cadet: left them'
|
||||
- 'Wounded cadet: took their weapons'
|
||||
- I walked away from the clearing
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: post_path
|
||||
attributes:
|
||||
label: 'After the fight: which way did you walk?'
|
||||
options:
|
||||
- Deeper into the forest
|
||||
- West
|
||||
- East
|
||||
- There was no fork
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: hour
|
||||
attributes:
|
||||
label: After an hour's walk
|
||||
options:
|
||||
- Climbed the tallest tree
|
||||
- Went to the valley centre
|
||||
- I died before the hour
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: branch
|
||||
attributes:
|
||||
label: Which bonding encounter did you reach?
|
||||
options:
|
||||
- 'Tree: the shadow (Blue or Black)'
|
||||
- 'Tree: climbed down quickly, jumped on the Red'
|
||||
- 'Tree: climbed down quickly, let the Red pass'
|
||||
- 'Tree: climbed down slowly to the two Greens'
|
||||
- 'Valley: faced the Red'
|
||||
- 'Valley: passed the Red, the meadow'
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: outcome
|
||||
attributes:
|
||||
label: How did the run end?
|
||||
options:
|
||||
- Bonded
|
||||
- Incinerated at a bonding encounter
|
||||
- Died before the hour
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: outcome_color
|
||||
attributes:
|
||||
label: Which dragon bonded you (or incinerated you)?
|
||||
options:
|
||||
- Red
|
||||
- Green
|
||||
- Orange
|
||||
- Brown
|
||||
- Blue
|
||||
- Black
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: tail
|
||||
attributes:
|
||||
label: 'If you bonded: the dragon''s tail'
|
||||
options:
|
||||
- Clubtail
|
||||
- Daggertail
|
||||
- Morningstartail
|
||||
- Scorpiontail
|
||||
- Swordtail
|
||||
- Not sure
|
||||
validations:
|
||||
required: false
|
||||
@@ -22,16 +22,55 @@ jobs:
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- run: pip install -r requirements.txt
|
||||
# The raw store (data/raw: the current and previous raw file of each source) is never committed. Between runs
|
||||
# it lives in the Actions cache, encrypted with the RAW_ARCHIVE_KEY secret, so only the previous version is
|
||||
# kept and nobody without the key can read it (the register has rider names). Without the secret there is no
|
||||
# archive: each run starts from the manifest alone.
|
||||
- name: Restore the private raw archive
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ runner.temp }}/raw-archive
|
||||
key: raw-archive-none
|
||||
restore-keys: raw-archive-
|
||||
- name: Decrypt it
|
||||
env:
|
||||
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
|
||||
run: |
|
||||
f="$RUNNER_TEMP/raw-archive/raw.tgz.enc"
|
||||
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then
|
||||
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data
|
||||
fi
|
||||
# extract -> transform -> load for every source, then the combined runs (collection/pipeline.yaml)
|
||||
- id: check
|
||||
run: python collection/sources/update_sources.py --report "$RUNNER_TEMP/report.md"
|
||||
- name: Rebuild graph and run tests
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
|
||||
REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
|
||||
run: python -m collection update --report "$RUNNER_TEMP/report.md"
|
||||
- name: Rebuild the analysis and run tests
|
||||
if: steps.check.outputs.changed == 'true'
|
||||
run: |
|
||||
node analysis/network/build-graph.js
|
||||
npm run policy
|
||||
npm run stats
|
||||
npm run metadata
|
||||
python -m collection build
|
||||
npm test
|
||||
- name: Encrypt the raw store
|
||||
id: archive
|
||||
env:
|
||||
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
|
||||
run: |
|
||||
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -d data/raw ]; then
|
||||
mkdir -p "$RUNNER_TEMP/raw-archive"
|
||||
tar -czf - -C data raw | openssl enc -aes-256-cbc -pbkdf2 -salt -pass env:RAW_ARCHIVE_KEY -out "$RUNNER_TEMP/raw-archive/raw.tgz.enc"
|
||||
echo "save=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- name: Save it (a new copy only when a raw file changed)
|
||||
if: steps.archive.outputs.save == 'true'
|
||||
uses: actions/cache/save@v4
|
||||
with:
|
||||
path: ${{ runner.temp }}/raw-archive
|
||||
key: raw-archive-${{ hashFiles('data/manifest.json') }}
|
||||
# The PR is opened with ATLAS_BOT_TOKEN (a fine-grained token, see docs/REPO_SETTINGS.md) when it is set:
|
||||
# changes made with the built-in GITHUB_TOKEN do not start other workflows, so its merge would not
|
||||
# redeploy the site. Without the secret it falls back to GITHUB_TOKEN and the PR waits for a manual merge.
|
||||
|
||||
Reference in New Issue
Block a user