Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+42 -3
View File
@@ -22,16 +22,55 @@ jobs:
with:
python-version: "3.12"
- run: pip install -r requirements.txt
# The raw store (data/raw: the current and previous raw file of each source) is never committed. Between runs
# it lives in the Actions cache, encrypted with the RAW_ARCHIVE_KEY secret, so only the previous version is
# kept and nobody without the key can read it (the register has rider names). Without the secret there is no
# archive: each run starts from the manifest alone.
- name: Restore the private raw archive
uses: actions/cache/restore@v4
with:
path: ${{ runner.temp }}/raw-archive
key: raw-archive-none
restore-keys: raw-archive-
- name: Decrypt it
env:
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
run: |
f="$RUNNER_TEMP/raw-archive/raw.tgz.enc"
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data
fi
# extract -> transform -> load for every source, then the combined runs (collection/pipeline.yaml)
- id: check
run: python collection/sources/update_sources.py --report "$RUNNER_TEMP/report.md"
- name: Rebuild graph and run tests
env:
GITHUB_TOKEN: ${{ github.token }}
REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
run: python -m collection update --report "$RUNNER_TEMP/report.md"
- name: Rebuild the analysis and run tests
if: steps.check.outputs.changed == 'true'
run: |
node analysis/network/build-graph.js
npm run policy
npm run stats
npm run metadata
python -m collection build
npm test
- name: Encrypt the raw store
id: archive
env:
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
run: |
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -d data/raw ]; then
mkdir -p "$RUNNER_TEMP/raw-archive"
tar -czf - -C data raw | openssl enc -aes-256-cbc -pbkdf2 -salt -pass env:RAW_ARCHIVE_KEY -out "$RUNNER_TEMP/raw-archive/raw.tgz.enc"
echo "save=true" >> "$GITHUB_OUTPUT"
fi
- name: Save it (a new copy only when a raw file changed)
if: steps.archive.outputs.save == 'true'
uses: actions/cache/save@v4
with:
path: ${{ runner.temp }}/raw-archive
key: raw-archive-${{ hashFiles('data/manifest.json') }}
# The PR is opened with ATLAS_BOT_TOKEN (a fine-grained token, see docs/REPO_SETTINGS.md) when it is set:
# changes made with the built-in GITHUB_TOKEN do not start other workflows, so its merge would not
# redeploy the site. Without the secret it falls back to GITHUB_TOKEN and the PR waits for a manual merge.