Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+54
View File
@@ -0,0 +1,54 @@
"""Load: the only stage that writes public files. Writes are byte-stable (the same table gives the same bytes)
and only happen when the bytes change, so git shows exactly what changed."""
import csv
import io
import json
from .common import json_bytes, jsonl_bytes, path
def public(obj):
"""Drop transient keys (those starting with "_"), which are for the report only."""
if isinstance(obj, dict):
return {k: public(v) for k, v in obj.items() if not str(k).startswith("_")}
if isinstance(obj, list):
return [public(v) for v in obj]
return obj
def read_json(rel):
p = path(rel)
return json.loads(p.read_text(encoding="utf-8")) if p.exists() else None
def write_bytes(rel, data, check=False):
"""Write when different. Returns True when the file changed (or, with check, would change)."""
p = path(rel)
if p.exists() and p.read_bytes() == data:
return False
if not check:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(data)
return True
def json_file(rel, obj, check=False, compact=True):
return write_bytes(rel, json_bytes(public(obj), compact), check)
def jsonl_file(rel, rows, check=False):
return write_bytes(rel, jsonl_bytes(rows), check)
def csv_bytes(rows, columns):
"""True/False for booleans, empty for missing, everything else as text."""
buf = io.StringIO()
w = csv.writer(buf, lineterminator="\n")
w.writerow(columns)
for r in rows:
w.writerow(["" if r.get(c) is None else r[c] for c in columns])
return buf.getvalue().encode("utf-8")
def csv_file(rel, rows, columns, check=False):
return write_bytes(rel, csv_bytes(rows, columns), check)