Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
@@ -0,0 +1,54 @@
|
||||
"""Load: the only stage that writes public files. Writes are byte-stable (the same table gives the same bytes)
|
||||
and only happen when the bytes change, so git shows exactly what changed."""
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
|
||||
from .common import json_bytes, jsonl_bytes, path
|
||||
|
||||
|
||||
def public(obj):
|
||||
"""Drop transient keys (those starting with "_"), which are for the report only."""
|
||||
if isinstance(obj, dict):
|
||||
return {k: public(v) for k, v in obj.items() if not str(k).startswith("_")}
|
||||
if isinstance(obj, list):
|
||||
return [public(v) for v in obj]
|
||||
return obj
|
||||
|
||||
|
||||
def read_json(rel):
|
||||
p = path(rel)
|
||||
return json.loads(p.read_text(encoding="utf-8")) if p.exists() else None
|
||||
|
||||
|
||||
def write_bytes(rel, data, check=False):
|
||||
"""Write when different. Returns True when the file changed (or, with check, would change)."""
|
||||
p = path(rel)
|
||||
if p.exists() and p.read_bytes() == data:
|
||||
return False
|
||||
if not check:
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_bytes(data)
|
||||
return True
|
||||
|
||||
|
||||
def json_file(rel, obj, check=False, compact=True):
|
||||
return write_bytes(rel, json_bytes(public(obj), compact), check)
|
||||
|
||||
|
||||
def jsonl_file(rel, rows, check=False):
|
||||
return write_bytes(rel, jsonl_bytes(rows), check)
|
||||
|
||||
|
||||
def csv_bytes(rows, columns):
|
||||
"""True/False for booleans, empty for missing, everything else as text."""
|
||||
buf = io.StringIO()
|
||||
w = csv.writer(buf, lineterminator="\n")
|
||||
w.writerow(columns)
|
||||
for r in rows:
|
||||
w.writerow(["" if r.get(c) is None else r[c] for c in columns])
|
||||
return buf.getvalue().encode("utf-8")
|
||||
|
||||
|
||||
def csv_file(rel, rows, columns, check=False):
|
||||
return write_bytes(rel, csv_bytes(rows, columns), check)
|
||||
Reference in New Issue
Block a user