Collection as an ETL pipeline; runs_clean.csv from every source; Report a run

- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
ekcSchiano
2026-11-04 02:46:48 +00:00
co-authored by Claude Opus 5.5
parent 61bc50583c
commit 3b24a11f5a
64 changed files with 8626 additions and 6909 deletions
+9 -6
View File
@@ -4,9 +4,10 @@ The tables (how each question is found, what every answer means, which answers c
core/spec/form.yaml. This module holds the logic that combines answers into a run: which way the rider went,
which branch they took after the hour, how the run ended, and which rows look inconsistent (data_flags).
python -m atlas.coding [form_responses.json|.xlsx|.csv] [runs_clean.csv]
python -m atlas.coding [form_responses.json|.xlsx|.csv] [out.csv]
Defaults: data/public/form_responses.json -> data/derived/runs_clean.csv
Prints a summary of the coded export (default: data/public/form_responses.json), and writes the coded runs to
out.csv when given. The published runs (every source) are written by `python -m collection build`.
"""
import json
import re
@@ -276,7 +277,8 @@ def read_export(path):
"""An export as a DataFrame: the bundled JSON (data/public/form_responses.json), a CSV or the .xlsx."""
p = str(path).lower()
if p.endswith(".json"):
d = json.load(open(path, encoding="utf-8"))
with open(path, encoding="utf-8") as f:
d = json.load(f)
return pd.DataFrame(d["rows"], columns=d["headers"])
if p.endswith(".csv"):
return pd.read_csv(path)
@@ -300,14 +302,15 @@ def load(path):
def main(argv=None):
argv = sys.argv[1:] if argv is None else argv
src = argv[0] if argv else ROOT / "data/public/form_responses.json"
out = argv[1] if len(argv) > 1 else ROOT / "data/derived/runs_clean.csv"
out = argv[1] if len(argv) > 1 else None
df, missing = load(src)
if missing:
print("WARNING: columns not found:", missing)
# a whole-number column with gaps would otherwise be written as 1.0, 2.0, ... (data/dictionary.yaml: integer)
df["attempt"] = df["attempt"].astype("Int64")
df.to_csv(out, index=False)
print(f"{len(df)} runs -> {out}")
if out:
df.to_csv(out, index=False)
print(f"{len(df)} runs" + (f" -> {out}" if out else ""))
print(df.outcome.value_counts().to_string())
print("rows with flags:", (df["data_flags"] != "").sum())