Core: coding rules as declarative form.yaml + Python package (atlas.coding) (#9)

Step 1 of moving the repo to Python with declarative rules.

- core/spec/form.yaml: every question (how it is found in an export), every answer and what it
  is coded as, missing answers, email redaction, burn order, first-dragon pools.
- core/atlas/coding.py reads it and replaces core/python/clean.py (python -m atlas.coding).
- tests/core/test_coding.py: 16 rule tests, one hand-written row per rule, checking the rules
  are right rather than only that JS and Python agree.
- analysis/stats/analyze.py takes the colours and missing answers from the spec instead of its
  own copy; the daily update job codes rows through the package.
- pyproject.toml for the package; pyyaml and pytest added to requirements.

The coded runs are byte-identical to before (5,988 runs); all parity checks still match.


Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ekcSchiano
2026-11-03 15:27:13 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 432e04c2c7
commit 3536007ade
19 changed files with 680 additions and 427 deletions
+4 -2
View File
@@ -31,6 +31,7 @@ from datetime import datetime, timezone
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
sys.path.insert(0, HERE)
sys.path.insert(0, os.path.join(ROOT, "core"))
from export_raw import cell, load_raw # noqa: E402
P = lambda *a: os.path.join(ROOT, *a) # noqa: E731
@@ -74,7 +75,8 @@ def update_form(src, report):
report.append(f"- **Threshing Routes form:** no change ({len(rows)} responses).")
return False
subprocess.run([sys.executable, P("collection", "sources", "export_raw.py"), src, out], check=True, cwd=HERE)
subprocess.run([sys.executable, P("core", "python", "clean.py"), out, P("data", "derived", "runs_clean.csv")], check=True, cwd=HERE)
from atlas import coding # noqa: E402
coding.main([out, P("data", "derived", "runs_clean.csv")])
raw_copy = P("data", "raw", "form", "Threshing_Routes_Responses.xlsx")
if os.path.abspath(src) != raw_copy: # local copy only; data/raw/form/*.xlsx is git-ignored
os.makedirs(os.path.dirname(raw_copy), exist_ok=True)
@@ -84,7 +86,7 @@ def update_form(src, report):
report.append(f"- **Threshing Routes form:** {len(current['rows'])} → {len(rows)} responses "
f"({added} new or edited, {removed} removed or edited).")
if new_q:
report.append(f" - New questions (check `COLUMN_KEYS` in engine.js and clean.py): {', '.join(new_q)}")
report.append(f" - New questions (add them to `core/spec/form.yaml`): {', '.join(new_q)}")
if gone_q:
report.append(f" - Questions no longer in the sheet: {', '.join(gone_q)}")
return True