Core: coding rules as declarative form.yaml + Python package (atlas.coding) (#9)

Step 1 of moving the repo to Python with declarative rules.

- core/spec/form.yaml: every question (how it is found in an export), every answer and what it
  is coded as, missing answers, email redaction, burn order, first-dragon pools.
- core/atlas/coding.py reads it and replaces core/python/clean.py (python -m atlas.coding).
- tests/core/test_coding.py: 16 rule tests, one hand-written row per rule, checking the rules
  are right rather than only that JS and Python agree.
- analysis/stats/analyze.py takes the colours and missing answers from the spec instead of its
  own copy; the daily update job codes rows through the package.
- pyproject.toml for the package; pyyaml and pytest added to requirements.

The coded runs are byte-identical to before (5,988 runs); all parity checks still match.


Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ekcSchiano
2026-11-03 15:27:13 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 432e04c2c7
commit 3536007ade
19 changed files with 680 additions and 427 deletions
+10 -7
View File
@@ -10,9 +10,10 @@ Unofficial fan project. Not affiliated with Dragonkind or the Fourth Wing publis
The project has four parts, built on one shared core. Each folder has its own README.
```
core/ the shared model every part uses: the run format and coding rules (engine.js, with the
Python mirror in core/python/clean.py), the route graph and bot crosswalk (graph.js), and the
decision model and bot advisor (solver.js). No dependencies; runs in Node and the browser.
core/ the shared model every part uses. spec/ holds the declarative rules (form.yaml: every question
and answer of the community form); atlas/ is the Python package that reads them (coding.py).
engine.js, graph.js and solver.js are the JavaScript the site and Ken's bot still use; they move
to Python + spec files step by step (docs/ROADMAP.md).
collection/ getting data in: sources/ (the form sheet, the Dragon Register, the route PDF check, the daily
update job) and bot/ (the DragonKind bot importer). Reddit is not collected yet.
analysis/ making sense of it: stats/ (Python statistics, mirrored in core/engine.js), network/ (builds
@@ -72,20 +73,22 @@ looks like prose. See `docs/BOT_DATA_PROTOCOL.md` for what the bot should log.
In the browser: open the page and use **Load a newer export** with the responses spreadsheet
(Google Sheets → File → Download → Microsoft Excel). Everything recomputes. Questions are matched by
wording; a renamed question is listed as "not found", and then `COLUMN_KEYS` in
`core/engine.js` and `core/python/clean.py` need updating.
`core/spec/form.yaml` (and, until the site stops coding rows itself, `core/engine.js`) need updating.
To update the bundled snapshot, put the new export in `data/raw/form/`, then:
```
python collection/sources/export_raw.py data/raw/form/NEW_EXPORT.xlsx data/public/form_responses.json
python core/python/clean.py data/public/form_responses.json data/derived/runs_clean.csv
PYTHONPATH=core python -m atlas.coding data/public/form_responses.json data/derived/runs_clean.csv
npm test
```
## Changing a coding rule
Every rule lives in `codeRow()` in `core/engine.js` and `code_row()` in
`core/python/clean.py`. Change both and run `npm test`.
The tables (how each question is found, what each answer means, which answers count as missing) are in
`core/spec/form.yaml`; the logic that combines them is `code_row()` in `core/atlas/coding.py`. Add a case to
`tests/core/test_coding.py` for every rule you change, then run `npm test`. Until the site switches to
precomputed statistics, `codeRow()` in `core/engine.js` must be changed the same way (the parity tests check it).
## Statistics in one paragraph