Step 1 of moving the repo to Python with declarative rules. - core/spec/form.yaml: every question (how it is found in an export), every answer and what it is coded as, missing answers, email redaction, burn order, first-dragon pools. - core/atlas/coding.py reads it and replaces core/python/clean.py (python -m atlas.coding). - tests/core/test_coding.py: 16 rule tests, one hand-written row per rule, checking the rules are right rather than only that JS and Python agree. - analysis/stats/analyze.py takes the colours and missing answers from the spec instead of its own copy; the daily update job codes rows through the package. - pyproject.toml for the package; pyyaml and pytest added to requirements. The coded runs are byte-identical to before (5,988 runs); all parity checks still match. Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
20 lines
1.3 KiB
Markdown
20 lines
1.3 KiB
Markdown
# collection
|
|
|
|
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
|
|
under `data/`.
|
|
|
|
| Source | Code | Output |
|
|
|---|---|---|
|
|
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/atlas/coding.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
|
|
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
|
|
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
|
|
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
|
|
| Reddit megathreads | not collected yet | — |
|
|
|
|
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
|
|
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
|
|
merges itself when CI passes. A broken link opens an issue.
|
|
|
|
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
|
|
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.
|