Files
threshing-atlas/collection/README.md
T
432e04c2c7 Refactor: lay the repo out by section (core, collection, analysis, planner, site) (#8)
Moves only; no behaviour changes. Every file now lives under the part of the project it
belongs to, each section has a README, and docs/PRIVACY.md collects the rules all parts follow.

  core/        engine.js, graph.js, solver.js, python/clean.py
  collection/  sources/ (form, register, PDF check, daily update), bot/ (importer)
  analysis/    stats/ (Python statistics), network/ (build-graph.js), reports/ (POLICY.md)
  planner/     build-policy.js, policy-report.js
  site/        atlas/ (the page), build.js, qa/
  tests/       core/ (unit), parity/ (JS vs Python), fixtures/

Checked against the previous layout: tests pass; coded runs, test outputs, graph.json and
policy.json are byte-identical; the built site differs only in comments that name file paths.
The published file names Ken's bot downloads (solver.js, graph.js, data/policy.json) are unchanged.

Also removes the undefined/ folder of QA screenshots committed by mistake (the QA scripts now
default to a temp folder) and test.txt, and drops the unused npm workspaces.


Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 19:46:33 +00:00

20 lines
1.3 KiB
Markdown

# collection
Getting data in. Everything here turns a source into the shared run format (see `core/`) or into public files
under `data/`.
| Source | Code | Output |
|---|---|---|
| Community "Threshing Routes" form (Google Sheet) | `sources/export_raw.py`, then `core/python/clean.py` | `data/public/form_responses.json`, `data/derived/runs_clean.csv` |
| Unofficial Dragon Register (Google Sheet) | `sources/export_register.py` | `data/public/register.json` (rider names stay local) |
| Community route PDF | `sources/update_sources.py` | `data/derived/pdf_fingerprint.json` only; the PDF is never committed |
| DragonKind bot (Ken) | `bot/import-bot.js`, `bot/bot.js` | `data/bot/scenes.json`, `data/bot/trajectories-<tag>.jsonl` |
| Reddit megathreads | not collected yet | — |
`sources/update_sources.py` runs every day from `.github/workflows/update-sources.yml`. It downloads the sheets
and the PDF (links in `data/sources.json`), rebuilds the data if anything changed, and opens a pull request that
merges itself when CI passes. A broken link opens an issue.
The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that
looks like game prose or an account identifier. What the bot should log is in `docs/BOT_DATA_PROTOCOL.md`.