Files
threshing-atlas/collection
ekcSchianoandClaude Opus 5.5 63b5d40e54 Step 5: data dictionary and Croissant metadata
- data/dictionary.yaml: every published file, and every column of runs_clean.csv
  with its type, meaning and allowed values
- collection/publish_metadata.py writes data/croissant.json (Croissant 1.0) from
  it, with a sha256 per file; `npm run metadata`, run by the daily update
- tests/core/test_dictionary.py: every column and value described, types hold,
  croissant.json up to date and checksums match
- attempt is now written as a whole number (was 1.0, 2.0); no statistic changes
- the site ships both files and the Methods text links them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-04 01:37:10 +00:00
..

collection

Getting data in. Everything here turns a source into the shared run format (see core/) or into public files under data/.

Source Code Output
Community "Threshing Routes" form (Google Sheet) sources/export_raw.py, then core/atlas/coding.py data/public/form_responses.json, data/derived/runs_clean.csv
Unofficial Dragon Register (Google Sheet) sources/export_register.py data/public/register.json (rider names stay local)
Community route PDF sources/update_sources.py data/derived/pdf_fingerprint.json only; the PDF is never committed
DragonKind bot (Ken) bot/import-bot.js, bot/bot.js data/bot/scenes.json, data/bot/trajectories-<tag>.jsonl
Reddit megathreads not collected yet —

sources/update_sources.py runs every day from .github/workflows/update-sources.yml. It downloads the sheets and the PDF (links in data/sources.json), rebuilds the data if anything changed, and opens a pull request that merges itself when CI passes. A broken link opens an issue.

The bot importer only ever writes fingerprints, short labels, choice text and outcomes. It refuses anything that looks like game prose or an account identifier. What the bot should log is in docs/BOT_DATA_PROTOCOL.md.