# Data sources Every source is declared in `collection/pipeline.yaml` and collected by `python -m collection update` (see `collection/README.md`). Every run from every source is in `public/runs_clean.csv`, with a `source` column. | Source (`source` column) | Where it lives here | |---|---| | Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) | | Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` | | Runs described in the r/fourthwing megathread comments (`reddit comment`) | `public/reddit_comments.json` (comment id, day and answers; no usernames or comment text) | | The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) | | DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) | | Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs | | Unofficial Dragon Register | `public/register.json` (rider names and photos dropped); no runs | | Discord | not yet: planned, with admin consent and anonymised usernames | Raw files (`raw/`, git-ignored) are never published: the current version of each source and the one before it. `manifest.json` records the current raw file's checksum and when it last changed. ## What each column means `data/dictionary.yaml` describes every published file and every column of `data/public/runs_clean.csv`: its type, what it means and every value it can take. `data/croissant.json` says the same in the [Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so dataset tools can load and cite the data. `python -m collection build` rewrites it; the tests fail if a column or value is not described or the metadata is out of date.