CI / test (pull_request) Successful in 46s
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs riders describe in the r/fourthwing megathreads and delivers them here by pull request as data/public/reddit_comments.json: each run is the "Report a run" answers (option labels of collection/specs/report_form.yaml), with the comment id and day only; no usernames, no comment text. - pipeline.yaml: a reddit_comments source of extract kind "delivered" (the atlas fetches nothing; `update` skips it with a note) - runs.py: the coder, sharing one answers_row() with the reported coder; every row is flagged as read from a Reddit comment (and "not checked by hand" until corrected); a result page a comment already carries is not counted twice as a "reddit thread" row - collection/manual/reddit_comments.yaml: reject / fix / add, applied at build time - dictionary, SOURCES, README, croissant: the new source The route model and statistics do not use the source until it is named in planner/policy.yaml / analysis/stats/measures.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27 lines
1.9 KiB
Markdown
27 lines
1.9 KiB
Markdown
# Data sources
|
|
|
|
Every source is declared in `collection/pipeline.yaml` and collected by `python -m collection update`
|
|
(see `collection/README.md`). Every run from every source is in `public/runs_clean.csv`, with a `source` column.
|
|
|
|
| Source (`source` column) | Where it lives here |
|
|
|---|---|
|
|
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
|
|
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
|
|
| Runs described in the r/fourthwing megathread comments (`reddit comment`) | `public/reddit_comments.json` (comment id, day and answers; no usernames or comment text) |
|
|
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
|
|
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
|
|
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
|
|
| Unofficial Dragon Register | `public/register.json` (rider names and photos dropped); no runs |
|
|
| Discord | not yet: planned, with admin consent and anonymised usernames |
|
|
|
|
Raw files (`raw/`, git-ignored) are never published: the current version of each source and the one before it.
|
|
`manifest.json` records the current raw file's checksum and when it last changed.
|
|
|
|
## What each column means
|
|
|
|
`data/dictionary.yaml` describes every published file and every column of `data/public/runs_clean.csv`: its type,
|
|
what it means and every value it can take. `data/croissant.json` says the same in the
|
|
[Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so
|
|
dataset tools can load and cite the data. `python -m collection build` rewrites it; the tests fail if a column or
|
|
value is not described or the metadata is out of date.
|