Files
kschianoandClaude Opus 5.5 197c16dac0
CI / test (pull_request) Successful in 46s
Receive runs from the Reddit megathread comments (source "reddit comment")
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs
riders describe in the r/fourthwing megathreads and delivers them here by
pull request as data/public/reddit_comments.json: each run is the "Report a
run" answers (option labels of collection/specs/report_form.yaml), with the
comment id and day only; no usernames, no comment text.

- pipeline.yaml: a reddit_comments source of extract kind "delivered" (the
  atlas fetches nothing; `update` skips it with a note)
- runs.py: the coder, sharing one answers_row() with the reported coder;
  every row is flagged as read from a Reddit comment (and "not checked by
  hand" until corrected); a result page a comment already carries is not
  counted twice as a "reddit thread" row
- collection/manual/reddit_comments.yaml: reject / fix / add, applied at
  build time
- dictionary, SOURCES, README, croissant: the new source

The route model and statistics do not use the source until it is named in
planner/policy.yaml / analysis/stats/measures.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:06:14 -05:00

27 lines
1.9 KiB
Markdown

# Data sources
Every source is declared in `collection/pipeline.yaml` and collected by `python -m collection update`
(see `collection/README.md`). Every run from every source is in `public/runs_clean.csv`, with a `source` column.
| Source (`source` column) | Where it lives here |
|---|---|
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
| Runs described in the r/fourthwing megathread comments (`reddit comment`) | `public/reddit_comments.json` (comment id, day and answers; no usernames or comment text) |
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
| Unofficial Dragon Register | `public/register.json` (rider names and photos dropped); no runs |
| Discord | not yet: planned, with admin consent and anonymised usernames |
Raw files (`raw/`, git-ignored) are never published: the current version of each source and the one before it.
`manifest.json` records the current raw file's checksum and when it last changed.
## What each column means
`data/dictionary.yaml` describes every published file and every column of `data/public/runs_clean.csv`: its type,
what it means and every value it can take. `data/croissant.json` says the same in the
[Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so
dataset tools can load and cite the data. `python -m collection build` rewrites it; the tests fail if a column or
value is not described or the metadata is out of date.