Receive runs from the Reddit megathread comments (source "reddit comment") #18

Merged
kathleen merged 1 commits from data/reddit-comment-runs into main 2026-11-19 03:17:00 +00:00
Owner

Note: the game-rule checks described below were pushed after this PR merged, so they are not part of it. They are in #37.

Adds the receiving side for runs from the r/fourthwing Dragonkind megathread comments, a new source reddit comment.

How the data arrives

The threshing-reddit scraper reads the megathreads every 6 hours. It delivers by pull request only: it opens (or updates) one PR from data/reddit-comments that changes data/public/reddit_comments.json, for a person to review. It never pushes to main and never merges itself.

Each run is the "Report a run" answers: the option labels of collection/specs/report_form.yaml. The scraper checks them against this repo's live form before sending. Kept per run: comment id, thread id, day, answers, and a result-page id if linked. No usernames, no comment text.

What changes here

  • collection/pipeline.yaml: a reddit_comments source with extract kind delivered. The atlas fetches nothing; update skips it with a note.
  • collection/etl/runs.py: the coder. It shares one answers_row() with the reported coder (that refactor is byte-identical: runs_clean.csv is unchanged).
    • Every row is flagged "read from a Reddit comment", plus "automatically, not checked by hand" until a correction touches it.
    • A result page that a comment already carries is not counted again as a reddit thread row.
  • collection/manual/reddit_comments.yaml: reject / fix / add, applied at build time, so a correction never waits for the threads to change.
  • Dictionary, SOURCES.md, collection README, croissant: the new source value and file.
  • Tests: hand corrections, flags, dictionary validity, and the delivered source being skipped by update.

data/public/reddit_comments.json starts empty; the first scraper PR fills it.

Game-rule checks (second commit)

Runs answered through "Report a run" (reported and reddit comment) now get the checks the community form gets in core/atlas/coding.py. Each broken rule is a data_flags entry; the run is kept.

  • First dragon: outside the documented pool for the company and direction.
  • Encounter dragon: a bond or burn by a dragon the encounter doesn't have. This uses a new encounter_colours table in core/spec/form.yaml; Orange/Brown after the two Greens is allowed, as a Green's rescue.
  • Valley Green: a Green on a valley run.
  • Cadet path: an outcome off its scream path; fight answers or a post-fight path on a run that didn't help; a second fight move without assessing.
  • Early death: died before the hour, yet reached the tree, the valley or an encounter.

Route model: planner/policy.yaml gains drop_flagged. Flagged runs of reddit comment stay out of the model; the "read from a Reddit comment" note doesn't count. A hand fix in collection/manual/reddit_comments.yaml that clears the flags brings a run back. This has no effect until reddit comment is added to sources; reported is unchanged.

On the first 82 scraped runs: 6 are flagged (shadow with Brown/Red, Green after jumping on the Red, Red in the meadow, Orange first for Alone/Center, cadet-path mixes).

Not in this PR

Route model and statistics: they don't use the source until reddit comment is named in planner/policy.yaml sources / analysis/stats/measures.yaml.

Checks

python -m collection build --check, 89 Python tests, 17 JS tests, and graph.json unchanged.

🤖 Generated with Claude Code

> **Note:** the game-rule checks described below were pushed after this PR merged, so they are not part of it. They are in #37. Adds the receiving side for runs from the r/fourthwing Dragonkind megathread comments, a new source `reddit comment`. ## How the data arrives The [threshing-reddit](https://git.kenschiano.net/DragonKind/threshing-reddit) scraper reads the megathreads every 6 hours. It delivers by **pull request only**: it opens (or updates) one PR from `data/reddit-comments` that changes `data/public/reddit_comments.json`, for a person to review. It never pushes to `main` and never merges itself. Each run is the **"Report a run" answers**: the option labels of `collection/specs/report_form.yaml`. The scraper checks them against this repo's live form before sending. Kept per run: comment id, thread id, day, answers, and a result-page id if linked. No usernames, no comment text. ## What changes here - **`collection/pipeline.yaml`**: a `reddit_comments` source with extract kind `delivered`. The atlas fetches nothing; `update` skips it with a note. - **`collection/etl/runs.py`**: the coder. It shares one `answers_row()` with the `reported` coder (that refactor is byte-identical: `runs_clean.csv` is unchanged). - Every row is flagged "read from a Reddit comment", plus "automatically, not checked by hand" until a correction touches it. - A result page that a comment already carries is not counted again as a `reddit thread` row. - **`collection/manual/reddit_comments.yaml`**: `reject` / `fix` / `add`, applied at build time, so a correction never waits for the threads to change. - **Dictionary, `SOURCES.md`, collection README, croissant**: the new source value and file. - **Tests**: hand corrections, flags, dictionary validity, and the delivered source being skipped by `update`. `data/public/reddit_comments.json` starts empty; the first scraper PR fills it. ## Game-rule checks (second commit) Runs answered through "Report a run" (`reported` and `reddit comment`) now get the checks the community form gets in `core/atlas/coding.py`. Each broken rule is a `data_flags` entry; the run is kept. - **First dragon:** outside the documented pool for the company and direction. - **Encounter dragon:** a bond or burn by a dragon the encounter doesn't have. This uses a new `encounter_colours` table in `core/spec/form.yaml`; Orange/Brown after the two Greens is allowed, as a Green's rescue. - **Valley Green:** a Green on a valley run. - **Cadet path:** an outcome off its scream path; fight answers or a post-fight path on a run that didn't help; a second fight move without assessing. - **Early death:** died before the hour, yet reached the tree, the valley or an encounter. **Route model:** `planner/policy.yaml` gains `drop_flagged`. Flagged runs of `reddit comment` stay out of the model; the "read from a Reddit comment" note doesn't count. A hand fix in `collection/manual/reddit_comments.yaml` that clears the flags brings a run back. This has no effect until `reddit comment` is added to `sources`; `reported` is unchanged. **On the first 82 scraped runs:** 6 are flagged (shadow with Brown/Red, Green after jumping on the Red, Red in the meadow, Orange first for Alone/Center, cadet-path mixes). ## Not in this PR **Route model and statistics:** they don't use the source until `reddit comment` is named in `planner/policy.yaml` `sources` / `analysis/stats/measures.yaml`. ## Checks `python -m collection build --check`, 89 Python tests, 17 JS tests, and graph.json unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kschiano added 1 commit 2026-11-19 03:06:46 +00:00
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs
riders describe in the r/fourthwing megathreads and delivers them here by
pull request as data/public/reddit_comments.json: each run is the "Report a
run" answers (option labels of collection/specs/report_form.yaml), with the
comment id and day only; no usernames, no comment text.

- pipeline.yaml: a reddit_comments source of extract kind "delivered" (the
  atlas fetches nothing; `update` skips it with a note)
- runs.py: the coder, sharing one answers_row() with the reported coder;
  every row is flagged as read from a Reddit comment (and "not checked by
  hand" until corrected); a result page a comment already carries is not
  counted twice as a "reddit thread" row
- collection/manual/reddit_comments.yaml: reject / fix / add, applied at
  build time
- dictionary, SOURCES, README, croissant: the new source

The route model and statistics do not use the source until it is named in
planner/policy.yaml / analysis/stats/measures.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kathleen merged commit 27ba5fbcbc into main 2026-11-19 03:17:00 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#18