Receive runs from the Reddit megathread comments (source "reddit comment")
CI / test (pull_request) Successful in 46s
CI / test (pull_request) Successful in 46s
The threshing-reddit scraper (DragonKind/threshing-reddit) reads the runs riders describe in the r/fourthwing megathreads and delivers them here by pull request as data/public/reddit_comments.json: each run is the "Report a run" answers (option labels of collection/specs/report_form.yaml), with the comment id and day only; no usernames, no comment text. - pipeline.yaml: a reddit_comments source of extract kind "delivered" (the atlas fetches nothing; `update` skips it with a note) - runs.py: the coder, sharing one answers_row() with the reported coder; every row is flagged as read from a Reddit comment (and "not checked by hand" until corrected); a result page a comment already carries is not counted twice as a "reddit thread" row - collection/manual/reddit_comments.yaml: reject / fix / add, applied at build time - dictionary, SOURCES, README, croissant: the new source The route model and statistics do not use the source until it is named in planner/policy.yaml / analysis/stats/measures.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ Every source is declared in `collection/pipeline.yaml` and collected by `python
|
||||
|---|---|
|
||||
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
|
||||
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
|
||||
| Runs described in the r/fourthwing megathread comments (`reddit comment`) | `public/reddit_comments.json` (comment id, day and answers; no usernames or comment text) |
|
||||
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
|
||||
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
|
||||
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
|
||||
|
||||
+17
-2
@@ -78,6 +78,11 @@
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Runs posted in the r/fourthwing Dragonkind megathreads",
|
||||
"url": "https://www.reddit.com/r/fourthwing/comments/1wvwfp9/"
|
||||
},
|
||||
{
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Runs riders describe in the r/fourthwing Dragonkind megathread comments",
|
||||
"url": "https://www.reddit.com/r/fourthwing/comments/1wvwfp9/"
|
||||
}
|
||||
],
|
||||
"distribution": [
|
||||
@@ -101,6 +106,16 @@
|
||||
"sha256": "319f6217e0a431454114a67276f429379ec383cd6ed2666443b48b8f48c66490",
|
||||
"contentSize": "12623695 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "reddit_comments",
|
||||
"name": "reddit_comments",
|
||||
"description": "Runs described in the r/fourthwing megathread comments, read into the \"Report a run\" answers: comment id, day and answers only, no usernames or comment text.",
|
||||
"contentUrl": "https://git.kenschiano.net/DragonKind/threshing-atlas/raw/branch/main/data/public/reddit_comments.json",
|
||||
"encodingFormat": "application/json",
|
||||
"sha256": "c180f8b82ed1119d4ff28697fcf45d0c7f22859597cd10f5e82a2e50ca1dc51a",
|
||||
"contentSize": "105 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "reports",
|
||||
@@ -166,7 +181,7 @@
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/run_id",
|
||||
"name": "run_id",
|
||||
"description": "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>.",
|
||||
"description": "Unique id of the run: sheet-<row of the form export>, thread-<result id>, comment-<Reddit comment id>, reported-<issue number> or bot-<hash>.",
|
||||
"dataType": "sc:Text",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
@@ -181,7 +196,7 @@
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/source",
|
||||
"name": "source",
|
||||
"description": "Where the run came from. \"reddit spreadsheet\": the community Threshing Routes form and spreadsheet linked from the r/fourthwing megathreads. \"reddit thread\": a dragonkind.com result page posted in a megathread (bonded, route not recorded). \"reported\": the Atlas's own \"Report a run\" form. \"bot\": an attempt logged by the DragonKind bot. \"register\": a bonded dragon entered in the community's Unofficial Dragon Register (outcome only, no route; the same run may also be in the spreadsheet). \"reddit pdf\": the community route PDF (a flowchart of the routes, not a log of runs, so it adds no rows today). Values: reddit spreadsheet; reddit thread; register; reported; bot; reddit pdf.",
|
||||
"description": "Where the run came from. \"reddit spreadsheet\": the community Threshing Routes form and spreadsheet linked from the r/fourthwing megathreads. \"reddit thread\": a dragonkind.com result page posted in a megathread (bonded, route not recorded). \"reddit comment\": a run a rider described in a megathread comment, read automatically into the \"Report a run\" answers by the threshing-reddit scraper, which delivers them by pull request (hand corrections in collection/manual/reddit_comments.yaml). \"reported\": the Atlas's own \"Report a run\" form. \"bot\": an attempt logged by the DragonKind bot. \"register\": a bonded dragon entered in the community's Unofficial Dragon Register (outcome only, no route; the same run may also be in the spreadsheet). \"reddit pdf\": the community route PDF (a flowchart of the routes, not a log of runs, so it adds no rows today). Values: reddit spreadsheet; reddit thread; reddit comment; register; reported; bot; reddit pdf.",
|
||||
"dataType": "sc:Text",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
|
||||
+11
-3
@@ -35,6 +35,12 @@ files:
|
||||
description: >-
|
||||
The community "Threshing Routes" form export as published (headers and rows), with email addresses in
|
||||
free-text answers redacted. The source of runs_clean.
|
||||
reddit_comments:
|
||||
path: data/public/reddit_comments.json
|
||||
format: application/json
|
||||
description: >-
|
||||
Runs described in the r/fourthwing megathread comments, read into the "Report a run" answers: comment id, day
|
||||
and answers only, no usernames or comment text.
|
||||
reports:
|
||||
path: data/public/reports.json
|
||||
format: application/json
|
||||
@@ -63,18 +69,20 @@ files:
|
||||
# The columns of runs_clean. type: text, integer, boolean or date. values: every value the column may take
|
||||
# (empty cells are always allowed and mean "does not apply" or "not recorded").
|
||||
columns:
|
||||
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>." }
|
||||
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, comment-<Reddit comment id>, reported-<issue number> or bot-<hash>." }
|
||||
source:
|
||||
type: text
|
||||
required: true
|
||||
description: >-
|
||||
Where the run came from. "reddit spreadsheet": the community Threshing Routes form and spreadsheet linked from
|
||||
the r/fourthwing megathreads. "reddit thread": a dragonkind.com result page posted in a megathread (bonded,
|
||||
route not recorded). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
|
||||
route not recorded). "reddit comment": a run a rider described in a megathread comment, read automatically into
|
||||
the "Report a run" answers by the threshing-reddit scraper, which delivers them by pull request (hand
|
||||
corrections in collection/manual/reddit_comments.yaml). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
|
||||
DragonKind bot. "register": a bonded dragon entered in the community's Unofficial Dragon Register (outcome
|
||||
only, no route; the same run may also be in the spreadsheet). "reddit pdf": the community route PDF (a
|
||||
flowchart of the routes, not a log of runs, so it adds no rows today).
|
||||
values: [reddit spreadsheet, reddit thread, register, reported, bot, reddit pdf]
|
||||
values: [reddit spreadsheet, reddit thread, reddit comment, register, reported, bot, reddit pdf]
|
||||
entry:
|
||||
type: text
|
||||
description: >-
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"source":"Runs riders describe in the r/fourthwing Dragonkind megathread comments","runs":[],"unread":0}
|
||||
Reference in New Issue
Block a user