Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+18
-11
@@ -1,18 +1,25 @@
|
||||
# Data sources
|
||||
|
||||
| Source | Where it lives here | How to update |
|
||||
|---|---|---|
|
||||
| Community "Threshing Routes" Google Form (u/ClumsyBadger) | `raw/form/*.xlsx` → `public/form_responses.json` | Download the sheet as .xlsx, then `python collection/sources/export_raw.py` |
|
||||
| Community route PDF (u/ClumsyBadger) | not redistributed; linked from the Reddit megathreads | A scene crosswalk will be kept in `derived/` |
|
||||
| Unofficial Dragon Register | `raw/register/` (git-ignored, has rider names) → `public/register.json` | `python collection/sources/export_register.py` |
|
||||
| Reddit megathreads (r/fourthwing, Dragonkind bonding Parts 3 and 4) | qualitative; routes cited in the valley map | manual |
|
||||
| DragonKind bot exports | `bot/runs-YYYY-MM-DD.jsonl` | by pull request, see `docs/BOT_DATA_PROTOCOL.md` |
|
||||
| Discord | not yet | planned, with admin consent and anonymised usernames |
|
||||
Every source is declared in `collection/pipeline.yaml` and collected by `python -m collection update`
|
||||
(see `collection/README.md`). Every run from every source is in `public/runs_clean.csv`, with a `source` column.
|
||||
|
||||
| Source (`source` column) | Where it lives here |
|
||||
|---|---|
|
||||
| Community "Threshing Routes" form and spreadsheet, u/ClumsyBadger (`reddit spreadsheet`) | `public/form_responses.json` (emails redacted) |
|
||||
| Result pages posted in the r/fourthwing megathreads (`reddit thread`) | `collection/manual/reddit_results.yaml` |
|
||||
| The Atlas's "Report a run" form (`reported`) | `public/reports.json` (answers only, no usernames) |
|
||||
| DragonKind bot exports (`bot`) | `bot/scenes.json`, `bot/trajectories-<date>.jsonl` (anonymised; see `docs/BOT_DATA_PROTOCOL.md`) |
|
||||
| Community route PDF, u/ClumsyBadger (`reddit pdf`) | not redistributed: `derived/pdf_fingerprint.json` only; it maps the choices and holds no runs |
|
||||
| Unofficial Dragon Register | `public/register.json` (rider names and photos dropped); no runs |
|
||||
| Discord | not yet: planned, with admin consent and anonymised usernames |
|
||||
|
||||
Raw files (`raw/`, git-ignored) are never published: the current version of each source and the one before it.
|
||||
`manifest.json` records the current raw file's checksum and when it last changed.
|
||||
|
||||
## What each column means
|
||||
|
||||
`data/dictionary.yaml` describes every published file and every column of `data/derived/runs_clean.csv`: its type,
|
||||
`data/dictionary.yaml` describes every published file and every column of `data/public/runs_clean.csv`: its type,
|
||||
what it means and every value it can take. `data/croissant.json` says the same in the
|
||||
[Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so
|
||||
dataset tools can load and cite the data. After changing the data or the dictionary, run `npm run metadata`; the
|
||||
tests fail if a column or value is not described or the metadata is out of date.
|
||||
dataset tools can load and cite the data. `python -m collection build` rewrites it; the tests fail if a column or
|
||||
value is not described or the metadata is out of date.
|
||||
|
||||
+50
-15
@@ -67,13 +67,23 @@
|
||||
"isBasedOn": [
|
||||
{
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Threshing Routes form responses (u/ClumsyBadger)",
|
||||
"name": "Threshing Routes form and spreadsheet (u/ClumsyBadger, linked from the r/fourthwing megathreads)",
|
||||
"url": "https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0"
|
||||
},
|
||||
{
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Unofficial Dragon Register responses",
|
||||
"name": "Unofficial Dragon Register (which dragon each rider bonded)",
|
||||
"url": "https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ"
|
||||
},
|
||||
{
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Runs posted in the r/fourthwing Dragonkind megathreads",
|
||||
"url": "https://www.reddit.com/r/fourthwing/comments/1wvwfp9/"
|
||||
},
|
||||
{
|
||||
"@type": "sc:CreativeWork",
|
||||
"name": "Runs reported on the Atlas (\"Report a run\", a GitHub issue form)",
|
||||
"url": "https://github.com/ekcSchiano/threshing-atlas/issues/new?template=report-a-run.yml"
|
||||
}
|
||||
],
|
||||
"distribution": [
|
||||
@@ -81,11 +91,11 @@
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "runs_clean",
|
||||
"name": "runs_clean",
|
||||
"description": "One coded run per row of the community form (core/atlas/coding.py, rules in core/spec/form.yaml).",
|
||||
"description": "Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's rules are core/spec/form.yaml). The `source` column says where each run came from.",
|
||||
"contentUrl": "https://ekcschiano.github.io/threshing-atlas/data/runs_clean.csv",
|
||||
"encodingFormat": "text/csv",
|
||||
"sha256": "ef34b36881b6bc0f57453e425b1c7d76155e6572463aff0d7e89b9857823f7db",
|
||||
"contentSize": "1991643 B"
|
||||
"sha256": "766d8708dad77327534063df42cd4ff806ab707497618ab5263cf3dce98a30c7",
|
||||
"contentSize": "2146368 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
@@ -97,6 +107,16 @@
|
||||
"sha256": "4c5791f78ce81f5e88f1fb99d933acd06d8223ff9a573e96b6fcb3a6fd9c8d22",
|
||||
"contentSize": "8293395 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "reports",
|
||||
"name": "reports",
|
||||
"description": "Runs reported through the Atlas's \"Report a run\" form (GitHub issues), answers only, no usernames.",
|
||||
"contentUrl": "https://github.com/ekcSchiano/threshing-atlas/blob/main/data/public/reports.json",
|
||||
"encodingFormat": "application/json",
|
||||
"sha256": "d42eb04b9923eb1d2a8e5c356452f52daab78293be5a29a448cfb69c974eab30",
|
||||
"contentSize": "92 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "register",
|
||||
@@ -114,8 +134,8 @@
|
||||
"description": "Every number the Atlas page shows, for six filter presets (analysis/stats/site_stats.py; what is measured is analysis/stats/measures.yaml).",
|
||||
"contentUrl": "https://ekcschiano.github.io/threshing-atlas/data/stats.json",
|
||||
"encodingFormat": "application/json",
|
||||
"sha256": "d48b1da5176d6e85f6ee093cfb4f50dfbc6f71cfa7fec8ecb49a7c32c14f6d53",
|
||||
"contentSize": "4269825 B"
|
||||
"sha256": "dc79c0762e5c27423f9fa2086f06aa7019b0aec6ae41d6a6dcbe59a1016cffab",
|
||||
"contentSize": "4269546 B"
|
||||
},
|
||||
{
|
||||
"@type": "cr:FileObject",
|
||||
@@ -131,11 +151,11 @@
|
||||
"@type": "cr:FileObject",
|
||||
"@id": "graph",
|
||||
"name": "graph",
|
||||
"description": "The route graph over every source (form and bot), counted per source (analysis/network/build-graph.js).",
|
||||
"description": "The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).",
|
||||
"contentUrl": "https://ekcschiano.github.io/threshing-atlas/data/graph.json",
|
||||
"encodingFormat": "application/json",
|
||||
"sha256": "043227ff7d858fa8e73e7447f4cd7171bb216ab8d203b036df056d8ed542e446",
|
||||
"contentSize": "48634 B"
|
||||
"sha256": "72ef6fa6d6edc3a53026eff3f9cf39a883f7a709eacaaf21351c9ca1251a986a",
|
||||
"contentSize": "48783 B"
|
||||
}
|
||||
],
|
||||
"recordSet": [
|
||||
@@ -143,7 +163,7 @@
|
||||
"@type": "cr:RecordSet",
|
||||
"@id": "runs",
|
||||
"name": "runs",
|
||||
"description": "One coded threshing run per row (data/derived/runs_clean.csv).",
|
||||
"description": "One coded threshing run per row, from every source (data/public/runs_clean.csv).",
|
||||
"key": {
|
||||
"@id": "runs/run_id"
|
||||
},
|
||||
@@ -152,8 +172,8 @@
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/run_id",
|
||||
"name": "run_id",
|
||||
"description": "Row number in the form export (1 = first row).",
|
||||
"dataType": "sc:Integer",
|
||||
"description": "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>.",
|
||||
"dataType": "sc:Text",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
"@id": "runs_clean"
|
||||
@@ -167,7 +187,7 @@
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/source",
|
||||
"name": "source",
|
||||
"description": "\"Form\" for a form submission (it has a timestamp). Bulk-pasted logs keep the label they were pasted with; they record every run of the players who kept them, so they are the least selectively reported rows. Values: Form; Data Import; BulkUpload3; Manual Upload; Historical import; Unknown import.",
|
||||
"description": "Where the run came from. \"reddit spreadsheet\": the community Threshing Routes form and spreadsheet linked from the r/fourthwing megathreads. \"reddit thread\": a dragonkind.com result page posted in a megathread (bonded, route not recorded). \"reported\": the Atlas's own \"Report a run\" form. \"bot\": an attempt logged by the DragonKind bot. \"reddit pdf\": the community route PDF (a flowchart of the routes, not a log of runs, so it adds no rows today). Values: reddit spreadsheet; reddit thread; reported; bot; reddit pdf.",
|
||||
"dataType": "sc:Text",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
@@ -178,11 +198,26 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/entry",
|
||||
"name": "entry",
|
||||
"description": "For spreadsheet runs, the sheet's own row label: \"Form\" for a form submission (it has a timestamp); bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept them, so they are the least selectively reported rows. Empty for other sources. Values: Form; Data Import; BulkUpload3; Manual Upload; Historical import; Unknown import.",
|
||||
"dataType": "sc:Text",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
"@id": "runs_clean"
|
||||
},
|
||||
"extract": {
|
||||
"column": "entry"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"@type": "cr:Field",
|
||||
"@id": "runs/date",
|
||||
"name": "date",
|
||||
"description": "Date of a form submission (YYYY-MM-DD). Empty for bulk-pasted rows.",
|
||||
"description": "Date of the run or its report (YYYY-MM-DD): the form submission, the issue, or the bot attempt. Empty for bulk-pasted rows and result pages.",
|
||||
"dataType": "sc:Date",
|
||||
"source": {
|
||||
"fileObject": {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+26
-9
@@ -1,7 +1,7 @@
|
||||
# Data dictionary: every published file and every column of the coded runs.
|
||||
#
|
||||
# collection/publish_metadata.py builds data/croissant.json (Croissant, the MLCommons metadata format) from this
|
||||
# file, data/sources.json and the licences. tests/core/test_dictionary.py checks that every column of the coded runs
|
||||
# collection/etl/metadata.py builds data/croissant.json (Croissant, the MLCommons metadata format) from this
|
||||
# file, collection/pipeline.yaml and the licences. tests/core/test_dictionary.py checks that every column of the coded runs
|
||||
# is described here and that every value it takes is listed.
|
||||
#
|
||||
# Missing answers: "Unknown", "Unsure", "(not asked)" and the other values in core/spec/form.yaml missing_values carry
|
||||
@@ -25,16 +25,22 @@ dataset:
|
||||
|
||||
files:
|
||||
runs_clean:
|
||||
path: data/derived/runs_clean.csv
|
||||
path: data/public/runs_clean.csv
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/runs_clean.csv
|
||||
format: text/csv
|
||||
description: One coded run per row of the community form (core/atlas/coding.py, rules in core/spec/form.yaml).
|
||||
description: >-
|
||||
Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's
|
||||
rules are core/spec/form.yaml). The `source` column says where each run came from.
|
||||
form_responses:
|
||||
path: data/public/form_responses.json
|
||||
format: application/json
|
||||
description: >-
|
||||
The community "Threshing Routes" form export as published (headers and rows), with email addresses in
|
||||
free-text answers redacted. The source of runs_clean.
|
||||
reports:
|
||||
path: data/public/reports.json
|
||||
format: application/json
|
||||
description: Runs reported through the Atlas's "Report a run" form (GitHub issues), answers only, no usernames.
|
||||
register:
|
||||
path: data/public/register.json
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/register.json
|
||||
@@ -58,19 +64,30 @@ files:
|
||||
path: data/public/graph.json
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/graph.json
|
||||
format: application/json
|
||||
description: The route graph over every source (form and bot), counted per source (analysis/network/build-graph.js).
|
||||
description: The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).
|
||||
|
||||
# The columns of runs_clean. type: text, integer, boolean or date. values: every value the column may take
|
||||
# (empty cells are always allowed and mean "does not apply" or "not recorded").
|
||||
columns:
|
||||
run_id: { type: integer, description: Row number in the form export (1 = first row). }
|
||||
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>." }
|
||||
source:
|
||||
type: text
|
||||
required: true
|
||||
description: >-
|
||||
"Form" for a form submission (it has a timestamp). Bulk-pasted logs keep the label they were pasted with;
|
||||
they record every run of the players who kept them, so they are the least selectively reported rows.
|
||||
Where the run came from. "reddit spreadsheet": the community Threshing Routes form and spreadsheet linked from
|
||||
the r/fourthwing megathreads. "reddit thread": a dragonkind.com result page posted in a megathread (bonded,
|
||||
route not recorded). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
|
||||
DragonKind bot. "reddit pdf": the community route PDF (a flowchart of the routes, not a log of runs, so it
|
||||
adds no rows today).
|
||||
values: [reddit spreadsheet, reddit thread, reported, bot, reddit pdf]
|
||||
entry:
|
||||
type: text
|
||||
description: >-
|
||||
For spreadsheet runs, the sheet's own row label: "Form" for a form submission (it has a timestamp);
|
||||
bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept
|
||||
them, so they are the least selectively reported rows. Empty for other sources.
|
||||
values: [Form, Data Import, BulkUpload3, Manual Upload, Historical import, Unknown import]
|
||||
date: { type: date, description: Date of a form submission (YYYY-MM-DD). Empty for bulk-pasted rows. }
|
||||
date: { type: date, description: "Date of the run or its report (YYYY-MM-DD): the form submission, the issue, or the bot attempt. Empty for bulk-pasted rows and result pages." }
|
||||
wing: { type: text, description: The rider's wing, from the form., values: [First, Second, Third, Fourth, Multiple, Unknown] }
|
||||
section: { type: text, description: The rider's section within the wing (free text as answered). }
|
||||
squad: { type: text, description: The rider's squad., values: [First, Second, Third, "Don't know", Unknown] }
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"reported": {
|
||||
"sha256": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945",
|
||||
"bytes": 2,
|
||||
"changed": "2026-11-04"
|
||||
}
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"source":"Runs reported on the Atlas (\"Report a run\", a GitHub issue form)","reports":[]}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"form": {
|
||||
"name": "Threshing Routes form responses (u/ClumsyBadger)",
|
||||
"view": "https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0",
|
||||
"download": "https://docs.google.com/spreadsheets/d/1gnw1AdRg1XT4JsOfRM6QTA0jSaS3n9ZF-rPWnYewzv0/export?format=xlsx",
|
||||
"filename": "Threshing_Routes_Responses.xlsx"
|
||||
},
|
||||
"register": {
|
||||
"name": "Unofficial Dragon Register responses",
|
||||
"view": "https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ",
|
||||
"download": "https://docs.google.com/spreadsheets/d/1KMiE9sec0l597Pzue1OhGYf07NF4howtQ_lPeqx7jXQ/export?format=xlsx",
|
||||
"filename": "Unofficial_Dragon_Register_Responses.xlsx"
|
||||
},
|
||||
"pdf": {
|
||||
"name": "Community route PDF (u/ClumsyBadger)",
|
||||
"view": "https://drive.google.com/file/d/17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg/view",
|
||||
"download": "https://drive.google.com/uc?export=download&id=17o72ehHsSPW4UdsKF-sXTxMLEuinWeEg",
|
||||
"filename": "Threshing_Options.pdf"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user