Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+26
-9
@@ -1,7 +1,7 @@
|
||||
# Data dictionary: every published file and every column of the coded runs.
|
||||
#
|
||||
# collection/publish_metadata.py builds data/croissant.json (Croissant, the MLCommons metadata format) from this
|
||||
# file, data/sources.json and the licences. tests/core/test_dictionary.py checks that every column of the coded runs
|
||||
# collection/etl/metadata.py builds data/croissant.json (Croissant, the MLCommons metadata format) from this
|
||||
# file, collection/pipeline.yaml and the licences. tests/core/test_dictionary.py checks that every column of the coded runs
|
||||
# is described here and that every value it takes is listed.
|
||||
#
|
||||
# Missing answers: "Unknown", "Unsure", "(not asked)" and the other values in core/spec/form.yaml missing_values carry
|
||||
@@ -25,16 +25,22 @@ dataset:
|
||||
|
||||
files:
|
||||
runs_clean:
|
||||
path: data/derived/runs_clean.csv
|
||||
path: data/public/runs_clean.csv
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/runs_clean.csv
|
||||
format: text/csv
|
||||
description: One coded run per row of the community form (core/atlas/coding.py, rules in core/spec/form.yaml).
|
||||
description: >-
|
||||
Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's
|
||||
rules are core/spec/form.yaml). The `source` column says where each run came from.
|
||||
form_responses:
|
||||
path: data/public/form_responses.json
|
||||
format: application/json
|
||||
description: >-
|
||||
The community "Threshing Routes" form export as published (headers and rows), with email addresses in
|
||||
free-text answers redacted. The source of runs_clean.
|
||||
reports:
|
||||
path: data/public/reports.json
|
||||
format: application/json
|
||||
description: Runs reported through the Atlas's "Report a run" form (GitHub issues), answers only, no usernames.
|
||||
register:
|
||||
path: data/public/register.json
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/register.json
|
||||
@@ -58,19 +64,30 @@ files:
|
||||
path: data/public/graph.json
|
||||
url: https://ekcschiano.github.io/threshing-atlas/data/graph.json
|
||||
format: application/json
|
||||
description: The route graph over every source (form and bot), counted per source (analysis/network/build-graph.js).
|
||||
description: The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).
|
||||
|
||||
# The columns of runs_clean. type: text, integer, boolean or date. values: every value the column may take
|
||||
# (empty cells are always allowed and mean "does not apply" or "not recorded").
|
||||
columns:
|
||||
run_id: { type: integer, description: Row number in the form export (1 = first row). }
|
||||
run_id: { type: text, required: true, description: "Unique id of the run: sheet-<row of the form export>, thread-<result id>, reported-<issue number> or bot-<hash>." }
|
||||
source:
|
||||
type: text
|
||||
required: true
|
||||
description: >-
|
||||
"Form" for a form submission (it has a timestamp). Bulk-pasted logs keep the label they were pasted with;
|
||||
they record every run of the players who kept them, so they are the least selectively reported rows.
|
||||
Where the run came from. "reddit spreadsheet": the community Threshing Routes form and spreadsheet linked from
|
||||
the r/fourthwing megathreads. "reddit thread": a dragonkind.com result page posted in a megathread (bonded,
|
||||
route not recorded). "reported": the Atlas's own "Report a run" form. "bot": an attempt logged by the
|
||||
DragonKind bot. "reddit pdf": the community route PDF (a flowchart of the routes, not a log of runs, so it
|
||||
adds no rows today).
|
||||
values: [reddit spreadsheet, reddit thread, reported, bot, reddit pdf]
|
||||
entry:
|
||||
type: text
|
||||
description: >-
|
||||
For spreadsheet runs, the sheet's own row label: "Form" for a form submission (it has a timestamp);
|
||||
bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept
|
||||
them, so they are the least selectively reported rows. Empty for other sources.
|
||||
values: [Form, Data Import, BulkUpload3, Manual Upload, Historical import, Unknown import]
|
||||
date: { type: date, description: Date of a form submission (YYYY-MM-DD). Empty for bulk-pasted rows. }
|
||||
date: { type: date, description: "Date of the run or its report (YYYY-MM-DD): the form submission, the issue, or the bot attempt. Empty for bulk-pasted rows and result pages." }
|
||||
wing: { type: text, description: The rider's wing, from the form., values: [First, Second, Third, Fourth, Multiple, Unknown] }
|
||||
section: { type: text, description: The rider's section within the wing (free text as answered). }
|
||||
squad: { type: text, description: The rider's squad., values: [First, Second, Third, "Don't know", Unknown] }
|
||||
|
||||
Reference in New Issue
Block a user