Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
61bc50583c
commit
3b24a11f5a
+14
-4
@@ -13,7 +13,7 @@ Rules every part of the project follows. If a change would break one, stop and a
|
||||
- **The community route PDF:** never committed or redistributed. Only a fingerprint
|
||||
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
|
||||
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
|
||||
names. The importer (`collection/bot/bot.js`) refuses to write anything that looks like either.
|
||||
names. The importer (`collection/etl/transform/bot.py`, rules in `collection/pipeline.yaml`) refuses to write anything that looks like either.
|
||||
|
||||
## The bot
|
||||
|
||||
@@ -25,10 +25,20 @@ Rules every part of the project follows. If a change would break one, stop and a
|
||||
## Sources we do not scrape
|
||||
|
||||
- **Discord:** not without the server admins' consent; never with a personal account token.
|
||||
- **Reddit:** not collected yet. When it is, it should go through Reddit's official API with an app key, not page
|
||||
scraping.
|
||||
- **Reddit:** only through Reddit's official API with an app key (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`),
|
||||
never by page scraping. Only dragonkind.com result links are taken from the threads; usernames and comment text
|
||||
are never kept, and each result page is read and added by hand (`collection/manual/reddit_results.yaml`).
|
||||
|
||||
## Run reports
|
||||
|
||||
"Report a run" creates a public GitHub issue. The pipeline keeps only the issue number, the date and the answers
|
||||
(`data/public/reports.json`); GitHub usernames are dropped. The form asks people not to include account names. A
|
||||
report closed as "not planned" or labelled `invalid` or `duplicate` is left out of the data.
|
||||
|
||||
## Secrets
|
||||
|
||||
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
|
||||
(`docs/REPO_SETTINGS.md`). Tokens go in secrets only: never in a file, issue, chat or commit.
|
||||
(`docs/REPO_SETTINGS.md`).
|
||||
- `RAW_ARCHIVE_KEY` encrypts the raw store (the current and previous raw file of each source, which include the
|
||||
register's rider names) in the Actions cache between daily runs. Without it nothing is archived.
|
||||
- `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` let the daily update read the megathreads through the Reddit API. Tokens go in secrets only: never in a file, issue, chat or commit.
|
||||
|
||||
Reference in New Issue
Block a user