- collection/pipeline.yaml declares every source (extract kind, transform, privacy rules, load target, run source); `python -m collection update|build` - extract keeps data/raw/<source>/current + previous only (git-ignored); data/manifest.json records the current raw checksum and change date - transforms are pure; validate checks runs against data/dictionary.yaml before load; load writes byte-stable files only when they change - data/public/runs_clean.csv: every run from every source with a new `source` column (reddit spreadsheet, reddit thread, reported, bot, reddit pdf); the sheet's own row label moves to `entry`; run_id is text - bot importer ported to Python (same output as bot.js on the fixture); labels and crosswalk in collection/specs/bot.yaml, written into graph.js - Reddit through the official API (result links to add by hand); the 12 result pages move to collection/manual/reddit_results.yaml - "Report a run": a GitHub issue form generated from collection/specs/report_form.yaml, collected as source "reported", plus a site tab pointing to it, the community spreadsheet and the megathread - analysis and the route model read runs_clean.csv (sources in measures.yaml); stats.json and policy.json unchanged apart from the factor's new name - daily workflow runs the pipeline; the raw store is kept between runs in the Actions cache, encrypted with RAW_ARCHIVE_KEY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
45 lines
2.6 KiB
Markdown
45 lines
2.6 KiB
Markdown
# Privacy and licences
|
|
|
|
Rules every part of the project follows. If a change would break one, stop and ask first.
|
|
|
|
## What may be published
|
|
|
|
- **Code** is MIT (`LICENSE`). **Data this project produces** is CC BY 4.0 (`DATA_LICENSE.md`). Third-party
|
|
sources keep their own terms.
|
|
- **Community form responses:** published as exported, with email addresses in free-text answers redacted by
|
|
the coding rules (`core/spec/form.yaml`, `core/atlas/coding.py`).
|
|
- **Dragon Register:** only `data/public/register.json`. The raw export has rider names and stays local
|
|
(`data/raw/register/` is git-ignored).
|
|
- **The community route PDF:** never committed or redistributed. Only a fingerprint
|
|
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
|
|
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
|
|
names. The importer (`collection/etl/transform/bot.py`, rules in `collection/pipeline.yaml`) refuses to write anything that looks like either.
|
|
|
|
## The bot
|
|
|
|
- Ken's bot repository stays private. Only the anonymised import comes here.
|
|
- Account ids are salted and hashed. The salt (`DK_ACCOUNT_SALT`) lives in the environment and is never
|
|
committed.
|
|
- The bot's browser profiles, `.env` and credential files are never committed or shared.
|
|
|
|
## Sources we do not scrape
|
|
|
|
- **Discord:** not without the server admins' consent; never with a personal account token.
|
|
- **Reddit:** only through Reddit's official API with an app key (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`),
|
|
never by page scraping. Only dragonkind.com result links are taken from the threads; usernames and comment text
|
|
are never kept, and each result page is read and added by hand (`collection/manual/reddit_results.yaml`).
|
|
|
|
## Run reports
|
|
|
|
"Report a run" creates a public GitHub issue. The pipeline keeps only the issue number, the date and the answers
|
|
(`data/public/reports.json`); GitHub usernames are dropped. The form asks people not to include account names. A
|
|
report closed as "not planned" or labelled `invalid` or `duplicate` is left out of the data.
|
|
|
|
## Secrets
|
|
|
|
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
|
|
(`docs/REPO_SETTINGS.md`).
|
|
- `RAW_ARCHIVE_KEY` encrypts the raw store (the current and previous raw file of each source, which include the
|
|
register's rider names) in the Actions cache between daily runs. Without it nothing is archived.
|
|
- `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` let the daily update read the megathreads through the Reddit API. Tokens go in secrets only: never in a file, issue, chat or commit.
|