Files
threshing-atlas/docs/PRIVACY.md
T
ekcSchianoandClaude Opus 5.5 3b24a11f5a Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-04 02:46:48 +00:00

45 lines
2.6 KiB
Markdown

# Privacy and licences
Rules every part of the project follows. If a change would break one, stop and ask first.
## What may be published
- **Code** is MIT (`LICENSE`). **Data this project produces** is CC BY 4.0 (`DATA_LICENSE.md`). Third-party
sources keep their own terms.
- **Community form responses:** published as exported, with email addresses in free-text answers redacted by
the coding rules (`core/spec/form.yaml`, `core/atlas/coding.py`).
- **Dragon Register:** only `data/public/register.json`. The raw export has rider names and stays local
(`data/raw/register/` is git-ignored).
- **The community route PDF:** never committed or redistributed. Only a fingerprint
(`data/derived/pdf_fingerprint.json`) so we can tell when it changes, and a link to the original.
- **Bot data:** only scene fingerprints, short labels, choice text and outcomes. Never game prose, never account
names. The importer (`collection/etl/transform/bot.py`, rules in `collection/pipeline.yaml`) refuses to write anything that looks like either.
## The bot
- Ken's bot repository stays private. Only the anonymised import comes here.
- Account ids are salted and hashed. The salt (`DK_ACCOUNT_SALT`) lives in the environment and is never
committed.
- The bot's browser profiles, `.env` and credential files are never committed or shared.
## Sources we do not scrape
- **Discord:** not without the server admins' consent; never with a personal account token.
- **Reddit:** only through Reddit's official API with an app key (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET`),
never by page scraping. Only dragonkind.com result links are taken from the threads; usernames and comment text
are never kept, and each result page is read and added by hand (`collection/manual/reddit_results.yaml`).
## Run reports
"Report a run" creates a public GitHub issue. The pipeline keeps only the issue number, the date and the answers
(`data/public/reports.json`); GitHub usernames are dropped. The form asks people not to include account names. A
report closed as "not planned" or labelled `invalid` or `duplicate` is left out of the data.
## Secrets
- `ATLAS_BOT_TOKEN` (repository secret) lets the daily data update open and auto-merge its pull request
(`docs/REPO_SETTINGS.md`).
- `RAW_ARCHIVE_KEY` encrypts the raw store (the current and previous raw file of each source, which include the
register's rider names) in the Actions cache between daily runs. Without it nothing is archived.
- `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` let the daily update read the megathreads through the Reddit API. Tokens go in secrets only: never in a file, issue, chat or commit.