Commit Graph
89 Commits
Author SHA1 Message Date
kathleenandClaude Opus 5.5 ef14ecaaa5 Paths of all riders: an ink for the Reddit comments layer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 03:14:50 +00:00
kathleenandClaude Opus 5.5 6467ce86a7 Paths of all riders: a layer for runs read from Reddit megathread comments (source \"reddit comment\", PR #18)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 03:14:48 +00:00
ekcSchianoandClaude Opus 5.5 a9836c3191 Valley map: show the paths of all riders (per source, plus deaths); the six traced silhouettes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-18 22:13:51 -05:00
kathleen f3bbde24b7 New data from the community sources (#17)
Send the route model to the bot / send (push) Successful in 7s
CI / test (push) Successful in 39s
2026-11-19 02:55:25 +00:00
kathleen e61134ecc8 Merge pull request 'Send a changed route PDF to the bot by pull request' (#16) from send-pdf-to-bot into main
CI / test (push) Successful in 40s
2026-11-19 02:50:30 +00:00
kathleenandClaude Opus 5.5 66181b8489 Send a changed route PDF to the bot by pull request
CI / test (pull_request) Successful in 38s
The daily check keeps the community route PDF only in its private raw store (never published here). When that
copy differs from "Threshing Options.pdf" in the bot's private repo, `python -m collection send-pdf` opens (or
updates) a pull request there with the new PDF and the lines its text gained and lost, so Ken can update the
hand-written routes in src/pdfRoutes.js (the bot does not read the PDF itself).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:49:25 +00:00
kathleen df0e576a0a Merge pull request 'The Atlas now lives in the DragonKind organization' (#15) from move-to-dragonkind into main
CI / test (push) Successful in 39s
2026-11-19 02:46:06 +00:00
kathleenandClaude Opus 5.5 c7978b4538 The Atlas now lives in the DragonKind organization
CI / test (pull_request) Successful in 3m36s
forge.yaml: repo DragonKind/threshing-atlas, and the links generated from it (data/public/links.json,
data/croissant.json); the project notes too. Old kathleen/threshing-atlas links redirect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:42:19 +00:00
kathleen d6894bbd63 Merge pull request 'push-policy: install the requirements' (#14) from fix/push-policy-deps into main
CI / test (push) Successful in 39s
2026-11-19 02:36:40 +00:00
kathleenandClaude Opus 5.5 3206c65d59 push-policy: install the requirements (python -m collection imports pandas)
CI / test (pull_request) Successful in 1m55s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:34:34 +00:00
kathleen 574fa50b2a New data from the community sources (#13)
Send the route model to the bot / send (push) Failing after 29s
CI / test (push) Successful in 41s
2026-11-19 02:31:20 +00:00
kathleen 30aabdf084 Merge pull request 'Bot handoff: pull Ken's bot data daily; send route model updates to the bot by PR' (#12) from bot-handoff into main
CI / test (push) Successful in 38s
2026-11-19 02:21:38 +00:00
kathleenandClaude Opus 5.5 78e069f03f Bot handoff: the daily check pulls Ken's bot repo; route model updates go to the bot by pull request
CI / test (pull_request) Successful in 39s
- update-sources: download DragonKind/DragonKind (read with ATLAS_BOT_TOKEN, or DRAGONKIND_TOKEN if set) and
  import it as the bot source. Needs DK_ACCOUNT_SALT (the same salt as earlier imports, so accounts keep their keys);
  without it the bot source is skipped as before.
- The bot load keeps one file, data/bot/trajectories-bot.jsonl: the bot's repo holds every account's full history,
  so each import is merged by attempt id (attempts already published stay; older per-import files are folded in).
  One copy of each attempt also keeps build-graph.js from counting an attempt twice.
- New workflow push-policy: when policy.json, solver.js or graph.js change on main, open (or update) a pull request
  on DragonKind/DragonKind that replaces vendor/threshing-atlas/ with them. Ken merges it.
- forge open-pr: --repo/--base for another repository, and reuse the open pull request for the same branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:20:35 +00:00
kathleen f1027212b6 New data from the community sources (#11)
CI / test (push) Successful in 39s
2026-11-18 10:28:08 +00:00
kathleen f41bd22439 New data from the community sources (#10)
CI / test (push) Successful in 2m15s
2026-11-18 00:00:18 +00:00
kathleen d17155282a Merge pull request 'croissant.json: the new entry value' (#9) from fix/croissant-entry into main
CI / test (push) Successful in 37s
2026-11-17 23:53:09 +00:00
kathleenandClaude Opus 5.5 d301e71d79 croissant.json: the new entry value (follow-up to #8)
CI / test (pull_request) Successful in 2m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:50:54 +00:00
kathleen 9af9762314 Merge pull request 'Code the sheet's new answers; report failed builds as issues' (#8) from fix/new-sheet-answers-1117 into main
CI / test (push) Failing after 8s
2026-11-17 23:49:04 +00:00
kathleenandClaude Opus 5.5 6fae8a9833 Daily update: no data pull request when the combined runs failed validation
CI / test (pull_request) Failing after 1m22s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:48:54 +00:00
kathleenandClaude Opus 5.5 47da9f1f82 Code the sheet's new answers; a failed build opens an issue instead of crashing the rebuild
The evening source check stopped (correctly) on two answers the coding rules did not know:
- first sighting "2 Greens": a new form option. Coded as Green, with a data flag that keeps the
  original answer (first_sighting_alias in core/spec/form.yaml).
- row label "Unknown" on a bulk-pasted row: added to the entry values in data/dictionary.yaml.

Workflow: when `collection update` reports the build failed, skip the rebuild (it would only fail
again) and always open the "Source check failed" issue, so new answers are reported, not lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:48:47 +00:00
kathleen a8a4b1901d Merge pull request 'Daily update: one branch per run' (#7) from fix/daily-branch-name into main
CI / test (push) Successful in 1m31s
2026-11-17 23:38:32 +00:00
kathleenandClaude Opus 5.5 bd89a8941b Daily update: one branch per run, so a leftover branch never blocks the push
CI / test (pull_request) Successful in 1m24s
Run #24 (2026-11-17) failed at the push: data/source-update-20261117 already
existed from an earlier run that day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:37:14 +00:00
kathleen 3e21fd8992 New data from the community sources (#5)
CI / test (push) Successful in 2m28s
2026-11-17 03:23:52 +00:00
kathleen 7d74b6ab8c Merge pull request 'Site/repairs' (#4) from site/repairs into main
CI / test (push) Successful in 2m3s
Reviewed-on: kathleen/threshing-atlas#4
2026-11-16 18:26:46 +00:00
kathleen 9610765b61 Merge pull request 'Refresh to 8,748 spreadsheet runs; squad answers ignore case; rebuild checks the runs first' (#3) from data/refresh-2026-11-16 into main
CI / test (push) Successful in 2m1s
Reviewed-on: kathleen/threshing-atlas#3
2026-11-16 18:26:34 +00:00
ekcSchianoandClaude Opus 5.5 73a6f5d355 Site repairs: first-attempt Black, shadow evidence, author's statements
CI / test (pull_request) Successful in 2m12s
- analysis/black/evidence.py also writes data/public/black.json (--check covers both)
- Odds: the Black card shows first-attempt odds; new "The Black is not dice" panel from black.json
- "Earlier choices barely change who waits" card replaced by the first-attempt finding
- Journey eyebrow and bonding step no longer call the rolls pure luck
- Footer, lore line and Reddit post: the author's statements (no roll worse than 1 in 20,
  the Black is not luck) replace the "10% chaos" claim
- Report tab: the community Google Form first (no account needed), the Gitea form second;
  register listed as a source
- register.js: snapshot label reads raw.source (was undefined)
- build.js copies black.json, tree.json and experiments.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-16 18:22:21 +00:00
ekcSchianoandClaude Opus 5.5 3b5c655d51 Refresh to 8,748 spreadsheet runs; squad answers ignore case; rebuild checks the runs first
CI / test (pull_request) Successful in 1m59s
- spreadsheet 8,132 -> 8,748 rows (about 600 new; the sheet's keeper fixed 'Sqaudmate' in 72 answers,
  corrected 15 attempt numbers to 'First Attempt' and re-pasted the daily bulk logs); register 288 -> 292
- form.yaml squad_any_case: a typed 'second' is Second (it stopped the build)
- collection rebuild: the runs are built and checked first, so a failed check stops it before the model,
  plan, reports and statistics are rebuilt from stale runs
- route model, plan, Black evidence, screen tree, statistics, graph and metadata rebuilt (9,473 runs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-16 17:13:23 +00:00
kathleen e5bcd13db2 Merge pull request 'One rebuild list for the refresh, the bot cycle and the daily workflows; graph.json rebuilt' (#2) from first-attempt-cycle into main
CI / test (push) Canceled after 0s
Reviewed-on: kathleen/threshing-atlas#2
2026-11-16 16:40:42 +00:00
ekcSchianoandClaude Opus 5.5 93bdfcd4da One rebuild list for the refresh, the bot cycle and the daily workflows; graph.json rebuilt
CI / test (pull_request) Canceled after 0s
- python -m collection rebuild: graph, route model, experiment plan, the Black evidence, the screen
  tree, statistics, build. refresh, cycle and both update-sources workflows use it, so the morning
  update also rebuilds the reports the tests check (it rebuilt only graph, model and stats before)
- graph.json had not been rebuilt since the played-steps fix (no Node on the Mac)
- docs: the branch is pushed; README mentions rebuild

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-16 16:27:20 +00:00
kathleen b3fa6c4c5b Merge pull request 'First attempt cycle' (#1) from first-attempt-cycle into main
CI / test (push) Canceled after 0s
Reviewed-on: kathleen/threshing-atlas#1
2026-11-16 16:18:09 +00:00
ekcSchianoandClaude Fable 5.2 f8e4f575f4 Screen tree, register rows in the master list, and bot runs coded from the picks that were played
CI / test (pull_request) Canceled after 0s
- analysis/tree/build_tree.py -> data/public/tree.json, analysis/reports/TREE.md: every screen the bot has seen,
  every option, where each led (fixed or dice, with ranges), best known picks per colour chosen cautiously, where
  any picks end, and the untested options. solver.js screenPick(): least-tried and Thompson picks from the tree.
- The bot logs a pick it then overrode as a step: 37 runs were coded with choices they never made. Importer and
  graph.js now use the played steps only.
- runs_clean.csv: the Unofficial Dragon Register's 288 bonded dragons are in the master list as outcome-only
  rows, flagged (no analysis uses them).
- Cycle, refresh and the bot kit rebuild and carry the tree; tests check every generated report is current.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
2026-11-16 15:50:12 +00:00
ekcSchianoandClaude Fable 5.2 d01ec46fcc The Black is not dice: developer statements, evidence report, and a plan that looks for the rule
- planner/knowledge.yaml: what the game's developer has said (paraphrased, dated, graded), what the Atlas does
  with each statement, and the open questions with the experiment that tests each.
- analysis/black/evidence.py -> analysis/reports/BLACK.md: counts from the runs. On the same recorded route the
  shadow shows on 21 of 79 first attempts and 1 of 80 later ones; no single route; Blue then Black after the shadow.
- game.yaml: the shadow is a guard (not a roll); what follows it is the Blue, then the Black; on-screen text for
  the squad's direction screen.
- planner/experiments.yaml v2: one exact path on every fresh account (fresh-replicate), then one choice changed
  at a time (fresh-ablate); shadow-again, second-attempt-same-path, veteran-repeat, veteran-variety; what the bot
  records about the account; the sequence to follow once the shadow shows.
- solver.js: assign() takes account facts, account offsets and cycled arms; recommendOption() answers the
  squad screen (stay + direction) in one pick.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
2026-11-16 15:24:03 +00:00
ekcSchianoandClaude Fable 5.2 51842e1aa9 Host: the repository is kathleen/threshing-atlas on git.kenschiano.net
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
2026-11-12 03:35:03 +00:00
ekcSchianoandClaude Fable 5.2 b56bab9165 docs: the bot's side of the automated cycle, and the project context
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
2026-11-12 03:32:54 +00:00
ekcSchianoandClaude Fable 5.2 ab3664f369 Route model: first attempts are their own shadow roll; experiment plan and automated bot cycle
- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
  depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
  so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
  0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
  experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
2026-11-12 03:31:33 +00:00
ekcSchianoandClaude Opus 5.5 ee07ee143e docs: project context for picking the project up in a new conversation
CI / test (push) Canceled after 0s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-12 02:56:00 +00:00
ekcSchianoandClaude Opus 5.5 4837332a7d Refresh to 8,132 spreadsheet runs; rebuild generated files for the Gitea links
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-12 02:46:31 +00:00
ekcSchianoandClaude Opus 5.5 614bfcda7c Host on Gitea: forge.yaml, Gitea workflows, host-neutral links
- forge.yaml: the repository's host (Gitea at git.kenschiano.net, limited visibility);
  the "Report a run" collector, the page's links (data/public/links.json, filled in by
  site/build.js) and the dataset metadata read it
- collection/etl/forge.py: issues, pull requests (merge when checks succeed) and labels
  for Gitea and GitHub; `python -m collection forge ...`
- .gitea/workflows: CI and the daily update (pushes a branch, opens a pull request that
  merges itself, opens an issue when a source breaks); .github/workflows kept for GitHub
- docs: Gitea setup (REPO_SETTINGS), the bot reading atlas files from the repository

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-12 02:46:17 +00:00
ekcSchianoandClaude Opus 5.5 8713ccff1a Refresh to 7,775 spreadsheet runs; route model learns from the bot and corrects for reporting bias
- spreadsheet 5,988 -> 7,775 runs; new route PDF (Black Dragon Gates); register unchanged
- form.yaml: 'Protect / Help Squadmate' -> Protect squadmate
- policy, statistics, graph, metadata rebuilt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-12 01:31:11 +00:00
ekcSchianoandClaude Opus 5.5 8f1f0383fe Game: which dragons each encounter can give; impossible bonds are not counted
A form row recording a Green bonded at the shadow Blue let the "eventually" objective
find a route whose only possible bond was that Green (ratio 1, near-zero chance).
game.yaml now lists the possible bonds per encounter (own colour; the two Greens also
Orange and Brown); anything else is a data-entry error and is not counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:36:49 +00:00
ekcSchianoandClaude Opus 5.5 64fd3af235 Game: the shadow is gated on investigating the scream and helping
- game.yaml: chance events can have `requires` (a gate) and `otherwise`; the shadow
  requires "Investigate the scream" and "Help your squadmate" (the route PDF's Black
  Dragon Gates; spreadsheet: 78 shadows in 4199 gated tree runs, 2 in 646 others)
- model and advisor: a closed gate fixes the roll; runs off the gate are not counted
- form.yaml: "Protect / Help Squadmate" (new answer) -> Protect squadmate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:33:07 +00:00
ekcSchianoandClaude Opus 5.5 2271fe0976 Route model: correct for reporting bias
- core/atlas/bias.py: for each rare roll (shadow, meadow Blue) and each source group,
  the factor by which the group over-reports the rare outcome relative to complete
  logging (bot, bulk logs), with a 90% range; rare outcomes from biased groups count
  1/rho; a roll the complete logs have (almost) never shown is learned from them only
- planner/policy.yaml: groups, rolls and thresholds; estimates published in
  policy.json under model.reportingBias
- fit_alpha: a roll seen only one way is pooled fully
- docs: goal accounts follow the best route, research accounts use Thompson sampling
  (decisionMode atlas_thompson)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:29:09 +00:00
ekcSchianoandClaude Opus 5.5 c0355616b6 Route model learns from bot runs; Thompson sampling in the advisor (work in progress)
- planner/policy.yaml: which sources the route model learns from (spreadsheet,
  reported, bot) and the low-bias sample
- bot runs: deaths coded by stage, Malek intent for runs with a target
- model: survival of the first dragon is not counted when nothing after it is
  known; the meadow dragon falls back to what the bot saw
- solver.js: advisor(..., {sampling: "thompson", seed, draws}) follows one
  posterior draw per attempt and reports each pick's propensity

Not shipped yet: mixing sources lowers the fitted shrinkage for the shadow roll
(alpha 500 -> 50 in the low-bias sample), see the conversation of 2026-11-09.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:29:09 +00:00
ekcSchianoandClaude Opus 5.5 d37eaa3378 Import Ken's bot runs (13 accounts, 415 attempts, to 2026-11-10)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:09:26 +00:00
ekcSchianoandClaude Opus 5.5 43066ad67f Bot import: one folder per account, run records with steps
- the importer reads data/accounts/*/runs.jsonl and logs/accounts/*/encounters.jsonl;
  with account folders, the top-level files (a mirror of the first account) are skipped
- run records that carry their steps (schema 2) become trajectories directly, keeping
  what the bot saw (shadow, meadow dragons), propensities, decision mode and policy id;
  older runs are still rebuilt from the decision log; bonded runs without steps are kept
- account names become salted hashes (DK_ACCOUNT_SALT, or a private salt in data/raw)
- runs_clean.csv: new bot_mode column; saw_shadow and saw_blue_meadow from what the bot
  saw; deliberate burns say why
- crosswalk: the valley Red, leaving squadmates, squad direction, hold your ground,
  hide/approach reactions, the wounded cadet's other two choices

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 02:08:57 +00:00
ekcSchianoandClaude Opus 5.5 dd25dfaa32 Record the local bot data as imported (same 6 runs as the sample)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 01:52:46 +00:00
ekcSchianoandClaude Opus 5.5 5c49591dae Hand-offs without GitHub: bot kit zip, bot zip import, local refresh
- python -m collection export-bot-kit: policy.json, solver.js, graph.js and a
  README with checksums, zipped for the bot
- the bot import takes a folder or a zip (files may sit under any top folder)
- python -m collection refresh: the daily workflow, run locally
- trajectories files are named by date and raw checksum, so two imports on
  one day never overwrite each other

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-10 01:50:12 +00:00
ekcSchianoandClaude Opus 5.5 19209a2ad6 Report a run: link the Threshing Routes Google Form
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-04 02:57:19 +00:00
ekcSchianoandGitHub 9f1bb3d3b5 Merge pull request #15 from ekcSchiano/collection/etl
Data collection as an ETL pipeline; every source in runs_clean.csv; Report a run
2026-11-03 21:47:59 -05:00
ekcSchianoandClaude Opus 5.5 3b24a11f5a Collection as an ETL pipeline; runs_clean.csv from every source; Report a run
- collection/pipeline.yaml declares every source (extract kind, transform,
  privacy rules, load target, run source); `python -m collection update|build`
- extract keeps data/raw/<source>/current + previous only (git-ignored);
  data/manifest.json records the current raw checksum and change date
- transforms are pure; validate checks runs against data/dictionary.yaml
  before load; load writes byte-stable files only when they change
- data/public/runs_clean.csv: every run from every source with a new
  `source` column (reddit spreadsheet, reddit thread, reported, bot,
  reddit pdf); the sheet's own row label moves to `entry`; run_id is text
- bot importer ported to Python (same output as bot.js on the fixture);
  labels and crosswalk in collection/specs/bot.yaml, written into graph.js
- Reddit through the official API (result links to add by hand); the 12
  result pages move to collection/manual/reddit_results.yaml
- "Report a run": a GitHub issue form generated from
  collection/specs/report_form.yaml, collected as source "reported", plus a
  site tab pointing to it, the community spreadsheet and the megathread
- analysis and the route model read runs_clean.csv (sources in
  measures.yaml); stats.json and policy.json unchanged apart from the
  factor's new name
- daily workflow runs the pipeline; the raw store is kept between runs in
  the Actions cache, encrypted with RAW_ARCHIVE_KEY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-04 02:46:48 +00:00