14 Commits
Author SHA1 Message Date
kathleen 30aabdf084 Merge pull request 'Bot handoff: pull Ken's bot data daily; send route model updates to the bot by PR' (#12) from bot-handoff into main
CI / test (push) Successful in 38s
2026-11-19 02:21:38 +00:00
kathleenandClaude Opus 5.5 78e069f03f Bot handoff: the daily check pulls Ken's bot repo; route model updates go to the bot by pull request
CI / test (pull_request) Successful in 39s
- update-sources: download DragonKind/DragonKind (read with ATLAS_BOT_TOKEN, or DRAGONKIND_TOKEN if set) and
  import it as the bot source. Needs DK_ACCOUNT_SALT (the same salt as earlier imports, so accounts keep their keys);
  without it the bot source is skipped as before.
- The bot load keeps one file, data/bot/trajectories-bot.jsonl: the bot's repo holds every account's full history,
  so each import is merged by attempt id (attempts already published stay; older per-import files are folded in).
  One copy of each attempt also keeps build-graph.js from counting an attempt twice.
- New workflow push-policy: when policy.json, solver.js or graph.js change on main, open (or update) a pull request
  on DragonKind/DragonKind that replaces vendor/threshing-atlas/ with them. Ken merges it.
- forge open-pr: --repo/--base for another repository, and reuse the open pull request for the same branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-19 02:20:35 +00:00
kathleen f1027212b6 New data from the community sources (#11)
CI / test (push) Successful in 39s
2026-11-18 10:28:08 +00:00
kathleen f41bd22439 New data from the community sources (#10)
CI / test (push) Successful in 2m15s
2026-11-18 00:00:18 +00:00
kathleen d17155282a Merge pull request 'croissant.json: the new entry value' (#9) from fix/croissant-entry into main
CI / test (push) Successful in 37s
2026-11-17 23:53:09 +00:00
kathleenandClaude Opus 5.5 d301e71d79 croissant.json: the new entry value (follow-up to #8)
CI / test (pull_request) Successful in 2m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:50:54 +00:00
kathleen 9af9762314 Merge pull request 'Code the sheet's new answers; report failed builds as issues' (#8) from fix/new-sheet-answers-1117 into main
CI / test (push) Failing after 8s
2026-11-17 23:49:04 +00:00
kathleenandClaude Opus 5.5 6fae8a9833 Daily update: no data pull request when the combined runs failed validation
CI / test (pull_request) Failing after 1m22s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:48:54 +00:00
kathleenandClaude Opus 5.5 47da9f1f82 Code the sheet's new answers; a failed build opens an issue instead of crashing the rebuild
The evening source check stopped (correctly) on two answers the coding rules did not know:
- first sighting "2 Greens": a new form option. Coded as Green, with a data flag that keeps the
  original answer (first_sighting_alias in core/spec/form.yaml).
- row label "Unknown" on a bulk-pasted row: added to the entry values in data/dictionary.yaml.

Workflow: when `collection update` reports the build failed, skip the rebuild (it would only fail
again) and always open the "Source check failed" issue, so new answers are reported, not lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:48:47 +00:00
kathleen a8a4b1901d Merge pull request 'Daily update: one branch per run' (#7) from fix/daily-branch-name into main
CI / test (push) Successful in 1m31s
2026-11-17 23:38:32 +00:00
kathleenandClaude Opus 5.5 bd89a8941b Daily update: one branch per run, so a leftover branch never blocks the push
CI / test (pull_request) Successful in 1m24s
Run #24 (2026-11-17) failed at the push: data/source-update-20261117 already
existed from an earlier run that day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
2026-11-17 23:37:14 +00:00
kathleen 3e21fd8992 New data from the community sources (#5)
CI / test (push) Successful in 2m28s
2026-11-17 03:23:52 +00:00
kathleen 7d74b6ab8c Merge pull request 'Site/repairs' (#4) from site/repairs into main
CI / test (push) Successful in 2m3s
Reviewed-on: kathleen/threshing-atlas#4
2026-11-16 18:26:46 +00:00
kathleen 9610765b61 Merge pull request 'Refresh to 8,748 spreadsheet runs; squad answers ignore case; rebuild checks the runs first' (#3) from data/refresh-2026-11-16 into main
CI / test (push) Successful in 2m1s
Reviewed-on: kathleen/threshing-atlas#3
2026-11-16 18:26:34 +00:00
18 changed files with 6108 additions and 5536 deletions
+60
View File
@@ -0,0 +1,60 @@
# When the route model changes on main, send it to the bot: open (or update) a pull request on the bot's repo,
# DragonKind/DragonKind, that replaces vendor/threshing-atlas/{policy.json,solver.js,graph.js} with this repo's
# data/public/policy.json, core/solver.js and core/graph.js. The three always travel together, so the policy and the
# solver that reads it never get out of step. Ken reviews and merges it; the running bot reloads the files after a pull.
#
# Secrets: ATLAS_BOT_TOKEN (repository: read and write, with write access to DragonKind/DragonKind).
name: Send the route model to the bot
on:
push:
branches: [main]
paths:
- data/public/policy.json
- core/solver.js
- core/graph.js
workflow_dispatch:
jobs:
send:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.ATLAS_BOT_TOKEN }}
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install pyyaml
- name: Open or update the pull request on the bot's repo
env:
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
run: |
set -e
dst="$RUNNER_TEMP/dragonkind"
export GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0="http.https://git.kenschiano.net/.extraheader" \
GIT_CONFIG_VALUE_0="AUTHORIZATION: token $ATLAS_BOT_TOKEN"
git clone --quiet --depth 1 https://git.kenschiano.net/DragonKind/DragonKind.git "$dst"
mkdir -p "$dst/vendor/threshing-atlas"
cp data/public/policy.json core/solver.js core/graph.js "$dst/vendor/threshing-atlas/"
id=$(python3 -c "import json; print(json.load(open('data/public/policy.json')).get('id', ''))")
sha=$(git rev-parse --short HEAD)
cd "$dst"
if [ -z "$(git status --porcelain -- vendor/threshing-atlas)" ]; then
echo "The bot already has route model $id."; exit 0
fi
git config user.name "Atlas data bot"
git config user.email "atlas-bot@noreply.localhost"
git checkout -q -b atlas/route-model
git add vendor/threshing-atlas
git commit -q -m "Threshing Atlas route model $id (atlas $sha)"
git push -q --force origin atlas/route-model
cd "$GITHUB_WORKSPACE"
printf '%s\n' \
"The Threshing Atlas route model changed (policy \`$id\`, atlas commit \`$sha\`)." \
"" \
"This replaces \`vendor/threshing-atlas/\` (policy.json, solver.js, graph.js) together, as \`npm run atlas:sync\` would." \
"This pull request is updated in place each time the model changes. After merging, pull on the bot's machine:" \
"the running bot reloads the files on its next decision." \
"" \
"Opened automatically by the Atlas's push-policy workflow." > "$RUNNER_TEMP/body.md"
python -m collection forge open-pr --repo DragonKind/DragonKind --base main --head atlas/route-model \
--title "Threshing Atlas route model update" --body-file "$RUNNER_TEMP/body.md"
+23 -4
View File
@@ -8,6 +8,9 @@
# update branch and open the pull request; the built-in token cannot start the CI it needs. # update branch and open the pull request; the built-in token cannot start the CI it needs.
# RAW_ARCHIVE_KEY optional: encrypts the private raw store kept in the Actions cache between runs # RAW_ARCHIVE_KEY optional: encrypts the private raw store kept in the Actions cache between runs
# REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET optional: read the megathreads through the Reddit API # REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET optional: read the megathreads through the Reddit API
# DK_ACCOUNT_SALT the salt that turns bot account names into anonymous keys (keep it the same forever). With it,
# the bot's private repo (DragonKind/DragonKind) is downloaded and imported; without it, skipped.
# DRAGONKIND_TOKEN optional: a read-only token for the bot's repo (ATLAS_BOT_TOKEN is used otherwise)
name: Check sources for new data name: Check sources for new data
on: on:
schedule: schedule:
@@ -41,14 +44,29 @@ jobs:
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data
fi fi
- name: Download the bot's data (DragonKind/DragonKind, private)
env:
DK_TOKEN: ${{ secrets.DRAGONKIND_TOKEN || secrets.ATLAS_BOT_TOKEN }}
DK_ACCOUNT_SALT: ${{ secrets.DK_ACCOUNT_SALT }}
run: |
if [ -n "$DK_TOKEN" ] && [ -n "$DK_ACCOUNT_SALT" ]; then
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0="http.https://git.kenschiano.net/.extraheader" \
GIT_CONFIG_VALUE_0="AUTHORIZATION: token $DK_TOKEN" \
git clone --quiet --depth 1 https://git.kenschiano.net/DragonKind/DragonKind.git "$RUNNER_TEMP/dragonkind"
echo "DRAGONKIND_BOT_DIR=$RUNNER_TEMP/dragonkind" >> "$GITHUB_ENV"
else
echo "DK_ACCOUNT_SALT is not set: the bot source is skipped"
fi
- id: check - id: check
env: env:
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }} ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
DK_ACCOUNT_SALT: ${{ secrets.DK_ACCOUNT_SALT }}
REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }} REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }} REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
run: python -m collection update --report "$RUNNER_TEMP/report.md" run: python -m collection update --report "$RUNNER_TEMP/report.md"
- name: Rebuild the analysis and run tests - name: Rebuild the analysis and run tests
if: steps.check.outputs.changed == 'true' # not when the combined runs failed validation: the issue below reports the new answers instead
if: steps.check.outputs.changed == 'true' && !contains(steps.check.outputs.failed, 'build')
run: | run: |
python -m collection rebuild python -m collection rebuild
npm test npm test
@@ -69,11 +87,12 @@ jobs:
path: ${{ runner.temp }}/raw-archive path: ${{ runner.temp }}/raw-archive
key: raw-archive-${{ hashFiles('data/manifest.json') }} key: raw-archive-${{ hashFiles('data/manifest.json') }}
- name: Open a pull request with the new data (it merges itself once CI passes) - name: Open a pull request with the new data (it merges itself once CI passes)
if: steps.check.outputs.changed == 'true' if: steps.check.outputs.changed == 'true' && !contains(steps.check.outputs.failed, 'build')
env: env:
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }} ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
run: | run: |
branch="data/source-update-$(date -u +%Y%m%d)" # one branch per run: a second run on the same day (or a leftover branch) never blocks the push
branch="data/source-update-$(date -u +%Y%m%d)-run${{ github.run_number }}"
git config user.name "Atlas data bot" git config user.name "Atlas data bot"
git config user.email "atlas-bot@noreply.localhost" git config user.email "atlas-bot@noreply.localhost"
git checkout -b "$branch" git checkout -b "$branch"
@@ -82,7 +101,7 @@ jobs:
git push origin "$branch" git push origin "$branch"
python -m collection forge open-pr --head "$branch" --title "New data from the community sources" --body-file "$RUNNER_TEMP/report.md" --automerge python -m collection forge open-pr --head "$branch" --title "New data from the community sources" --body-file "$RUNNER_TEMP/report.md" --automerge
- name: Report a broken source or new answers - name: Report a broken source or new answers
if: steps.check.outputs.failed != '' if: always() && steps.check.outputs.failed != ''
env: env:
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }} ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
run: | run: |
+31 -31
View File
@@ -6,15 +6,15 @@ Written by `analysis/black/evidence.py` from `data/public/runs_clean.csv`. Count
| Attempt | Form submissions | Bulk logs | Bot | | Attempt | Form submissions | Bulk logs | Bot |
|---|---|---|---| |---|---|---|---|
| 1 | 89 / 879 (10.1%) | 0 / 28 (0.0%) | 0 / 0 | | 1 | 92 / 920 (10.0%) | 0 / 29 (0.0%) | 0 / 0 |
| 2 | 0 / 288 (0.0%) | 0 / 21 (0.0%) | 0 / 4 (0.0%) | | 2 | 0 / 296 (0.0%) | 0 / 21 (0.0%) | 0 / 4 (0.0%) |
| 3 | 0 / 184 (0.0%) | 0 / 14 (0.0%) | 0 / 5 (0.0%) | | 3 | 0 / 192 (0.0%) | 0 / 14 (0.0%) | 0 / 5 (0.0%) |
| 4-10 | 0 / 506 (0.0%) | 0 / 102 (0.0%) | 0 / 36 (0.0%) | | 4-10 | 0 / 536 (0.0%) | 0 / 102 (0.0%) | 0 / 36 (0.0%) |
| 11-30 | 0 / 573 (0.0%) | 0 / 57 (0.0%) | 0 / 137 (0.0%) | | 11-30 | 0 / 614 (0.0%) | 0 / 57 (0.0%) | 0 / 137 (0.0%) |
| 31 or more | 1 / 610 (0.2%) | 0 / 0 | 0 / 54 (0.0%) | | 31 or more | 1 / 679 (0.1%) | 0 / 0 | 0 / 54 (0.0%) |
| not given | 3 / 1509 (0.2%) | 0 / 112 (0.0%) | 0 / 0 | | not given | 3 / 1576 (0.2%) | 0 / 149 (0.0%) | 0 / 0 |
Complete logs (bot and bulk) have 28 first attempts at the tree and 430 later ones, with 0 shadows. Zero in 430 later attempts rules out a rate above 0.7% for them; zero in 28 first attempts only rules out more than 10%. Complete logs (bot and bulk) have 29 first attempts at the tree and 430 later ones, with 0 shadows. Zero in 430 later attempts rules out a rate above 0.7% for them; zero in 29 first attempts only rules out more than 10%.
## 2. The same route on a first attempt and on a later one ## 2. The same route on a first attempt and on a later one
@@ -22,39 +22,39 @@ If the shadow depended only on the choices the form records (company, first dire
| Route | First attempt | Later attempts | | Route | First attempt | Later attempts |
|---|---|---| |---|---|---|
| Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West | 22 / 88 (25.0%) | 0 / 89 (0.0%) | | Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West | 23 / 93 (24.7%) | 0 / 94 (0.0%) |
| Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / Forest | 5 / 29 (17.2%) | 0 / 93 (0.0%) | | Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / Forest | 5 / 29 (17.2%) | 0 / 95 (0.0%) |
| Alone / Center / Assess the situation / Engage the smaller cadet / Squadmate's revenge / Forest | 4 / 28 (14.3%) | 0 / 76 (0.0%) | | Alone / Center / Assess the situation / Engage the smaller cadet / Squadmate's revenge / Forest | 4 / 28 (14.3%) | 0 / 78 (0.0%) |
| 1 Squadmate / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 2 / 31 (6.5%) | 0 / 64 (0.0%) | | 1 Squadmate / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 2 / 32 (6.2%) | 0 / 67 (0.0%) |
| Alone / Center / Assess the situation / Daggers or not (protect squadmate) / Left bullies to the dragons / Forest | 1 / 28 (3.6%) | 0 / 63 (0.0%) | | Alone / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 1 / 34 (2.9%) | 0 / 60 (0.0%) |
| 2 Squadmates / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 2 / 45 (4.4%) | 0 / 48 (0.0%) |
| Alone / Center / Assess the situation / Daggers or not (protect squadmate) / Left bullies to the dragons / Forest | 1 / 28 (3.6%) | 0 / 64 (0.0%) |
| Alone / Center / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West | 1 / 8 (12.5%) | 1 / 82 (1.2%) | | Alone / Center / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West | 1 / 8 (12.5%) | 1 / 82 (1.2%) |
| 2 Squadmates / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 2 / 43 (4.7%) | 0 / 45 (0.0%) |
| Alone / East / Assess the situation / Engage the smaller cadet / Squadmate's revenge / East | 1 / 33 (3.0%) | 0 / 52 (0.0%) |
Across all 321 recorded routes, comparing first and later attempts on the same route: 85 first-attempt shadows where 36.4 would be expected if the attempt did not matter (odds ratio 626.3, z = 11.9). First and later attempts choose almost the same routes, so the difference is not in the recorded choices: it is something about the account, or a choice the form does not record. Across all 323 recorded routes, comparing first and later attempts on the same route: 88 first-attempt shadows where 37.2 would be expected if the attempt did not matter (odds ratio 653.8, z = 12.1). First and later attempts choose almost the same routes, so the difference is not in the recorded choices: it is something about the account, or a choice the form does not record.
## 3. On one route, is the shadow decided by the cadet's wing, section and squad? ## 3. On one route, is the shadow decided by the cadet's wing, section and squad?
The developer says the Black is decided by choices and by what the site knows about the account, with no dice. If the recorded route and the cadet assignment were all of it, every first attempt with the same route and assignment would agree. On the best-recorded route (Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West; 22 / 88 (25.0%) on first attempts), 22 assignments have two or more runs and 10 of them disagree (some runs with the shadow, some without). So something else decides it: a choice the form does not record, or a fact about the account other than its assignment. No wing, section or squad stands out: The developer says the Black is decided by choices and by what the site knows about the account, with no dice. If the recorded route and the cadet assignment were all of it, every first attempt with the same route and assignment would agree. On the best-recorded route (Alone / West / Assess the situation / Engage the smaller cadet / Squadmate's revenge / West; 23 / 93 (24.7%) on first attempts), 26 assignments have two or more runs and 12 of them disagree (some runs with the shadow, some without). So something else decides it: a choice the form does not record, or a fact about the account other than its assignment. No wing, section or squad stands out:
| | shadow / runs | | | shadow / runs |
|---|---| |---|---|
| wing | Fourth: 5/27, Third: 7/26, First: 6/20, Second: 3/12 | | wing | Fourth: 5/28, Third: 7/27, First: 6/21, Second: 4/14 |
| section | Claw: 9/30, Flame: 5/29, Tail: 7/26, (blank): 1/3 | | section | Claw: 9/31, Flame: 5/30, Tail: 8/29, (blank): 1/3 |
| squad | Second: 11/34, First: 6/27, Third: 4/25 | | squad | Second: 12/35, First: 6/29, Third: 4/27 |
| first_sighting | Brown: 11/45, Green: 9/41 | | first_sighting | Brown: 12/49, Green: 9/42 |
| sighting_action | Hold still: 10/49, Respect / face it: 4/18, Unknown: 8/14, Weapon: 0/5 | | sighting_action | Hold still: 10/50, Respect / face it: 5/22, Unknown: 8/14, Weapon: 0/5 |
## 4. There is no single route ## 4. There is no single route
The 93 form runs with a shadow (all attempts) took 46 different recorded routes: The 96 form runs with a shadow (all attempts) took 46 different recorded routes:
- **Company:** Alone 70, 2 Squadmates 12, 1 Squadmate 11 - **Company:** Alone 71, 1 Squadmate 13, 2 Squadmates 12
- **First direction:** West 58, Center 20, East 15 - **First direction:** West 60, Center 20, East 16
- **First move in the fight:** Assess the situation 82, (not recorded) 9, Even the numbers (smaller cadet) 2 - **First move in the fight:** Assess the situation 85, (not recorded) 9, Even the numbers (smaller cadet) 2
- **The dagger:** Engage the smaller cadet 60, (not recorded) 22, Daggers or not (protect squadmate) 11 - **The dagger:** Engage the smaller cadet 63, (not recorded) 22, Daggers or not (protect squadmate) 11
- **The bullies' fate:** Squadmate's revenge 58, Left bullies to the dragons 23, Killed the bullies yourself 9, Unknown 3 - **The bullies' fate:** Squadmate's revenge 60, Left bullies to the dragons 25, Killed the bullies yourself 9, Unknown 2
- **The fork:** West 44, Forest 36, East 12, Unknown 1 - **The fork:** West 46, Forest 36, East 13, Unknown 1
Shadows reported without both gates: 3 (likely mis-entered). Shadows reported without both gates: 3 (likely mis-entered).
@@ -62,9 +62,9 @@ Shadows reported without both gates: 3 (likely mis-entered).
The riders' notes describe one sequence: follow the shadow, a Blue lands and burns another cadet, the Blue leaves, then the Black appears. The Blue is a trial on the way, not an alternative to the Black. The riders' notes describe one sequence: follow the shadow, a Blue lands and burns another cadet, the Blue leaves, then the Black appears. The Blue is a trial on the way, not an alternative to the Black.
- Runs that reached the Black: 76, bonded 55. - Runs that reached the Black: 79, bonded 57.
- Runs that died at the Blue: 15. What their notes say they did: approached the Blue 10, warned the other cadet / intervened 4. - Runs that died at the Blue: 15. What their notes say they did: approached the Blue 10, warned the other cadet / intervened 4.
- The answer given to the Black, where the note says (bonded / runs): held its gaze 30/42; lowered eyes 0/7; ignored the sound 0/1. - The answer given to the Black, where the note says (bonded / runs): held its gaze 29/41; lowered eyes 0/7; ignored the sound 0/1.
Holding its gaze is the only answer recorded with a bond, and it still fails about a quarter of the time. The developer says a rider who knows the system bonds with certainty, so those failures are not explained yet: the notes of the riders who died are too short to show what else they did differently. Holding its gaze is the only answer recorded with a bond, and it still fails about a quarter of the time. The developer says a rider who knows the system bonds with certainty, so those failures are not explained yet: the notes of the riders who died are too short to show what else they did differently.
+16 -5
View File
@@ -1,6 +1,6 @@
"""The repository host (forge.yaml): links, and the few API calls the pipeline makes, for Gitea and GitHub. """The repository host (forge.yaml): links, and the few API calls the pipeline makes, for Gitea and GitHub.
python -m collection forge open-pr --head BRANCH --title T --body-file F [--automerge] python -m collection forge open-pr --head BRANCH --title T --body-file F [--automerge] [--repo OWNER/NAME --base B]
python -m collection forge open-issue --title T --body-file F (skipped if an open issue has that title) python -m collection forge open-issue --title T --body-file F (skipped if an open issue has that title)
python -m collection forge ensure-label --name N --color HEX python -m collection forge ensure-label --name N --color HEX
@@ -96,10 +96,19 @@ def list_issues(label, env=None, f=None):
return sorted(out, key=lambda i: i["number"]) return sorted(out, key=lambda i: i["number"])
def open_pr(head, title, body, automerge=False, env=None, f=None): def open_pr(head, title, body, automerge=False, env=None, f=None, repo=None, base=None):
f = f or config() """Open a pull request (on this repository, or `repo` on the same host). If one is open for the branch already,
it is reused: pushing to its branch has updated it."""
f = dict(f or config())
if repo:
f["repo"] = repo
base = base or f["branch"]
tok = token(env) tok = token(env)
pr = _call("POST", f"{api_base(f)}/pulls", {"head": head, "base": f["branch"], "title": title, "body": body}, tok, f) open_prs = _call("GET", f"{api_base(f)}/pulls?state=open&limit=50&per_page=100", tok=tok, f=f) or []
same = [p for p in open_prs if (p.get("head") or {}).get("ref") == head]
if same:
return same[0]["number"]
pr = _call("POST", f"{api_base(f)}/pulls", {"head": head, "base": base, "title": title, "body": body}, tok, f)
n = pr["number"] n = pr["number"]
if automerge: if automerge:
if f["kind"] == "github": if f["kind"] == "github":
@@ -141,6 +150,8 @@ def main(argv):
p.add_argument("--title", required=True) p.add_argument("--title", required=True)
p.add_argument("--body-file", required=True) p.add_argument("--body-file", required=True)
p.add_argument("--automerge", action="store_true") p.add_argument("--automerge", action="store_true")
p.add_argument("--repo", help="another repository on the same host (OWNER/NAME)")
p.add_argument("--base", help="the branch to merge into (default: forge.yaml branch)")
i = sub.add_parser("open-issue") i = sub.add_parser("open-issue")
i.add_argument("--title", required=True) i.add_argument("--title", required=True)
i.add_argument("--body-file", required=True) i.add_argument("--body-file", required=True)
@@ -150,7 +161,7 @@ def main(argv):
a = ap.parse_args(argv) a = ap.parse_args(argv)
if a.cmd == "open-pr": if a.cmd == "open-pr":
body = open(a.body_file, encoding="utf-8").read() body = open(a.body_file, encoding="utf-8").read()
print(f"pull request #{open_pr(a.head, a.title, body, a.automerge)}") print(f"pull request #{open_pr(a.head, a.title, body, a.automerge, repo=a.repo, base=a.base)}")
elif a.cmd == "open-issue": elif a.cmd == "open-issue":
n = open_issue(a.title, open(a.body_file, encoding="utf-8").read()) n = open_issue(a.title, open(a.body_file, encoding="utf-8").read())
print(f"issue #{n}" if n else "an open issue with that title exists already") print(f"issue #{n}" if n else "an open issue with that title exists already")
+1 -1
View File
@@ -95,5 +95,5 @@ sources:
# data/raw/bot/account-salt.txt); never committed # data/raw/bot/account-salt.txt); never committed
refuse_fields: [prose, proseNorm, proseExcerpt, email, password, accountId, accountName] refuse_fields: [prose, proseNorm, proseExcerpt, email, password, accountId, accountName]
max_string: 200 # anything longer looks like game prose max_string: 200 # anything longer looks like game prose
load: data/bot # scenes.json (merged) + trajectories-<date>.jsonl load: data/bot # scenes.json and trajectories-bot.jsonl, both merged with what is published
runs: bot runs: bot
+25 -4
View File
@@ -28,10 +28,33 @@ from datetime import datetime, timezone
from .etl import extract as E from .etl import extract as E
from .etl import load as L from .etl import load as L
from .etl import codegen, metadata, report_form, runs, validate from .etl import codegen, metadata, report_form, runs, validate
from .etl.common import Report, path, pipeline from .etl.common import Report, path, pipeline, read_jsonl
from .etl.transform import TRANSFORMS from .etl.transform import TRANSFORMS
BOT_FILE = "trajectories-bot.jsonl"
def merge_trajectories(target, new):
"""Every bot attempt ever imported, in one file (data/bot/trajectories-bot.jsonl), sorted by id. The bot's repo
holds each account's full history, so an import is a superset: a re-imported attempt replaces its old copy, and
attempts no longer in the bot's repo (an archived account) stay. Older per-import files are folded in."""
d = path(target)
olds = sorted(d.glob("trajectories-*.jsonl"))
by_id = {}
for f in olds:
for t in read_jsonl(f):
by_id.setdefault(t["id"], t)
for t in new:
by_id[t["id"]] = t
wrote = L.jsonl_file(f"{target}/{BOT_FILE}", [by_id[k] for k in sorted(by_id)])
for f in olds:
if f.name != BOT_FILE:
f.unlink()
wrote = True
return wrote
def update_source(name, spec, manifest, report, local_file=None, env=None): def update_source(name, spec, manifest, report, local_file=None, env=None):
env = os.environ if env is None else env env = os.environ if env is None else env
try: try:
@@ -54,9 +77,7 @@ def update_source(name, spec, manifest, report, local_file=None, env=None):
old = {"scenes": L.read_json(f"{target}/scenes.json") or {}} old = {"scenes": L.read_json(f"{target}/scenes.json") or {}}
lines = mod.describe(old, new) lines = mod.describe(old, new)
wrote = L.json_file(f"{target}/scenes.json", mod.merge_scenes(old["scenes"], new["scenes"]), compact=False) wrote = L.json_file(f"{target}/scenes.json", mod.merge_scenes(old["scenes"], new["scenes"]), compact=False)
# one file per import (date + raw checksum), so two imports on the same day (yours, Ken's) never overwrite wrote |= merge_trajectories(target, new["trajectories"])
tag = f"{ctx['changed']}-{manifest[name]['sha256'][:8]}"
wrote |= L.jsonl_file(f"{target}/trajectories-{tag}.jsonl", new["trajectories"])
elif target: elif target:
old = L.read_json(target) old = L.read_json(target)
lines = mod.describe(old, new) lines = mod.describe(old, new)
+4
View File
@@ -150,6 +150,10 @@ def code_row(r, C):
company = company_raw[: -len(suffix)] if left_squad else company_raw company = company_raw[: -len(suffix)] if left_squad else company_raw
if not left_squad and company in ("1 Squadmate", "2 Squadmates") and not (date and date >= F["left_squadmates_asked_from"]): if not left_squad and company in ("1 Squadmate", "2 Squadmates") and not (date and date >= F["left_squadmates_asked_from"]):
left_squad = None # the form did not offer "left them" yet: unknown, not "stayed together" left_squad = None # the form did not offer "left them" yet: unknown, not "stayed together"
alias = F.get("first_sighting_alias") or {}
if first in alias:
flags.append(f"first sighting answered as {first!r}")
first = alias[first]
pools = F["sighting_pool"] pools = F["sighting_pool"]
pool = list(pools.get(company, {}).get(direction, [])) + (list(pools["Alone"].get(direction, [])) if left_squad else []) pool = list(pools.get(company, {}).get(direction, [])) + (list(pools["Alone"].get(direction, [])) if left_squad else [])
if first and first != "Unsure" and pool and first not in pool: if first and first != "Unsure" and pool and first not in pool:
+5
View File
@@ -153,5 +153,10 @@ sighting_pool:
1 Squadmate: { East: [Green, Orange, Red], West: [Red, Brown, Green], Center: [Red, Brown] } 1 Squadmate: { East: [Green, Orange, Red], West: [Red, Brown, Green], Center: [Red, Brown] }
2 Squadmates: { East: [Red, Orange], West: [Green, Brown, Red], Center: [Red, Brown] } 2 Squadmates: { East: [Red, Orange], West: [Green, Brown, Red], Center: [Red, Brown] }
# First-sighting answers that are not a single colour: coded as this colour, and the row is flagged with the
# original answer (data_flags), so nothing is lost.
first_sighting_alias:
"2 Greens": Green
# How a bulk-pasted row's attempt answer is read # How a bulk-pasted row's attempt answer is read
attempt_words: { first: 1, second: 2, third: 3, fourth: 4, fifth: 5 } attempt_words: { first: 1, second: 2, third: 3, fourth: 4, fifth: 5 }
+13 -13
View File
@@ -88,8 +88,8 @@
"description": "Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's rules are core/spec/form.yaml). The `source` column says where each run came from.", "description": "Every run from every source, one per row, coded into the same columns (collection/etl/runs.py; the form's rules are core/spec/form.yaml). The `source` column says where each run came from.",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/runs_clean.csv", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/runs_clean.csv",
"encodingFormat": "text/csv", "encodingFormat": "text/csv",
"sha256": "efe9d8395e0fdd15d936a22915a9f9ee13e0ebf32f685141562bdc5681b6bf5e", "sha256": "ad595952e25f42ddd07d9ead5d12705b6f73330f2ab8f0f949d82140af93e3b2",
"contentSize": "3324468 B" "contentSize": "3475159 B"
}, },
{ {
"@type": "cr:FileObject", "@type": "cr:FileObject",
@@ -98,8 +98,8 @@
"description": "The community \"Threshing Routes\" form export as published (headers and rows), with email addresses in free-text answers redacted. The source of runs_clean.", "description": "The community \"Threshing Routes\" form export as published (headers and rows), with email addresses in free-text answers redacted. The source of runs_clean.",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/form_responses.json", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/form_responses.json",
"encodingFormat": "application/json", "encodingFormat": "application/json",
"sha256": "742635ebb21616020025063d2e6587e8ad06564a2f1807b94a99f38518479008", "sha256": "78122dc3e25ee9267e935722660d7ad7358d0602fdf5cd13f4c33a79a2194c02",
"contentSize": "11878421 B" "contentSize": "12453893 B"
}, },
{ {
"@type": "cr:FileObject", "@type": "cr:FileObject",
@@ -118,8 +118,8 @@
"description": "The Unofficial Dragon Register (which dragon each rider bonded), without rider names.", "description": "The Unofficial Dragon Register (which dragon each rider bonded), without rider names.",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/register.json", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/register.json",
"encodingFormat": "application/json", "encodingFormat": "application/json",
"sha256": "24ed1b9dfcae322df1d123d169557cc22a933426b48f7a4e5b0a0ba0d412b7a5", "sha256": "32d546c9e9ffb1536b21806e174c98bf60a078478926275404e23a9c90b8bb0f",
"contentSize": "117781 B" "contentSize": "118171 B"
}, },
{ {
"@type": "cr:FileObject", "@type": "cr:FileObject",
@@ -128,8 +128,8 @@
"description": "Every number the Atlas page shows, for six filter presets (analysis/stats/site_stats.py; what is measured is analysis/stats/measures.yaml).", "description": "Every number the Atlas page shows, for six filter presets (analysis/stats/site_stats.py; what is measured is analysis/stats/measures.yaml).",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/stats.json", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/stats.json",
"encodingFormat": "application/json", "encodingFormat": "application/json",
"sha256": "aa9e84d1336dcdd8d957e531ad6e5b9f95adcdc2b98363b0e856e2cc6e28dcb8", "sha256": "9285350c4d807bcdc7e8ea78cd1c9d207453913638fe4f32e8b7572e4f7927b4",
"contentSize": "5308966 B" "contentSize": "5426522 B"
}, },
{ {
"@type": "cr:FileObject", "@type": "cr:FileObject",
@@ -138,8 +138,8 @@
"description": "The route model: the game (core/spec/game.yaml), the count tables behind every roll, and the best decision tree for every dragon colour under two goals, with posterior intervals (planner/build_policy.py).", "description": "The route model: the game (core/spec/game.yaml), the count tables behind every roll, and the best decision tree for every dragon colour under two goals, with posterior intervals (planner/build_policy.py).",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/policy.json", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/policy.json",
"encodingFormat": "application/json", "encodingFormat": "application/json",
"sha256": "966d32a52d86fde0239ae8b1821b43f888a159d94b0bce908ef8aaf18119efa0", "sha256": "4709ad4f21cc47173b735d7fd2f33ae9982adef8797a9dd496cc7f230a31948d",
"contentSize": "1102627 B" "contentSize": "1108090 B"
}, },
{ {
"@type": "cr:FileObject", "@type": "cr:FileObject",
@@ -148,8 +148,8 @@
"description": "The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).", "description": "The route graph over every run in runs_clean.csv, counted per source (analysis/network/build-graph.js).",
"contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/graph.json", "contentUrl": "https://git.kenschiano.net/kathleen/threshing-atlas/raw/branch/main/data/public/graph.json",
"encodingFormat": "application/json", "encodingFormat": "application/json",
"sha256": "5b5dc0f215b42e9f65454fffb256172f2f70dedf703000aaf080ad42ee79d280", "sha256": "d03f29336b136ab800d6995644cb72a772c5f4181531e5486bcdb3d798c29a46",
"contentSize": "74020 B" "contentSize": "74424 B"
} }
], ],
"recordSet": [ "recordSet": [
@@ -196,7 +196,7 @@
"@type": "cr:Field", "@type": "cr:Field",
"@id": "runs/entry", "@id": "runs/entry",
"name": "entry", "name": "entry",
"description": "For spreadsheet runs, the sheet's own row label: \"Form\" for a form submission (it has a timestamp); bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept them, so they are the least selectively reported rows. Empty for other sources. Values: Form; Data Import; BulkUpload3; Manual Upload; Historical import; Unknown import.", "description": "For spreadsheet runs, the sheet's own row label: \"Form\" for a form submission (it has a timestamp); bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept them, so they are the least selectively reported rows. Empty for other sources. Values: Form; Data Import; BulkUpload3; Manual Upload; Historical import; Unknown import; Unknown.",
"dataType": "sc:Text", "dataType": "sc:Text",
"source": { "source": {
"fileObject": { "fileObject": {
+1 -1
View File
@@ -81,7 +81,7 @@ columns:
For spreadsheet runs, the sheet's own row label: "Form" for a form submission (it has a timestamp); For spreadsheet runs, the sheet's own row label: "Form" for a form submission (it has a timestamp);
bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept bulk-pasted logs keep the label they were pasted with. Bulk logs record every run of the players who kept
them, so they are the least selectively reported rows. Empty for other sources. them, so they are the least selectively reported rows. Empty for other sources.
values: [Form, Data Import, BulkUpload3, Manual Upload, Historical import, Unknown import] values: [Form, Data Import, BulkUpload3, Manual Upload, Historical import, Unknown import, Unknown]
bot_mode: bot_mode:
type: text type: text
description: >- description: >-
+6 -6
View File
@@ -10,14 +10,14 @@
"changed": "2026-11-16" "changed": "2026-11-16"
}, },
"reddit_spreadsheet": { "reddit_spreadsheet": {
"sha256": "cfc2cff4a0f69d65816a4c7ac7b6ce21a845f9e0324d796ec8ac5196916222ea", "sha256": "3de851ccfe9d4daeea16e621384d0eba33d42c389773d1a96ee12720193d468c",
"bytes": 6943129, "bytes": 4685924,
"changed": "2026-11-16" "changed": "2026-11-18"
}, },
"register": { "register": {
"sha256": "0f818a7c7e56e1346522153ddac169e726d0b663d030392e6114bffe55f6f649", "sha256": "1ed64e285cbc3bc3dc6969970430fd500455fee2fe14d93a7f14a8e10f9e85c3",
"bytes": 51026, "bytes": 51191,
"changed": "2026-11-16" "changed": "2026-11-18"
}, },
"reported": { "reported": {
"sha256": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", "sha256": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945",
+24 -24
View File
@@ -3,34 +3,34 @@
"by_attempt": { "by_attempt": {
"form": { "form": {
"1": [ "1": [
89, 92,
879 920
], ],
"2": [ "2": [
0, 0,
288 296
], ],
"3": [ "3": [
0, 0,
184 192
], ],
"4-10": [ "4-10": [
0, 0,
506 536
], ],
"11-30": [ "11-30": [
0, 0,
573 614
], ],
"31 or more": [ "31 or more": [
1, 1,
610 679
] ]
}, },
"bulk logs": { "bulk logs": {
"1": [ "1": [
0, 0,
28 29
], ],
"2": [ "2": [
0, 0,
@@ -81,19 +81,19 @@
} }
}, },
"same_route": { "same_route": {
"routes": 321, "routes": 323,
"first_shadows": 85, "first_shadows": 88,
"expected": 36.4, "expected": 37.2,
"odds_ratio": 626.3, "odds_ratio": 653.8,
"z": 11.9 "z": 12.1
}, },
"form_first": [ "form_first": [
89, 92,
879 920
], ],
"form_later": [ "form_later": [
1, 1,
2161 2317
], ],
"best_route": { "best_route": {
"route": [ "route": [
@@ -105,27 +105,27 @@
"West" "West"
], ],
"first": [ "first": [
22, 23,
88 93
], ],
"later": [ "later": [
0, 0,
89 94
] ]
}, },
"shadow_routes": { "shadow_routes": {
"runs": 93, "runs": 96,
"routes": 46, "routes": 46,
"off_gate": 3 "off_gate": 3
}, },
"after_shadow": { "after_shadow": {
"reached_black": 76, "reached_black": 79,
"bonded_black": 55, "bonded_black": 57,
"died_at_blue": 15, "died_at_blue": 15,
"answers": { "answers": {
"held its gaze": [ "held its gaze": [
30, 29,
42 41
], ],
"lowered eyes": [ "lowered eyes": [
0, 0,
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+5894 -5442
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long