Route model: first attempts are their own shadow roll; experiment plan and automated bot cycle

- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
  depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
  so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
  0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
  experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
This commit is contained in:
ekcSchiano
2026-11-12 03:31:33 +00:00
co-authored by Claude Fable 5.2
parent ee07ee143e
commit ab3664f369
25 changed files with 1938 additions and 573 deletions
+114
View File
@@ -0,0 +1,114 @@
# The experiment plan for Ken's bot: what each attempt of each account is for.
#
# This file is the design (written by hand). planner/build_experiments.py checks it against the game
# (core/spec/game.yaml), counts what the bot has run so far, decides which experiments are still running, and
# writes data/public/experiments.json. The bot reads that file with assign() in core/solver.js. The cycle
# (python -m collection cycle) rebuilds it after every import of bot runs, so the plan follows the data:
# an experiment that reaches its planned count, or its stopping rule, stops being assigned, and its share goes
# to the others.
#
# Rules that keep the results honest:
# - counts are planned in advance; an experiment does not stop because its results look good;
# - every started attempt is logged, including aborted ones;
# - the bot never accepts a bond that is not the experiment's target colour (it burns), and it does accept a
# genuine Black or Blue bond.
version: 1
seed: 20261112 # changes every arm assignment: leave it alone once accounts are running
# ---------------------------------------------------------------- an account's life
lifecycle:
# attempt 1 of every new account is the Black attempt (the shadow is close to a first-attempt event), and is
# itself an experiment: first-attempt-shadow below
research_days: 3 # then the account collects route data for this long (about 18 attempts at 4 hours each)
retire: true # and stops, except the long-lived accounts
long_lived: { every: 10, take: 1 } # one account in ten keeps running, so attempts past 18 are still observed
# ---------------------------------------------------------------- the experiments
# attempts: which of an account's attempts it covers (from / to, inclusive; no `to` = from then on).
# unit: what is randomised. account = one arm for the account's life (needed when only attempt 1 counts);
# attempt = arms in balanced blocks within each account.
# share: when several experiments cover an attempt, how often each gets it.
# exclusive: takes every attempt it covers; the shared experiments get the rest.
# accounts: { every: N, take: K } limits it to K accounts in every N (by the order accounts were made).
# factors: decisions that are randomised; every combination is an arm. route: decisions held on every arm.
# Everything else is the advisor's best play for the target.
# sampling: best (default), thompson, or uniform (every option on the screen equally likely: mapping).
# measures: the roll and the outcome that are counted. planned_per_arm: how many observed rolls per arm.
# stop_early: pause when, after `after` observed rolls in total, at most `rare_at_most` came out rare.
experiments:
- id: first-attempt-shadow
question: How often does the shadow appear on an account's first attempt, and does heading West (at the start, at the fork) raise it?
why: >
Form submissions show the shadow on 9% of first attempts and 0.3% of later ones, but form submissions
over-report it, complete logs have only 28 first attempts, and the bot has none. Among first attempts the
form shows more shadows after West (14%) than East (4%), which may be copied guide routes.
attempts: { from: 1, to: 1 }
unit: account
exclusive: true
share: 1
target: Black
objective: eventually
factors: { direction: [West, East], post: [West, East] }
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
measures: { roll: shadow, rare: "yes" }
planned_per_arm: 60 # 240 accounts: 120 per level of each factor
stop_early: { after: 120, rare_at_most: 0 } # no shadow in 120 first attempts rules out 2.5% or more
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
- id: attempt-2-shadow
question: Is the shadow really gone after the first attempt?
why: The form shows 3 shadows in 295 second attempts (1%), then almost none.
attempts: { from: 2, to: 2 }
unit: account
accounts: { every: 2, take: 1 }
exclusive: true
share: 1
target: Black
objective: eventually
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
measures: { roll: shadow, rare: "yes" }
planned_per_arm: 120
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
- id: meadow-blue-route
question: Does anything before the meadow change how often the Blue waits there?
why: >
Complete logs show a Blue on about 10% of meadow visits whatever the route, from routes the bot's rules chose.
Random assignment is what can confirm that the route does not matter (or find the route where it does).
attempts: { from: 2 }
unit: attempt
share: 0.6
target: Blue
objective: eventually
factors: { scream: [Investigate the scream, Keep walking], direction: [West, East, Center] }
route: { hour: Valley, valley: Pass the Red (meadow) }
measures: { roll: meadow, rare: Blue }
planned_per_arm: 70 # 210 per scream choice, 140 per direction: enough to tell 10% from 20%
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
- id: mapping
question: What does every branch of the valley lead to?
why: Probabilities for every path to every dragon need runs on the paths nobody chooses.
attempts: { from: 2 }
unit: attempt
share: 0.25
target: Blue
objective: eventually
sampling: uniform
planned_per_arm: 600
burn: { reason: NOT_TARGET_AUTO_BURN, when: the encounter is not a Blue or a Black }
- id: sentinel
question: Have the game's odds changed?
why: >
One fixed route, run all the time, by accounts that log everything: a change in its rates is a change in the
game, not in who reports. It is the most-logged meadow route.
attempts: { from: 2 }
unit: attempt
share: 0.15
target: Blue
objective: eventually
route: { company: Alone, direction: Center, scream: Keep walking, wounded: Tourniquet, hour: Valley, valley: Pass the Red (meadow) }
measures: { roll: meadow, rare: Blue }
ongoing: true # never completes
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }