- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
115 lines
6.2 KiB
YAML
115 lines
6.2 KiB
YAML
# The experiment plan for Ken's bot: what each attempt of each account is for.
|
|
#
|
|
# This file is the design (written by hand). planner/build_experiments.py checks it against the game
|
|
# (core/spec/game.yaml), counts what the bot has run so far, decides which experiments are still running, and
|
|
# writes data/public/experiments.json. The bot reads that file with assign() in core/solver.js. The cycle
|
|
# (python -m collection cycle) rebuilds it after every import of bot runs, so the plan follows the data:
|
|
# an experiment that reaches its planned count, or its stopping rule, stops being assigned, and its share goes
|
|
# to the others.
|
|
#
|
|
# Rules that keep the results honest:
|
|
# - counts are planned in advance; an experiment does not stop because its results look good;
|
|
# - every started attempt is logged, including aborted ones;
|
|
# - the bot never accepts a bond that is not the experiment's target colour (it burns), and it does accept a
|
|
# genuine Black or Blue bond.
|
|
version: 1
|
|
seed: 20261112 # changes every arm assignment: leave it alone once accounts are running
|
|
|
|
# ---------------------------------------------------------------- an account's life
|
|
lifecycle:
|
|
# attempt 1 of every new account is the Black attempt (the shadow is close to a first-attempt event), and is
|
|
# itself an experiment: first-attempt-shadow below
|
|
research_days: 3 # then the account collects route data for this long (about 18 attempts at 4 hours each)
|
|
retire: true # and stops, except the long-lived accounts
|
|
long_lived: { every: 10, take: 1 } # one account in ten keeps running, so attempts past 18 are still observed
|
|
|
|
# ---------------------------------------------------------------- the experiments
|
|
# attempts: which of an account's attempts it covers (from / to, inclusive; no `to` = from then on).
|
|
# unit: what is randomised. account = one arm for the account's life (needed when only attempt 1 counts);
|
|
# attempt = arms in balanced blocks within each account.
|
|
# share: when several experiments cover an attempt, how often each gets it.
|
|
# exclusive: takes every attempt it covers; the shared experiments get the rest.
|
|
# accounts: { every: N, take: K } limits it to K accounts in every N (by the order accounts were made).
|
|
# factors: decisions that are randomised; every combination is an arm. route: decisions held on every arm.
|
|
# Everything else is the advisor's best play for the target.
|
|
# sampling: best (default), thompson, or uniform (every option on the screen equally likely: mapping).
|
|
# measures: the roll and the outcome that are counted. planned_per_arm: how many observed rolls per arm.
|
|
# stop_early: pause when, after `after` observed rolls in total, at most `rare_at_most` came out rare.
|
|
experiments:
|
|
- id: first-attempt-shadow
|
|
question: How often does the shadow appear on an account's first attempt, and does heading West (at the start, at the fork) raise it?
|
|
why: >
|
|
Form submissions show the shadow on 9% of first attempts and 0.3% of later ones, but form submissions
|
|
over-report it, complete logs have only 28 first attempts, and the bot has none. Among first attempts the
|
|
form shows more shadows after West (14%) than East (4%), which may be copied guide routes.
|
|
attempts: { from: 1, to: 1 }
|
|
unit: account
|
|
exclusive: true
|
|
share: 1
|
|
target: Black
|
|
objective: eventually
|
|
factors: { direction: [West, East], post: [West, East] }
|
|
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
|
measures: { roll: shadow, rare: "yes" }
|
|
planned_per_arm: 60 # 240 accounts: 120 per level of each factor
|
|
stop_early: { after: 120, rare_at_most: 0 } # no shadow in 120 first attempts rules out 2.5% or more
|
|
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
|
|
|
- id: attempt-2-shadow
|
|
question: Is the shadow really gone after the first attempt?
|
|
why: The form shows 3 shadows in 295 second attempts (1%), then almost none.
|
|
attempts: { from: 2, to: 2 }
|
|
unit: account
|
|
accounts: { every: 2, take: 1 }
|
|
exclusive: true
|
|
share: 1
|
|
target: Black
|
|
objective: eventually
|
|
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
|
measures: { roll: shadow, rare: "yes" }
|
|
planned_per_arm: 120
|
|
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
|
|
|
- id: meadow-blue-route
|
|
question: Does anything before the meadow change how often the Blue waits there?
|
|
why: >
|
|
Complete logs show a Blue on about 10% of meadow visits whatever the route, from routes the bot's rules chose.
|
|
Random assignment is what can confirm that the route does not matter (or find the route where it does).
|
|
attempts: { from: 2 }
|
|
unit: attempt
|
|
share: 0.6
|
|
target: Blue
|
|
objective: eventually
|
|
factors: { scream: [Investigate the scream, Keep walking], direction: [West, East, Center] }
|
|
route: { hour: Valley, valley: Pass the Red (meadow) }
|
|
measures: { roll: meadow, rare: Blue }
|
|
planned_per_arm: 70 # 210 per scream choice, 140 per direction: enough to tell 10% from 20%
|
|
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|
|
|
|
- id: mapping
|
|
question: What does every branch of the valley lead to?
|
|
why: Probabilities for every path to every dragon need runs on the paths nobody chooses.
|
|
attempts: { from: 2 }
|
|
unit: attempt
|
|
share: 0.25
|
|
target: Blue
|
|
objective: eventually
|
|
sampling: uniform
|
|
planned_per_arm: 600
|
|
burn: { reason: NOT_TARGET_AUTO_BURN, when: the encounter is not a Blue or a Black }
|
|
|
|
- id: sentinel
|
|
question: Have the game's odds changed?
|
|
why: >
|
|
One fixed route, run all the time, by accounts that log everything: a change in its rates is a change in the
|
|
game, not in who reports. It is the most-logged meadow route.
|
|
attempts: { from: 2 }
|
|
unit: attempt
|
|
share: 0.15
|
|
target: Blue
|
|
objective: eventually
|
|
route: { company: Alone, direction: Center, scream: Keep walking, wounded: Tourniquet, hour: Valley, valley: Pass the Red (meadow) }
|
|
measures: { roll: meadow, rare: Blue }
|
|
ongoing: true # never completes
|
|
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|