Route model: first attempts are their own shadow roll; experiment plan and automated bot cycle
- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
This commit is contained in:
co-authored by
Claude Fable 5.2
parent
ee07ee143e
commit
ab3664f369
@@ -0,0 +1,114 @@
|
||||
# The experiment plan for Ken's bot: what each attempt of each account is for.
|
||||
#
|
||||
# This file is the design (written by hand). planner/build_experiments.py checks it against the game
|
||||
# (core/spec/game.yaml), counts what the bot has run so far, decides which experiments are still running, and
|
||||
# writes data/public/experiments.json. The bot reads that file with assign() in core/solver.js. The cycle
|
||||
# (python -m collection cycle) rebuilds it after every import of bot runs, so the plan follows the data:
|
||||
# an experiment that reaches its planned count, or its stopping rule, stops being assigned, and its share goes
|
||||
# to the others.
|
||||
#
|
||||
# Rules that keep the results honest:
|
||||
# - counts are planned in advance; an experiment does not stop because its results look good;
|
||||
# - every started attempt is logged, including aborted ones;
|
||||
# - the bot never accepts a bond that is not the experiment's target colour (it burns), and it does accept a
|
||||
# genuine Black or Blue bond.
|
||||
version: 1
|
||||
seed: 20261112 # changes every arm assignment: leave it alone once accounts are running
|
||||
|
||||
# ---------------------------------------------------------------- an account's life
|
||||
lifecycle:
|
||||
# attempt 1 of every new account is the Black attempt (the shadow is close to a first-attempt event), and is
|
||||
# itself an experiment: first-attempt-shadow below
|
||||
research_days: 3 # then the account collects route data for this long (about 18 attempts at 4 hours each)
|
||||
retire: true # and stops, except the long-lived accounts
|
||||
long_lived: { every: 10, take: 1 } # one account in ten keeps running, so attempts past 18 are still observed
|
||||
|
||||
# ---------------------------------------------------------------- the experiments
|
||||
# attempts: which of an account's attempts it covers (from / to, inclusive; no `to` = from then on).
|
||||
# unit: what is randomised. account = one arm for the account's life (needed when only attempt 1 counts);
|
||||
# attempt = arms in balanced blocks within each account.
|
||||
# share: when several experiments cover an attempt, how often each gets it.
|
||||
# exclusive: takes every attempt it covers; the shared experiments get the rest.
|
||||
# accounts: { every: N, take: K } limits it to K accounts in every N (by the order accounts were made).
|
||||
# factors: decisions that are randomised; every combination is an arm. route: decisions held on every arm.
|
||||
# Everything else is the advisor's best play for the target.
|
||||
# sampling: best (default), thompson, or uniform (every option on the screen equally likely: mapping).
|
||||
# measures: the roll and the outcome that are counted. planned_per_arm: how many observed rolls per arm.
|
||||
# stop_early: pause when, after `after` observed rolls in total, at most `rare_at_most` came out rare.
|
||||
experiments:
|
||||
- id: first-attempt-shadow
|
||||
question: How often does the shadow appear on an account's first attempt, and does heading West (at the start, at the fork) raise it?
|
||||
why: >
|
||||
Form submissions show the shadow on 9% of first attempts and 0.3% of later ones, but form submissions
|
||||
over-report it, complete logs have only 28 first attempts, and the bot has none. Among first attempts the
|
||||
form shows more shadows after West (14%) than East (4%), which may be copied guide routes.
|
||||
attempts: { from: 1, to: 1 }
|
||||
unit: account
|
||||
exclusive: true
|
||||
share: 1
|
||||
target: Black
|
||||
objective: eventually
|
||||
factors: { direction: [West, East], post: [West, East] }
|
||||
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
||||
measures: { roll: shadow, rare: "yes" }
|
||||
planned_per_arm: 60 # 240 accounts: 120 per level of each factor
|
||||
stop_early: { after: 120, rare_at_most: 0 } # no shadow in 120 first attempts rules out 2.5% or more
|
||||
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
||||
|
||||
- id: attempt-2-shadow
|
||||
question: Is the shadow really gone after the first attempt?
|
||||
why: The form shows 3 shadows in 295 second attempts (1%), then almost none.
|
||||
attempts: { from: 2, to: 2 }
|
||||
unit: account
|
||||
accounts: { every: 2, take: 1 }
|
||||
exclusive: true
|
||||
share: 1
|
||||
target: Black
|
||||
objective: eventually
|
||||
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
||||
measures: { roll: shadow, rare: "yes" }
|
||||
planned_per_arm: 120
|
||||
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
||||
|
||||
- id: meadow-blue-route
|
||||
question: Does anything before the meadow change how often the Blue waits there?
|
||||
why: >
|
||||
Complete logs show a Blue on about 10% of meadow visits whatever the route, from routes the bot's rules chose.
|
||||
Random assignment is what can confirm that the route does not matter (or find the route where it does).
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.6
|
||||
target: Blue
|
||||
objective: eventually
|
||||
factors: { scream: [Investigate the scream, Keep walking], direction: [West, East, Center] }
|
||||
route: { hour: Valley, valley: Pass the Red (meadow) }
|
||||
measures: { roll: meadow, rare: Blue }
|
||||
planned_per_arm: 70 # 210 per scream choice, 140 per direction: enough to tell 10% from 20%
|
||||
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|
||||
|
||||
- id: mapping
|
||||
question: What does every branch of the valley lead to?
|
||||
why: Probabilities for every path to every dragon need runs on the paths nobody chooses.
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.25
|
||||
target: Blue
|
||||
objective: eventually
|
||||
sampling: uniform
|
||||
planned_per_arm: 600
|
||||
burn: { reason: NOT_TARGET_AUTO_BURN, when: the encounter is not a Blue or a Black }
|
||||
|
||||
- id: sentinel
|
||||
question: Have the game's odds changed?
|
||||
why: >
|
||||
One fixed route, run all the time, by accounts that log everything: a change in its rates is a change in the
|
||||
game, not in who reports. It is the most-logged meadow route.
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.15
|
||||
target: Blue
|
||||
objective: eventually
|
||||
route: { company: Alone, direction: Center, scream: Keep walking, wounded: Tourniquet, hour: Valley, valley: Pass the Red (meadow) }
|
||||
measures: { roll: meadow, rare: Blue }
|
||||
ongoing: true # never completes
|
||||
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|
||||
Reference in New Issue
Block a user