Route model: first attempts are their own shadow roll; experiment plan and automated bot cycle
- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
This commit is contained in:
co-authored by
Claude Fable 5.2
parent
ee07ee143e
commit
ab3664f369
+6
-3
@@ -7,9 +7,12 @@ The route planner: the best route to every dragon, and what goes back to Ken's b
|
||||
| `build_policy.py` | Solves the route model (`core/atlas/solver.py`, built from the game in `core/spec/game.yaml`) for every colour and both goals ("this attempt" and "eventually"), with 200 posterior draws and a cross-fit check. Writes `data/public/policy.json`, which has a content id so each bot run can record which policy it followed. About 2 minutes. |
|
||||
| `policy_report.py` | Writes the readable version, `analysis/reports/POLICY.md`. |
|
||||
|
||||
| `experiments.yaml` | The experiment plan, written by hand: what each attempt of each bot account is for (the Black attempt on attempt 1, then the meadow Blue experiment, mapping and the sentinel route), with planned counts and stopping rules. |
|
||||
| `build_experiments.py` | Checks the plan against the game, counts what the bot has logged for every arm, sets each experiment's status (running, complete, paused), and writes `data/public/experiments.json` for the bot. |
|
||||
|
||||
Run both with `npm run policy`. The site's valley map and The Odds read `policy.json`; the bot downloads it with
|
||||
`solver.js` and `graph.js` (see `docs/BOT_INTEGRATION.md`).
|
||||
|
||||
**The loop with the bot.** The planner's routes go out to the bot; the bot's runs come back through
|
||||
`collection/bot/`. Experiments the bot should run to fill gaps the community form cannot (for example the
|
||||
answers at each bonding encounter) belong here; `experiments.json` is on the roadmap.
|
||||
**The loop with the bot.** `python -m collection cycle --bot <the bot's folder> --deliver <its vendor folder>`
|
||||
imports the bot's new runs, refits the route model, rebuilds the experiment plan and hands both back, in one
|
||||
command that is safe to run on a timer (`docs/BOT_INTEGRATION.md`, "The automated cycle").
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
"""Build data/public/experiments.json: the experiment plan the bot follows, updated from the bot's own runs.
|
||||
|
||||
python planner/build_experiments.py [--out data/public/experiments.json] [--check]
|
||||
|
||||
Reads the design (planner/experiments.yaml), checks it against the game (core/spec/game.yaml), counts what the bot
|
||||
has logged for every arm (data/bot/trajectories-*.jsonl, matched to data/public/runs_clean.csv), and decides each
|
||||
experiment's status:
|
||||
|
||||
running still being assigned
|
||||
complete every arm has its planned number of observed rolls
|
||||
paused its stop_early rule was met (the plan needs a person)
|
||||
|
||||
The bot reads the result with assign() in core/solver.js; an experiment that is not running is no longer assigned
|
||||
and its share goes to the others. The plan has a content id (logged by the bot as planId).
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "core"))
|
||||
from atlas import model # noqa: E402
|
||||
from atlas import runs as runs_io # noqa: E402
|
||||
|
||||
DESIGN = ROOT / "planner/experiments.yaml"
|
||||
OUT = ROOT / "data/public/experiments.json"
|
||||
SAMPLING = ("best", "thompson", "uniform")
|
||||
|
||||
|
||||
def design():
|
||||
with open(DESIGN, encoding="utf-8") as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
|
||||
def check_options(where, fixed):
|
||||
for k, o in (fixed or {}).items():
|
||||
if k not in model.DECISIONS or k in ("encounter", "cadet"):
|
||||
raise ValueError(f"{where}: '{k}' is not a decision of the game")
|
||||
if o not in model.DECISIONS[k]:
|
||||
raise ValueError(f"{where}: '{k}' has no option '{o}' (game.yaml: {model.DECISIONS[k]})")
|
||||
|
||||
|
||||
def arms(exp):
|
||||
"""Every combination of the factors, each with the route held on every arm."""
|
||||
route = exp.get("route") or {}
|
||||
check_options(exp["id"] + " route", route)
|
||||
factors = exp.get("factors") or {}
|
||||
for k, opts in factors.items():
|
||||
if k in route:
|
||||
raise ValueError(f"{exp['id']}: '{k}' is both a factor and part of the route")
|
||||
for o in opts:
|
||||
check_options(exp["id"] + " factors", {k: o})
|
||||
if not factors:
|
||||
return [{"id": "all", "fix": dict(route)}]
|
||||
out = []
|
||||
for combo in itertools.product(*factors.values()):
|
||||
levels = dict(zip(factors, combo))
|
||||
out.append({"id": " / ".join(f"{k}={o}" for k, o in levels.items()), "levels": levels, "fix": {**route, **levels}})
|
||||
return out
|
||||
|
||||
|
||||
def validate(d):
|
||||
ids = [e["id"] for e in d["experiments"]]
|
||||
if len(set(ids)) != len(ids):
|
||||
raise ValueError("experiment ids must be unique")
|
||||
for e in d["experiments"]:
|
||||
if e["target"] not in model.COLOURS:
|
||||
raise ValueError(f"{e['id']}: unknown target {e['target']}")
|
||||
if e.get("sampling", "best") not in SAMPLING:
|
||||
raise ValueError(f"{e['id']}: sampling must be one of {SAMPLING}")
|
||||
if e["unit"] not in ("account", "attempt"):
|
||||
raise ValueError(f"{e['id']}: unit must be account or attempt")
|
||||
if not e.get("share", 0) > 0:
|
||||
raise ValueError(f"{e['id']}: share must be positive")
|
||||
m = e.get("measures")
|
||||
if m and (m["roll"] not in model.CHANCE or m["rare"] not in model.CHANCE[m["roll"]]["outcomes"]):
|
||||
raise ValueError(f"{e['id']}: measures {m} is not a roll and outcome of the game")
|
||||
if not e.get("ongoing") and not e.get("planned_per_arm"):
|
||||
raise ValueError(f"{e['id']}: needs planned_per_arm (or ongoing: true)")
|
||||
|
||||
|
||||
def interval(y, n, seed=1, sims=4000):
|
||||
"""Jeffreys 90% range for a rate."""
|
||||
if not n:
|
||||
return None
|
||||
r = random.Random(seed)
|
||||
s = sorted(r.betavariate(y + 0.5, n - y + 0.5) for _ in range(sims))
|
||||
return {"rate": (y + 0.5) / (n + 1), "lo": s[int(0.05 * sims)], "hi": s[int(0.95 * sims) - 1]}
|
||||
|
||||
|
||||
def logged(bot_dir=ROOT / "data/bot", runs_path=None):
|
||||
"""Bot runs that carry an experiment: [(experimentId, arm, prepared run)], each run once."""
|
||||
prepared = {r["run_id"]: model.prepare_run(r) for r in runs_io.read(runs_path or runs_io.PATH, sources=["bot"], drop=())}
|
||||
seen, out = set(), []
|
||||
for f in sorted(glob.glob(str(Path(bot_dir) / "trajectories-*.jsonl"))):
|
||||
if "sample" in Path(f).name:
|
||||
continue
|
||||
with open(f, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
t = json.loads(line)
|
||||
meta = t.get("meta") or {}
|
||||
if not meta.get("experimentId") or t["id"] in seen:
|
||||
continue
|
||||
seen.add(t["id"])
|
||||
out.append((meta["experimentId"], meta.get("arm"), prepared.get(t["id"])))
|
||||
return out
|
||||
|
||||
|
||||
def build(d=None, log=None):
|
||||
d = d or design()
|
||||
validate(d)
|
||||
log = logged() if log is None else log
|
||||
exps = []
|
||||
for e in d["experiments"]:
|
||||
m = e.get("measures")
|
||||
a = arms(e)
|
||||
tally = {x["id"]: {"runs": 0, "observed": 0, "rare": 0} for x in a}
|
||||
for exp_id, arm, v in log:
|
||||
if exp_id != e["id"]:
|
||||
continue
|
||||
t = tally.setdefault(arm or "?", {"runs": 0, "observed": 0, "rare": 0})
|
||||
t["runs"] += 1
|
||||
y = model.OBSERVE[m["roll"]](v) if m and v else None
|
||||
if y is not None:
|
||||
t["observed"] += 1
|
||||
t["rare"] += y == m["rare"]
|
||||
planned = e.get("planned_per_arm")
|
||||
# mapping has nothing to measure: its runs are its count
|
||||
done = {k: (t["observed"] if m else t["runs"]) for k, t in tally.items() if k in {x["id"] for x in a}}
|
||||
total_obs = sum(t["observed"] for t in tally.values())
|
||||
total_rare = sum(t["rare"] for t in tally.values())
|
||||
status, note = "running", None
|
||||
se = e.get("stop_early")
|
||||
if not e.get("ongoing") and all(n >= planned for n in done.values()):
|
||||
status, note = "complete", f"every arm has its planned {planned}"
|
||||
elif se and total_obs >= se["after"] and total_rare <= se["rare_at_most"]:
|
||||
status = "paused"
|
||||
note = f"stop_early: {total_rare} rare outcome(s) in {total_obs} observed rolls (rule: at most {se['rare_at_most']} after {se['after']})"
|
||||
for x in a:
|
||||
t = tally[x["id"]]
|
||||
x["progress"] = {**t, "planned": planned, **({"estimate": interval(t["rare"], t["observed"])} if m and t["observed"] else {})}
|
||||
exps.append({
|
||||
"id": e["id"], "status": status, **({"note": note} if note else {}),
|
||||
"question": e["question"], "attempts": {"from": e["attempts"]["from"], "to": e["attempts"].get("to")},
|
||||
"unit": e["unit"], "share": e["share"], **({"accounts": e["accounts"]} if e.get("accounts") else {}),
|
||||
**({"exclusive": True} if e.get("exclusive") else {}),
|
||||
"target": e["target"], "objective": e.get("objective", "eventually"), "sampling": e.get("sampling", "best"),
|
||||
"seed": d["seed"], "measures": m, "burn": e.get("burn"), "arms": a,
|
||||
"progress": {"runs": sum(t["runs"] for t in tally.values()), "observed": total_obs, "rare": total_rare,
|
||||
**({"estimate": interval(total_rare, total_obs)} if m and total_obs else {}),
|
||||
"unknownArms": sorted(k for k in tally if k not in {x["id"] for x in a})},
|
||||
})
|
||||
plan = {"schema": 1, "seed": d["seed"], "lifecycle": d["lifecycle"], "experiments": exps}
|
||||
plan["id"] = hashlib.sha1(json.dumps(plan, sort_keys=True, separators=(",", ":")).encode()).hexdigest()[:12]
|
||||
return plan
|
||||
|
||||
|
||||
def render(plan):
|
||||
return json.dumps(plan, indent=1, ensure_ascii=False) + "\n"
|
||||
|
||||
|
||||
def summary(plan):
|
||||
L = [f"experiment plan {plan['id']}"]
|
||||
for e in plan["experiments"]:
|
||||
p = e["progress"]
|
||||
est = p.get("estimate")
|
||||
L.append(f" {e['id']:<22} {e['status']:<9} {p['observed']} observed, {p['rare']} rare"
|
||||
+ (f" ({100 * est['rate']:.1f}%, 90% range {100 * est['lo']:.1f}-{100 * est['hi']:.1f})" if est else "")
|
||||
+ (f" [{e['note']}]" if e.get("note") else ""))
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--out", default=str(OUT))
|
||||
ap.add_argument("--check", action="store_true", help="write nothing; exit 1 if the file is out of date")
|
||||
a = ap.parse_args(argv)
|
||||
plan = build()
|
||||
text = render(plan)
|
||||
out = Path(a.out)
|
||||
if a.check:
|
||||
if not out.exists() or out.read_text(encoding="utf-8") != text:
|
||||
print(f"{out} is out of date (run python planner/build_experiments.py)", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
out.write_text(text, encoding="utf-8")
|
||||
print(summary(plan))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
# The experiment plan for Ken's bot: what each attempt of each account is for.
|
||||
#
|
||||
# This file is the design (written by hand). planner/build_experiments.py checks it against the game
|
||||
# (core/spec/game.yaml), counts what the bot has run so far, decides which experiments are still running, and
|
||||
# writes data/public/experiments.json. The bot reads that file with assign() in core/solver.js. The cycle
|
||||
# (python -m collection cycle) rebuilds it after every import of bot runs, so the plan follows the data:
|
||||
# an experiment that reaches its planned count, or its stopping rule, stops being assigned, and its share goes
|
||||
# to the others.
|
||||
#
|
||||
# Rules that keep the results honest:
|
||||
# - counts are planned in advance; an experiment does not stop because its results look good;
|
||||
# - every started attempt is logged, including aborted ones;
|
||||
# - the bot never accepts a bond that is not the experiment's target colour (it burns), and it does accept a
|
||||
# genuine Black or Blue bond.
|
||||
version: 1
|
||||
seed: 20261112 # changes every arm assignment: leave it alone once accounts are running
|
||||
|
||||
# ---------------------------------------------------------------- an account's life
|
||||
lifecycle:
|
||||
# attempt 1 of every new account is the Black attempt (the shadow is close to a first-attempt event), and is
|
||||
# itself an experiment: first-attempt-shadow below
|
||||
research_days: 3 # then the account collects route data for this long (about 18 attempts at 4 hours each)
|
||||
retire: true # and stops, except the long-lived accounts
|
||||
long_lived: { every: 10, take: 1 } # one account in ten keeps running, so attempts past 18 are still observed
|
||||
|
||||
# ---------------------------------------------------------------- the experiments
|
||||
# attempts: which of an account's attempts it covers (from / to, inclusive; no `to` = from then on).
|
||||
# unit: what is randomised. account = one arm for the account's life (needed when only attempt 1 counts);
|
||||
# attempt = arms in balanced blocks within each account.
|
||||
# share: when several experiments cover an attempt, how often each gets it.
|
||||
# exclusive: takes every attempt it covers; the shared experiments get the rest.
|
||||
# accounts: { every: N, take: K } limits it to K accounts in every N (by the order accounts were made).
|
||||
# factors: decisions that are randomised; every combination is an arm. route: decisions held on every arm.
|
||||
# Everything else is the advisor's best play for the target.
|
||||
# sampling: best (default), thompson, or uniform (every option on the screen equally likely: mapping).
|
||||
# measures: the roll and the outcome that are counted. planned_per_arm: how many observed rolls per arm.
|
||||
# stop_early: pause when, after `after` observed rolls in total, at most `rare_at_most` came out rare.
|
||||
experiments:
|
||||
- id: first-attempt-shadow
|
||||
question: How often does the shadow appear on an account's first attempt, and does heading West (at the start, at the fork) raise it?
|
||||
why: >
|
||||
Form submissions show the shadow on 9% of first attempts and 0.3% of later ones, but form submissions
|
||||
over-report it, complete logs have only 28 first attempts, and the bot has none. Among first attempts the
|
||||
form shows more shadows after West (14%) than East (4%), which may be copied guide routes.
|
||||
attempts: { from: 1, to: 1 }
|
||||
unit: account
|
||||
exclusive: true
|
||||
share: 1
|
||||
target: Black
|
||||
objective: eventually
|
||||
factors: { direction: [West, East], post: [West, East] }
|
||||
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
||||
measures: { roll: shadow, rare: "yes" }
|
||||
planned_per_arm: 60 # 240 accounts: 120 per level of each factor
|
||||
stop_early: { after: 120, rare_at_most: 0 } # no shadow in 120 first attempts rules out 2.5% or more
|
||||
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
||||
|
||||
- id: attempt-2-shadow
|
||||
question: Is the shadow really gone after the first attempt?
|
||||
why: The form shows 3 shadows in 295 second attempts (1%), then almost none.
|
||||
attempts: { from: 2, to: 2 }
|
||||
unit: account
|
||||
accounts: { every: 2, take: 1 }
|
||||
exclusive: true
|
||||
share: 1
|
||||
target: Black
|
||||
objective: eventually
|
||||
route: { scream: Investigate the scream, clearing: Help your squadmate, hour: Tree }
|
||||
measures: { roll: shadow, rare: "yes" }
|
||||
planned_per_arm: 120
|
||||
burn: { reason: TREE_NO_SHADOW_AUTO_BURN, when: no shadow at the tree }
|
||||
|
||||
- id: meadow-blue-route
|
||||
question: Does anything before the meadow change how often the Blue waits there?
|
||||
why: >
|
||||
Complete logs show a Blue on about 10% of meadow visits whatever the route, from routes the bot's rules chose.
|
||||
Random assignment is what can confirm that the route does not matter (or find the route where it does).
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.6
|
||||
target: Blue
|
||||
objective: eventually
|
||||
factors: { scream: [Investigate the scream, Keep walking], direction: [West, East, Center] }
|
||||
route: { hour: Valley, valley: Pass the Red (meadow) }
|
||||
measures: { roll: meadow, rare: Blue }
|
||||
planned_per_arm: 70 # 210 per scream choice, 140 per direction: enough to tell 10% from 20%
|
||||
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|
||||
|
||||
- id: mapping
|
||||
question: What does every branch of the valley lead to?
|
||||
why: Probabilities for every path to every dragon need runs on the paths nobody chooses.
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.25
|
||||
target: Blue
|
||||
objective: eventually
|
||||
sampling: uniform
|
||||
planned_per_arm: 600
|
||||
burn: { reason: NOT_TARGET_AUTO_BURN, when: the encounter is not a Blue or a Black }
|
||||
|
||||
- id: sentinel
|
||||
question: Have the game's odds changed?
|
||||
why: >
|
||||
One fixed route, run all the time, by accounts that log everything: a change in its rates is a change in the
|
||||
game, not in who reports. It is the most-logged meadow route.
|
||||
attempts: { from: 2 }
|
||||
unit: attempt
|
||||
share: 0.15
|
||||
target: Blue
|
||||
objective: eventually
|
||||
route: { company: Alone, direction: Center, scream: Keep walking, wounded: Tourniquet, hour: Valley, valley: Pass the Red (meadow) }
|
||||
measures: { roll: meadow, rare: Blue }
|
||||
ongoing: true # never completes
|
||||
burn: { reason: MEADOW_NO_BLUE_AUTO_BURN, when: no Blue in the meadow }
|
||||
+7
-1
@@ -25,7 +25,13 @@ reporting_bias:
|
||||
rolls: { shadow: "yes", meadow: Blue } # roll: its rare outcome
|
||||
min_runs: 30 # fewer runs than this in a group (or the complete groups): borrow
|
||||
min_complete_rare: 5 # fewer rare outcomes than this in complete logs: the bias cannot be
|
||||
# estimated, so the roll is learned from complete logs only
|
||||
# estimated, so the roll is learned from complete logs only, unless
|
||||
# `borrow` names another roll for it
|
||||
# An assumption, not an estimate: complete logs have never seen the shadow, so its reporting bias cannot be
|
||||
# measured. Each group is taken to over-report the shadow as much as it over-reports the meadow Blue. Without
|
||||
# this, form submissions are left out of the shadow roll and the first-attempt effect (seen only in them) is
|
||||
# lost. Remove the line to go back to complete logs only; the bot's first-attempt runs will replace it.
|
||||
borrow: { shadow: meadow }
|
||||
groups: # first match wins
|
||||
- { name: bot, when: { source: bot }, complete: true }
|
||||
- { name: bulk logs, when: { source: reddit spreadsheet, entry_not: Form }, complete: true }
|
||||
|
||||
@@ -74,6 +74,15 @@ def main():
|
||||
a, e, l = t["thisAttempt"], t["eventually"], lb["targets"][X]["thisAttempt"]
|
||||
att = e["expectedAttempts"]["mean"]
|
||||
L.append(f"| {X} | {pct(a['pX']['mean'])} ({pct(a['pX']['lo'])}–{pct(a['pX']['hi'])}) | {pct(l['pX']['mean'])} ({pct(l['pX']['lo'])}–{pct(l['pX']['hi'])}) | {pct(t['crossFit']['heldOut'])} | {pct(e['ratio']['mean'])} | {'–' if att is None else js_round(att)} |")
|
||||
L += ["", "## First attempts", ""]
|
||||
L += ["The shadow at the tree is close to a first-attempt event, so an account's first attempt is solved separately (`givens` in `core/spec/game.yaml`). Only the colours that can come from the shadow differ much.", ""]
|
||||
L += ["| Colour | First attempt | Low-bias sample | Later attempt |", "|---|---|---|---|"]
|
||||
for X, t in s["targets"].items():
|
||||
f, fl, a = t["firstAttempt"]["thisAttempt"]["pX"], lb["targets"][X]["firstAttempt"]["thisAttempt"]["pX"], t["thisAttempt"]["pX"]
|
||||
L.append(f"| {X} | {pct(f['mean'])} ({pct(f['lo'])}–{pct(f['hi'])}) | {pct(fl['mean'])} ({pct(fl['lo'])}–{pct(fl['hi'])}) | {pct(a['mean'])} ({pct(a['lo'])}–{pct(a['hi'])}) |")
|
||||
sh = (s["model"].get("reportingBias") or {}).get("form, chasing Blue/Black", {}).get("shadow", {})
|
||||
if sh.get("used", "").startswith("assumed"):
|
||||
L += ["", f"The shadow's reporting bias cannot be measured yet (complete logs have never seen one), so it is assumed equal to the meadow Blue's (`borrow` in `planner/policy.yaml`): form reports of a shadow count 1/{sh['rho']:.1f} each. The bot's first-attempt runs will replace this assumption."]
|
||||
L += ["", "## How much do earlier choices matter?", ""]
|
||||
L += ["Each chance event can depend on the choices before it. The model learns how much from the data (empirical-Bayes concentration; larger means the history matters less):", ""]
|
||||
L += ["| Chance event | Depends on (most important first) | Concentration |", "|---|---|---|"]
|
||||
|
||||
Reference in New Issue
Block a user