- game.yaml: `givens` (first or later attempt) and `distinct` levels that are not pooled; the shadow roll
depends on the attempt first. Form submissions show the shadow on 9% of first attempts and 0.3% of later ones.
- Reporting bias: a roll whose bias cannot be measured can borrow another roll's (`borrow: {shadow: meadow}`),
so form submissions inform the shadow again. An assumption, stated in planner/policy.yaml.
- policy.json: targets.<colour>.firstAttempt (this attempt, eventually). Black: about 1.4% on a first attempt,
0.07% on a later one.
- solver.js: advisor options `attempt`, `fix`, sampling "uniform"; assign() and armFor() for the plan.
- planner/experiments.yaml + build_experiments.py -> data/public/experiments.json (progress and status per
experiment, rebuilt from the bot's runs).
- python -m collection cycle: import bot runs, refit, re-plan, deliver the kit; .gitea/workflows/bot-cycle.yml.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqCC2LZ4x1WHQ1V57nTsv
103 lines
6.4 KiB
Python
103 lines
6.4 KiB
Python
"""Write analysis/reports/POLICY.md: a readable summary of data/public/policy.json.
|
||
|
||
python planner/policy_report.py
|
||
"""
|
||
import json
|
||
from pathlib import Path
|
||
|
||
ROOT = Path(__file__).resolve().parents[1]
|
||
|
||
|
||
def js_round(x):
|
||
"""Math.round: halves round up (toward +infinity), unlike Python's round()."""
|
||
import math
|
||
return math.floor(x + 0.5)
|
||
|
||
|
||
def to_fixed(x, digits):
|
||
"""Number.prototype.toFixed: round half away from zero on the decimal value."""
|
||
from decimal import Decimal, ROUND_HALF_UP
|
||
return str(Decimal(repr(x)).quantize(Decimal(1).scaleb(-digits), rounding=ROUND_HALF_UP))
|
||
|
||
|
||
def pct(p):
|
||
return "–" if p is None else to_fixed(100 * p, 1 if p < 0.1 else 0) + "%"
|
||
|
||
|
||
def conf(n):
|
||
return "" if n.get("stillBest") is None else f" (still best in {pct(n['stillBest'])} of posterior draws)"
|
||
|
||
|
||
def walk(n, out, indent=""):
|
||
if not n:
|
||
return
|
||
if n["type"] == "decision":
|
||
alts = ", ".join(f"{k} {pct(v['pX'])}" for k, v in sorted(n["options"].items(), key=lambda kv: -kv[1]["pX"]))
|
||
if n["tie"]:
|
||
out.append(f"{indent}- {n['label']}: any (no difference for this colour)")
|
||
else:
|
||
weak = " — no clear winner" if n.get("stillBest") is not None and n["stillBest"] < 0.75 else ""
|
||
out.append(f"{indent}- **{n['label']}: {n['choice']}**{conf(n)}{weak}. If chosen, then best play after: {alts}")
|
||
walk(n["next"], out, indent)
|
||
elif n["type"] == "chance":
|
||
br = sorted(n["branches"].items(), key=lambda kv: -kv[1]["p"])
|
||
live = [(k, b) for k, b in br if k != "die"]
|
||
die = next((b for k, b in br if k == "die"), None)
|
||
if die is not None:
|
||
out.append(f"{indent}- {n['label']}: {pct(1 - die['p'])} survive")
|
||
for _, b in live:
|
||
walk(b["next"], out, indent)
|
||
return
|
||
if len(live) == 1:
|
||
return walk(live[0][1]["next"], out, indent)
|
||
out.append(f"{indent}- {n['label']}: " + ", ".join(f"{k} {pct(b['p'])}" for k, b in br))
|
||
for k, b in br:
|
||
out.append(f"{indent} - if **{k}**:")
|
||
walk(b["next"], out, indent + " ")
|
||
elif n.get("outcomes"):
|
||
items = sorted(n["outcomes"].items(), key=lambda kv: -kv[1])
|
||
out.append(f"{indent}- Outcome if you attempt: " + ", ".join(f"{'incinerated' if k == 'burn' else 'bond ' + k} {pct(p)}" for k, p in items))
|
||
else:
|
||
out.append(f"{indent}- {n['what']}")
|
||
|
||
|
||
def main():
|
||
P = json.load(open(ROOT / "data/public/policy.json", encoding="utf-8"))
|
||
s, lb = P["samples"]["all"], P["samples"]["lowBias"]
|
||
L = ["# Best routes by dragon colour", ""]
|
||
L += [f"Generated from {s['data']['runs']} community runs by `planner/build_policy.py` from the game in `core/spec/game.yaml`. Machine-readable version: `data/public/policy.json`.", ""]
|
||
L += ["**How to read this.** \"This attempt\" maximises the chance of bonding the colour on one run. \"Eventually\" assumes you will die on purpose and try again whenever the run goes wrong, so it avoids any choice that can end the account on a different dragon. Percentages are posterior means with 95% credible intervals. \"Held-out\" is the same policy chosen on half of the runs and scored on the other half: when it is much lower than the in-sample number, the route's advantage is mostly noise.", ""]
|
||
L += ["**Caveats.** The form records choices only at the level of company, direction, first dragon, cadet encounter, fork, tree or valley, and which dragon you met. Choices inside each encounter are not recorded, so the bond chance at an encounter is the community's average with whatever answers people used. Riders chasing a rare dragon report sightings more often than riders who log every run; the \"low-bias\" columns use only bulk logs and non-chasers.", ""]
|
||
L += ["| Colour | This attempt | Low-bias sample | Held-out | Eventually: chance the account ends on this colour | Attempts needed (about) |",
|
||
"|---|---|---|---|---|---|"]
|
||
for X, t in s["targets"].items():
|
||
a, e, l = t["thisAttempt"], t["eventually"], lb["targets"][X]["thisAttempt"]
|
||
att = e["expectedAttempts"]["mean"]
|
||
L.append(f"| {X} | {pct(a['pX']['mean'])} ({pct(a['pX']['lo'])}–{pct(a['pX']['hi'])}) | {pct(l['pX']['mean'])} ({pct(l['pX']['lo'])}–{pct(l['pX']['hi'])}) | {pct(t['crossFit']['heldOut'])} | {pct(e['ratio']['mean'])} | {'–' if att is None else js_round(att)} |")
|
||
L += ["", "## First attempts", ""]
|
||
L += ["The shadow at the tree is close to a first-attempt event, so an account's first attempt is solved separately (`givens` in `core/spec/game.yaml`). Only the colours that can come from the shadow differ much.", ""]
|
||
L += ["| Colour | First attempt | Low-bias sample | Later attempt |", "|---|---|---|---|"]
|
||
for X, t in s["targets"].items():
|
||
f, fl, a = t["firstAttempt"]["thisAttempt"]["pX"], lb["targets"][X]["firstAttempt"]["thisAttempt"]["pX"], t["thisAttempt"]["pX"]
|
||
L.append(f"| {X} | {pct(f['mean'])} ({pct(f['lo'])}–{pct(f['hi'])}) | {pct(fl['mean'])} ({pct(fl['lo'])}–{pct(fl['hi'])}) | {pct(a['mean'])} ({pct(a['lo'])}–{pct(a['hi'])}) |")
|
||
sh = (s["model"].get("reportingBias") or {}).get("form, chasing Blue/Black", {}).get("shadow", {})
|
||
if sh.get("used", "").startswith("assumed"):
|
||
L += ["", f"The shadow's reporting bias cannot be measured yet (complete logs have never seen one), so it is assumed equal to the meadow Blue's (`borrow` in `planner/policy.yaml`): form reports of a shadow count 1/{sh['rho']:.1f} each. The bot's first-attempt runs will replace this assumption."]
|
||
L += ["", "## How much do earlier choices matter?", ""]
|
||
L += ["Each chance event can depend on the choices before it. The model learns how much from the data (empirical-Bayes concentration; larger means the history matters less):", ""]
|
||
L += ["| Chance event | Depends on (most important first) | Concentration |", "|---|---|---|"]
|
||
for k, ctx in s["model"]["chanceContext"].items():
|
||
L.append(f"| {k} | {', '.join(ctx) or 'nothing'} | {s['model']['alphas'][k]} |")
|
||
for X in s["targets"]:
|
||
for obj, title in (("thisAttempt", "this attempt"), ("eventually", "eventually")):
|
||
out = []
|
||
walk(s["targets"][X][obj]["tree"], out)
|
||
L += ["", f"## {X}: {title}", "", *out]
|
||
path = ROOT / "analysis/reports/POLICY.md"
|
||
path.write_text("\n".join(L) + "\n", encoding="utf-8")
|
||
print(f"wrote {path.relative_to(ROOT)} {len(L)} lines")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|