Data & Methods: explain how the bot's fallback picks are sampled #36

Open
opened 2026-11-19 13:44:29 +00:00 by kathleen · 0 comments
Owner

Follow-up to #35 (bot PR DragonKind/DragonKind#4). Once the fallback is live, the Data & Methods section should say how bot runs pick their choices on screens that nothing else covers. Those picks are part of the data behind the stats.

Points to cover:

  • Order of precedence:
    • the Account Lock Guard and deliberate pauses come first;
    • then the PDF guide and the atlas policy;
    • then known answers (the guide's encounter answers plus the bot's own results);
    • then Clef;
    • ntfy gets 2 minutes to override any pick except a clear known answer.
  • Goal accounts play the single best safe choice (propensity 1). The data from these choices is observational.
  • Research accounts sample 70% from Clef's probabilities and 30% uniformly over the safe options. Each step logs the exact sampling probability (propensity). That makes these choices randomized, and it is what lets us weight them in the estimates.
  • How the atlas uses propensity: which runs count as randomized, and how propensity-null (AI or manual) picks are handled.
  • Rule names as they appear in the data: KNOWN_ANSWER, KNOWN_ANSWER_WEAK, CLEF_GOAL, CLEF_EXPLORE, UNIFORM_EXPLORE.
  • Credit Clef (ollama.com/library/clef) and the community route PDF, which is the source of the known answers.

Not urgent: wait until the bot PR is merged and some runs with these rules have come in.

🤖 Generated with Claude Code

Follow-up to #35 (bot PR DragonKind/DragonKind#4). Once the fallback is live, the Data & Methods section should say how bot runs pick their choices on screens that nothing else covers. Those picks are part of the data behind the stats. Points to cover: - **Order of precedence**: - the Account Lock Guard and deliberate pauses come first; - then the PDF guide and the atlas policy; - then known answers (the guide's encounter answers plus the bot's own results); - then Clef; - ntfy gets 2 minutes to override any pick except a clear known answer. - **Goal accounts** play the single best safe choice (propensity 1). The data from these choices is observational. - **Research accounts** sample 70% from Clef's probabilities and 30% uniformly over the safe options. Each step logs the exact sampling probability (`propensity`). That makes these choices randomized, and it is what lets us weight them in the estimates. - How the atlas uses propensity: which runs count as randomized, and how propensity-null (AI or manual) picks are handled. - Rule names as they appear in the data: `KNOWN_ANSWER`, `KNOWN_ANSWER_WEAK`, `CLEF_GOAL`, `CLEF_EXPLORE`, `UNIFORM_EXPLORE`. - Credit Clef (ollama.com/library/clef) and the community route PDF, which is the source of the known answers. Not urgent: wait until the bot PR is merged and some runs with these rules have come in. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
kathleen added the site label 2026-11-19 13:44:29 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DragonKind/threshing-atlas#36