Choosing the Next Best Check

Ask what would change after the check.

Another page of logs may confirm that users are affected without separating any causal theories. A useful diagnostic action produces outcomes that distinguish competing hypotheses. Before calling a tool, write what each possible result would do to the candidate set. If all candidates predict the same result, the check may measure impact but has little diagnostic discrimination.

For four equally weighted hypotheses, uncertainty is two bits. A perfect test that separates them into two equal groups reduces uncertainty by one bit. A perfect test that uniquely identifies every candidate provides two bits. These figures depend on the assumed outcome model. Real checks can be noisy, incomplete, correlated, and missing, so the expected value should include those possibilities.

Information gain is a model of expected evidence.

Expected information gain is the current entropy minus the weighted entropy after possible check outcomes. It compares how much a check is expected to reduce uncertainty. You need candidate weights and predicted outcomes to compute it. If those estimates are unavailable, use a documented heuristic such as candidate relevance and verification prerequisites, and avoid presenting a heuristic score as a calibrated probability.

The lab uses a deliberately simple, noiseless outcome partition so you can inspect the calculation. Change the candidate weights and watch how the best choice changes. The capstone policy uses a separate explicit heuristic because its observations do not carry calibrated likelihoods.

Expected gain = H(current hypotheses)
              − sum over outcomes P(outcome) × H(hypotheses | outcome)

A teaching utility = expected gain / synthetic cost units

Cost includes more than the price of a query.

A cheap metadata read may be more valuable than an expensive replay early in the investigation. Once a specific mechanism has strong support, the replay may become the only check that can satisfy verification. Query latency, rate limits, data movement, privacy exposure, and operational risk all influence selection. A high gain-per-cost ratio is a useful teaching rule, not a universal production objective.

Permissions and prerequisites are hard constraints. Remove prohibited or unaffordable checks before ranking. Avoid collecting the same snapshot repeatedly under slightly different prompts. A fresh sample can be legitimate when the system changed, but the new time window and collection intent must be recorded as a different evidence request.

Budget the investigation before it begins.

Set limits for decisions, collector calls, model calls, synthetic cost, and elapsed time. Reserve the estimated cost before dispatch, enforce the runtime deadline, and preserve budget exhaustion as an unresolved outcome. Budget pressure should never lower the verification threshold. When another check is unaffordable, the system can still evaluate evidence already collected and hand off the remaining gaps.