Evaluating Investigation Systems

A correct final answer can conceal an unreliable process.

An investigator may name the right cause after skipping contradictory evidence, leaking tool authority, or relying on an unsourced guess. A final-answer metric would reward that run. Evaluate the process by recording the candidates considered, checks selected, eliminations, source quality, verification attempts, budget usage, and unresolved handoff. Unsafe verification deserves a separate metric because it can make a convincing report operationally dangerous.

Ground truth belongs to the evaluator, outside the policy and model context. The fixtures specify which mechanisms are true and which incidents have enough available evidence for closure. The engine sees only authored observations and contracts; it does not import the evaluator’s answer map.

Define every denominator.

Hypothesis recall measures inclusion of true mechanisms. Initial recall and final recall after expansion answer different questions. Hypothesis precision here uses the supported or verified shortlist; it does not score every early possibility as a mistake. False elimination divides true mechanisms incorrectly falsified by the number of true mechanisms. False verification divides incorrectly verified mechanisms by verified mechanisms, with zero when none are verified.

Evidence provenance coverage measures the fraction of ledger entries that retain required source fields. This is a schema metric, not proof that the source is authentic or the interpretation correct. Tool-call efficiency in this harness measures unique collected checks per collector call; it catches duplication but does not claim that every unique check was useful.

Make deterministic evaluation a controlled experiment.

Pin fixture and policy versions, input snapshots, tool observations, recorded judgments, tie-breaking rules, and metric definitions. The fixture judge uses stable ordering, and replay consumes recorded model decisions without calling a provider. A fresh live model run may choose different checks even with similar decoding settings; evaluate its repeated outcomes separately from deterministic replay.

Measure virtual fixture latency and cost for the offline suite, and mark them clearly. A production benchmark needs actual collector latency, model usage and price assumptions, sampling policies, and repeated noisy incidents. Four authored scenarios are useful regression checks but cannot estimate general production reliability.

Reward a useful unresolved outcome.

The insufficient-evidence fixture intentionally lacks a matching replay window. Correct behavior is escalation with a supported account, evidence references, a named owner, and unresolved scope. Closing that fixture would be a false incident-level verification even if the hypothesized cache mechanism happened to be the hidden truth.

Run the suite below and inspect initial recall in the missing-cause case, complete scope coverage in the two-fault case, and abstention quality in the insufficient case. The accompanying test harness also attacks malformed judgments, repeated checks, budget exhaustion, stale observations, premature verification, deadlines, and tampered replay reports.