The loudest signal may describe a consequence.
Database latency becomes the strongest visible signal after users retry incorrect quotes. The spike is operationally real, but its temporal order challenges the theory that it caused the first wrong quote. An investigation needs a timeline of effects and interventions, not only a ranked list of abnormal metrics. A retry storm can also become a contributing cause of additional failures, so keep onset and later amplification distinct.
HTTP 200 measures a transport-level outcome. It cannot guarantee that currency, price freshness, entitlement, or response meaning satisfies the user contract. Define black-box checks that compare expected and actual behavior at the application boundary. Green infrastructure dashboards can coexist with a broken experience when the dashboard never measures the relevant semantic invariant.
Partition the incident before choosing a single story.
In the advanced fixture, EU requests receive a wrong cached currency and APAC requests receive stale prices during supplier throttling. The cache mechanism explains one cohort; it does not explain the other. Group observations by user contract, region, dependency path, version, and onset before deciding whether one mechanism covers the incident.
The engine keeps per-hypothesis verification distinct from incident closure. After the cache replay passes, the incident remains open until the APAC supplier mechanism also passes its gate. This blocks the tempting shortcut of declaring an incident solved because one elegant explanation is already verified.
Time and sampling can hide intermittent mechanisms.
A healthy sample outside the affected interval does not eliminate a transient failure. For intermittent incidents, record sampling coverage, expected occurrence rate, and how the observation overlaps the affected cohort. A delayed effect may arise from TTLs, queues, eventual propagation, scheduled tasks, or cached configuration. “The change was earlier” can be compatible with the mechanism rather than evidence against it.
Partial outages often disappear inside aggregate averages. Compare affected and unaffected cohorts and preserve negative controls. A system-wide success rate can remain high when a small region fails, while a broad metric spike can lead investigators away from the specific request path.
Report layered causes without overstating certainty.
A useful account distinguishes the triggering mechanism, contributing conditions, downstream amplification, and evidence gaps. Multiple faults can coexist without a single shared root. Keep causal statements scoped, and state what would be needed to link separate failures. In the lab, verify the cache mechanism first, inspect the remaining scope gap, and continue toward the supplier route replay.
LLMs make uncertain judgments. Software controls state, authority, verification, and stopping.