Why Investigation Is a Decision Problem

A successful request can still be a failed experience.

At 09:04, customers report that a quote displays the wrong currency. The service returns HTTP 200, availability remains high, and the database dashboard turns red four minutes later. An investigator now has three different questions: what the user experienced, which mechanism produced it, and which observation would distinguish that mechanism from alternatives. Reading the dashboard provides inputs; it does not select the next useful action.

An observation is a recorded fact with a source and a time window, such as “request q-17 returned USD for an EUR quote.” A symptom describes an experienced deviation, such as incorrect totals or slow checkout. A hypothesis proposes a mechanism that could explain the deviation. A root-cause conclusion identifies a mechanism within a stated scope and carries the evidence that justifies the claim. The same log line can support one hypothesis and contradict another; its role depends on the claim being tested.

Every investigation step changes the next decision.

Production investigation is sequential because the value of a check depends on what is already known. A trace showing a cache hit makes a cache-key comparison useful. A healthy database sample collected outside the affected interval may leave the database hypothesis untouched. Treat the current state as the observations, candidate mechanisms, unresolved gaps, permissions, and remaining budget available at this moment.

A strong model can still choose a redundant query, eliminate a cause using stale telemetry, or accept a convincing explanation before testing it. These are local decision failures. Once stored as authoritative state, they influence subsequent checks and reports. If a chain required ten independently correct decisions, each with a 95% success probability, its all-correct probability would be about 60%; actual failures can be correlated, so this arithmetic is only an illustration of compounding risk.

Make uncertainty visible before it becomes a conclusion.

Write the next action as a testable decision: “compare the key generated for the same SKU in two currencies,” rather than “look deeper into the cache.” Record what each possible outcome would change. A collision supports the omitted-currency mechanism; distinct keys weaken that particular mechanism. Neither observation authorizes a production configuration change.

During an active incident, the commander can contain damage while a separate workflow investigates. Preserve snapshots before a mitigation changes the system, and record that intervention as a possible confounder. Diagnosis, mitigation, and prevention have different authority and evidence requirements. Keeping them explicit helps the investigator avoid interpreting recovery as automatic proof of a cause.

A plausible story is a candidate for testing.

“The deployment caused the incident” fits the timeline, but a shared configuration can break both the old and new application builds. A stronger conclusion would bind the claim to the affected cohort, demonstrate the proposed failure mechanism, and show what changes when that mechanism alone is altered in a controlled replay. Production certainty is always limited by sampling and assumptions; VERIFIED in this series means that a named gate passed within the declared fixture scope.