# From Agents to Investigation Systems

A practical series on designing AI systems that investigate incidents through bounded decisions, explicit evidence, deterministic controls, and verifiable stopping conditions.

LLMs make uncertain judgments. Software controls state, authority, verification, and stopping.

## Lesson 01 — Why Investigation Is a Decision Problem

Turn an incident narrative into observations, testable hypotheses, and a sequence of decisions whose errors can be contained.

Time: 35–45 min. Artifact: Incident reasoning ledger.

### A successful request can still be a failed experience.

At 09:04, customers report that a quote displays the wrong currency. The service returns HTTP 200, availability remains high, and the database dashboard turns red four minutes later. An investigator now has three different questions: what the user experienced, which mechanism produced it, and which observation would distinguish that mechanism from alternatives. Reading the dashboard provides inputs; it does not select the next useful action.

An observation is a recorded fact with a source and a time window, such as “request q-17 returned USD for an EUR quote.” A symptom describes an experienced deviation, such as incorrect totals or slow checkout. A hypothesis proposes a mechanism that could explain the deviation. A root-cause conclusion identifies a mechanism within a stated scope and carries the evidence that justifies the claim. The same log line can support one hypothesis and contradict another; its role depends on the claim being tested.

### Every investigation step changes the next decision.

Production investigation is sequential because the value of a check depends on what is already known. A trace showing a cache hit makes a cache-key comparison useful. A healthy database sample collected outside the affected interval may leave the database hypothesis untouched. Treat the current state as the observations, candidate mechanisms, unresolved gaps, permissions, and remaining budget available at this moment.

A strong model can still choose a redundant query, eliminate a cause using stale telemetry, or accept a convincing explanation before testing it. These are local decision failures. Once stored as authoritative state, they influence subsequent checks and reports. If a chain required ten independently correct decisions, each with a 95% success probability, its all-correct probability would be about 60%; actual failures can be correlated, so this arithmetic is only an illustration of compounding risk.

### Make uncertainty visible before it becomes a conclusion.

Write the next action as a testable decision: “compare the key generated for the same SKU in two currencies,” rather than “look deeper into the cache.” Record what each possible outcome would change. A collision supports the omitted-currency mechanism; distinct keys weaken that particular mechanism. Neither observation authorizes a production configuration change.

During an active incident, the commander can contain damage while a separate workflow investigates. Preserve snapshots before a mitigation changes the system, and record that intervention as a possible confounder. Diagnosis, mitigation, and prevention have different authority and evidence requirements. Keeping them explicit helps the investigator avoid interpreting recovery as automatic proof of a cause.

### A plausible story is a candidate for testing.

“The deployment caused the incident” fits the timeline, but a shared configuration can break both the old and new application builds. A stronger conclusion would bind the claim to the affected cohort, demonstrate the proposed failure mechanism, and show what changes when that mechanism alone is altered in a controlled replay. Production certainty is always limited by sampling and assumptions; VERIFIED in this series means that a named gate passed within the declared fixture scope.

### Lab — Classify the incident timeline

Each statement is written in a particular role. Classify it as used here, then inspect why that role limits the claim.

- 09:00 — The deployment log records quote binary v47 becoming active.
  - Model answer: Observation. A recorded deployment time describes a change; it does not establish causal responsibility.
- 09:04 — A client probe records HTTP 200 and currency USD for an EUR request.
  - Model answer: Observation. The probe records a semantic contract violation and preserves the expected and actual values.
- The new quote binary may have changed arithmetic for EU requests.
  - Model answer: Hypothesis. This proposes a specific mechanism. It predicts that old and new builds behave differently with otherwise identical inputs.
- Against the cache-key hypothesis, two different currencies produce the same key.
  - Model answer: Supporting evidence. This observation supports the particular collision mechanism; it still needs provenance and a scoped replay.
- Database waits rose after incorrect quotes triggered retries.
  - Model answer: Observation. Temporal order is recorded here. A later reasoning step may use it to challenge a database-onset hypothesis.
- The controlled replay identifies an omitted currency dimension as the cause of incorrect EU cached quotes.
  - Model answer: Conclusion. This is a scoped causal claim tied to a controlled replay, not a statement about every incident symptom.

Design record:
- Which user contract failed, and in which cohort?
- What observation would most directly challenge your leading explanation?

## Lesson 02 — Building the Hypothesis Set

Create a candidate space that covers services, dependencies, infrastructure, and changes without hiding the possibility of a missing cause.

Time: 40–50 min. Artifact: Hypothesis tree and prediction table.

### The correct choice needs a place in the candidate set.

An investigator can rank every available hypothesis perfectly and still fail when the causal mechanism is absent. Begin with the affected user contract and a scoped dependency graph. For incorrect quotes, candidate domains include application arithmetic, database inputs, supplier responses, cache behavior, network delivery, and effective configuration. A recent deployment is one branch of that tree, rather than the definition of the whole search space.

Make leaves concrete enough to predict an observation. “The database is broken” does not distinguish a stale replica, lock contention, or an incorrect query. “The quote reads a replica that is twenty minutes behind” predicts a version difference between primary and replica for the same record. Keep the scope, proposed mechanism, expected observation, and falsifying observation together.

### Coverage and precision create different engineering pressures.

Coverage asks whether the candidate set includes the causal mechanisms needed to explain the incident. Precision asks how much of the candidate space is useful. A huge list can hide a real cause under dozens of irrelevant theories, while a tidy list of three popular failure modes can omit a configuration change. Start with a small, diverse set, record why each candidate is in scope, and reserve an explicit expansion path.

Do not assign the absence of a candidate the meaning “impossible.” In this track, a service-catalog check can expose a separately managed cache dependency. The engine then permits adding the cache hypothesis. This is a limited teaching mechanism with an authored registry; a production expansion process needs topology ownership, scoped discovery permissions, and validation of any newly proposed diagnostic contract.

### Build a tree that preserves useful distinctions.

Use the top level for failure domains and the leaves for mechanisms. Put the new-binary hypothesis under service changes; put omitted key dimensions under configuration; put supplier throttling under an external dependency. A change can cross domains, so the tree is an organizing aid rather than proof of independence. If two leaves describe the same mechanism, merge them before they compete for budget.

Attach expected evidence to every leaf. This makes candidate generation reviewable and gives the check selector a discriminative target. When all surviving candidates predict something different from the user experience, mark candidate coverage as suspect. Revisit topology, recent effective configuration, and the original contract before inventing a more elaborate story.

### Preserve the boundary between gaps and explanations.

A known unknown is a gap you can name, such as an unavailable configuration history. An unknown unknown becomes visible when an unexpected observation exposes a missing dependency or failure mode. Record both without assigning invented probabilities. Your hypothesis tree should include an “unexplained / expand” condition so that elimination of current candidates leads to discovery or escalation instead of forced selection.

| Domain | Mechanism | Discriminating prediction |
| --- | --- | --- |
| Service / change | New binary changes arithmetic | Only the new build fails a matched replay |
| Dependency | Database causes initial incorrect totals | The relevant database failure precedes wrong quotes |
| Infrastructure | Packet loss corrupts this path | Scoped network evidence and a replay match impact |
| External dependency | Supplier throttles APAC prices | Current APAC calls and a route replay reproduce failure |
| Configuration | Key omits currency | Different currencies collide and a key-only change fixes the replay |

### Lab — Build a hypothesis tree from five signal families

For each leaf, choose its domain and its first discriminating check. The generated tree preserves your predictions, including an explicit unexplained branch.

- A new quote binary deployed near incident onset.
  - Model answer: Service / change; Compare old and new builds. The deployment is a candidate trigger. Hold configuration and inputs constant while comparing builds.
- Database wait time rose after user retries began.
  - Model answer: Dependency; Inspect pre-impact database waits. Check temporal precedence before treating a downstream load signal as the initial cause.
- One network zone may be unhealthy.
  - Model answer: Infrastructure; Inspect the affected request path. Global host averages do not describe a scoped request path.
- APAC supplier calls may return throttling responses.
  - Model answer: External dependency; Sample current APAC supplier responses. Bind the sample to the affected cohort and interval.
- Quote caching has its own configuration owner.
  - Model answer: Configuration; Compare effective cache key dimensions. Effective configuration can differ from repository defaults or deployment metadata.

Design record:
- What failure domain could your first candidate set have missed?
- Give one scoped prediction for a hypothesis you would add.

## Lesson 03 — Bounded Decisions Instead of Open-Ended Agents

Split investigation into small judgments with typed inputs, permitted choices, and software-owned transitions.

Time: 40–50 min. Artifact: Five bounded decision contracts.

### Define the decision before writing the prompt.

“Investigate this incident” lets a model choose the goal interpretation, queries, tools, memory updates, and ending condition together. Each of those choices creates a separate failure surface. A bounded decision supplies a limited question, the relevant evidence, and an allowed output space. The model can choose a permitted diagnostic check; ordinary software validates the choice and resolves that ID to a trusted collector.

Five useful judgments are candidate proposal, evidence classification, check ranking, next-check choice, and evidence-gap assessment. Keep their inputs narrow. A classifier needs an evidence item and a hypothesis prediction; a ranker needs permitted checks, expected outcomes, and cost. Neither needs production credentials or a generic shell tool.

### Structured output exposes the proposed action.

A JSON envelope lets software reject unknown fields, unavailable IDs, duplicate rankings, and unsupported judgment kinds. It does not make the reasoning correct. Keep the returned rationale bounded, bind evidence references to the current ledger, and validate the permitted action again immediately before collection. A model can propose a check that was available in an old snapshot but has already been completed.

The teaching engine accepts choose_check, rank_checks, classify_evidence, and discovered-candidate expansion. Initial hypotheses come from the authored scenario. Invalid proposals consume decision budget, so repeated schema failures cannot create an infinite free retry loop. Rankings influence selection among available checks; they cannot introduce a new tool or override prerequisites.

```javascript
{
  "kind": "choose_check",
  "checkId": "cache-keys",
  "reason": "Different currencies may collide under the same effective key."
}
```

### Reserve authority for deterministic code.

A classification proposal belongs in a judgment log. It must not directly set a hypothesis to FALSIFIED or VERIFIED. In this series, authored predicates interpret trusted synthetic observations; the independent gate reads those observations rather than trusting a model-written status. A production system would need validated parsers and service-specific diagnostic contracts at the same boundary.

A yes/no model assessment such as “does this observation support this claim?” remains uncertain. The authorization decision “may this tool run?” uses explicit permissions and budgets. The transition “may this incident close?” uses named verification requirements. This separation allows model errors to remain visible without becoming execution authority.

### Keep transitions small enough to replay.

Record the proposed judgment, whether validation accepted it, and the observation produced by the collector. The next view is reconstructed from that ledger. Replay should not ask a model to reason again; it should consume the recorded decisions and reproduce the deterministic control behavior. Version the fixture, policy, schema, and tool contracts so that a later software change cannot silently reinterpret an old report.

### Lab — Convert one instruction into five bounded decisions

Choose the appropriate output contract for each step. This assembles a workflow whose model outputs can be checked before they influence execution.

- Generate possible mechanisms before selecting a diagnostic path.
  - Model answer: Candidate IDs + predicted observations. Candidate proposal should expose the mechanism and predictions, then pass through a registry or review boundary.
- Compare the usefulness of the currently permitted checks.
  - Model answer: Ranked permitted check IDs. Ranking is a bounded preference among existing capabilities.
- Choose what the collector should inspect next.
  - Model answer: One permitted check ID. Software resolves the ID, rechecks prerequisites, and enforces budgets.
- Assess whether a recorded key collision supports the cache hypothesis.
  - Model answer: Evidence ID + hypothesis ID + relation. A judgment references evidence already collected; it does not invent a collector result.
- Assess whether meaningful uncertainty remains.
  - Model answer: Evidence gap assessment for a software-owned gate. The model can describe gaps. Software decides whether verification, continuation, or escalation is allowed.

Design record:
- List the inputs and valid outputs for one decision in your system.
- Which fields must the model never be allowed to write?

## Lesson 04 — Choosing the Next Best Check

Compare diagnostic checks by the hypotheses they separate, the evidence they can change, and their cost.

Time: 45–55 min. Artifact: Cost-aware check-selection record.

### Ask what would change after the check.

Another page of logs may confirm that users are affected without separating any causal theories. A useful diagnostic action produces outcomes that distinguish competing hypotheses. Before calling a tool, write what each possible result would do to the candidate set. If all candidates predict the same result, the check may measure impact but has little diagnostic discrimination.

For four equally weighted hypotheses, uncertainty is two bits. A perfect test that separates them into two equal groups reduces uncertainty by one bit. A perfect test that uniquely identifies every candidate provides two bits. These figures depend on the assumed outcome model. Real checks can be noisy, incomplete, correlated, and missing, so the expected value should include those possibilities.

### Information gain is a model of expected evidence.

Expected information gain is the current entropy minus the weighted entropy after possible check outcomes. It compares how much a check is expected to reduce uncertainty. You need candidate weights and predicted outcomes to compute it. If those estimates are unavailable, use a documented heuristic such as candidate relevance and verification prerequisites, and avoid presenting a heuristic score as a calibrated probability.

The lab uses a deliberately simple, noiseless outcome partition so you can inspect the calculation. Change the candidate weights and watch how the best choice changes. The capstone policy uses a separate explicit heuristic because its observations do not carry calibrated likelihoods.

```javascript
Expected gain = H(current hypotheses)
              − sum over outcomes P(outcome) × H(hypotheses | outcome)

A teaching utility = expected gain / synthetic cost units
```

### Cost includes more than the price of a query.

A cheap metadata read may be more valuable than an expensive replay early in the investigation. Once a specific mechanism has strong support, the replay may become the only check that can satisfy verification. Query latency, rate limits, data movement, privacy exposure, and operational risk all influence selection. A high gain-per-cost ratio is a useful teaching rule, not a universal production objective.

Permissions and prerequisites are hard constraints. Remove prohibited or unaffordable checks before ranking. Avoid collecting the same snapshot repeatedly under slightly different prompts. A fresh sample can be legitimate when the system changed, but the new time window and collection intent must be recorded as a different evidence request.

### Budget the investigation before it begins.

Set limits for decisions, collector calls, model calls, synthetic cost, and elapsed time. Reserve the estimated cost before dispatch, enforce the runtime deadline, and preserve budget exhaustion as an unresolved outcome. Budget pressure should never lower the verification threshold. When another check is unaffordable, the system can still evaluate evidence already collected and hand off the remaining gaps.

| Check | Predicted distinction | Cost units |
| --- | --- | --- |
| Global log count | All four candidates predict the same result | 1 |
| Scoped path split | Two equal groups | 2 |
| Isolated mechanism replay | Four distinct outcomes | 8 |
| Configuration contrast | One candidate versus three | 1 |

### Lab — Choose a discriminating check for four hypotheses

Adjust the relative weights, then choose the check with the highest expected information gain per cost unit under this explicit noiseless model.

Design record:
- What assumptions make your information-gain calculation meaningful?
- When would a lower gain-per-cost check be necessary for verification?

## Lesson 05 — Evidence, Support, and Falsification

Preserve provenance and identify what would support, contradict, or leave a causal claim unchanged.

Time: 40–50 min. Artifact: Support and falsification matrix.

### Evidence has meaning relative to a prediction.

A cache hit is ordinary behavior for a cache-backed application. A cache hit carrying a payload from a different currency is evidence for a specific key-collision mechanism. The strength of the evidence depends on source quality, collection scope, timing, and whether competing mechanisms also predict it. Preserve the observed value separately from the relationship an investigator assigns to it.

Supporting evidence raises the plausibility of a mechanism. Contradictory evidence challenges a prediction that the mechanism requires. Neutral evidence does not materially discriminate under the stated assumptions. Missing logs are usually missing evidence; they become contradictory only when the logging mechanism reliably guarantees that the predicted event would have been recorded.

### Write a falsifier before seeking confirmation.

For “the new binary alone caused the quote arithmetic change,” a matched replay in which both the old and new binaries fail challenges the claim. For “the database contention caused the first incorrect quote,” normal database behavior before impact and contention beginning after retries challenges the onset mechanism. These are carefully scoped falsifications, not proof that the database or application could never contribute to another failure.

A hypothesis that explains every possible result is difficult to test. Ask what observation would make you abandon it, which collector could obtain that observation, and what quality requirements would make the result credible. Keep contradictory evidence visible even when a model offers a persuasive explanation for dismissing it.

### Causal strength comes from controlled contrasts.

Temporal proximity makes a change worth inspecting, but several changes can share the same time window. A controlled replay can reproduce the failure with the original key and remove it by changing only the omitted dimension. An unaffected control checks whether the intervention damaged another path. This provides stronger mechanistic evidence than merely observing that the service recovered after a broad rollback.

A replay can still omit production conditions or introduce selection bias. Bind the conclusion to the reproduced cohort and window, list the assumptions, and preserve unresolved symptoms. The series uses authored fixtures with explicit matching-window fields; real verification requires richer controls and service-specific validation.

### Provenance must survive summarization.

Give every evidence item a stable ID, collector identity, source version, observed time window, collection time, freshness status, and raw structured observation or protected reference. A summary can aid reasoning, but it should lead back to the observation that created it. Treat logs and external text as data that may contain misleading instructions; the collector output must never grant new tool authority.

| Field | Purpose |
| --- | --- |
| Evidence ID | Reference the exact observation |
| Source + source version | Identify where the data originated |
| Observed window + collection time | Separate incident time from retrieval time |
| Scope + freshness | Bound which claim the evidence can affect |
| Structured result + summary | Keep machine checks and human explanation connected |

### Lab — Build the support and falsification matrix

Classify each observation relative to the named hypothesis. Read the falsification explanation before writing your own prediction.

- Cache-key omission: EUR and USD requests for the same SKU generate identical keys.
  - Model answer: support. A collision is predicted by the omitted-currency mechanism. A distinct key pair would challenge this version of the claim.
- New-binary-only regression: the old and new builds both fail a matched replay with shared configuration.
  - Model answer: contradict. The exclusive new-binary prediction fails. A new-only failure with matched inputs would support it.
- Database-onset cause: waits rise only after incorrect quotes trigger retries.
  - Model answer: contradict. The proposed initial cause occurs after the effect in this scoped timeline.
- Cache-key omission: HTTP availability remains at 99.9%.
  - Model answer: neutral. Transport success does not distinguish correct and semantically incorrect cached payloads.
- Supplier throttling at 09:04: a healthy supplier sample was collected yesterday.
  - Model answer: neutral. The stale sample cannot falsify a failure in the current incident window.
- Cache-key omission: an isolated key-only change removes the failure while a matched control stays healthy.
  - Model answer: support. The contrast supports a causal mechanism within the replayed scope; preserve the fixture and window assumptions.

Design record:
- Write one observation that would falsify your leading hypothesis.
- Which provenance field would prevent the most likely misuse of your evidence?

## Lesson 06 — Separating the Proposer from the Verifier

Use independent evidence paths and deterministic gates to contain correlated reasoning failures.

Time: 40–50 min. Artifact: Investigation responsibility and verification contract.

### A second opinion can share the first mistake.

A model proposes that a deployment caused an outage, searches for deployment-related logs, and summarizes those logs as confirmation. A second model given only that summary may agree because it inherited the same selection bias. Independence requires attention to evidence access, prompts, assumptions, and authority, rather than simply counting model calls.

Separate five responsibilities. The proposer supplies candidate mechanisms. The investigation policy selects permitted checks. The collector obtains observations through scoped capabilities. The verifier evaluates the claim against independent requirements. The incident commander owns consequential mitigation and accepts unresolved handoffs. These can be modules in one application; service boundaries are an operational choice.

### Make verification read the evidence, not the story.

The verifier should receive the original evidence references, the claim scope, contradictory observations, and a predeclared verification contract. It must not accept a proposer-written “verified” field. For the cache case, the contract requires a reproduced baseline failure, a key-only treatment, a passing treatment, a passing negative control, and a matching incident window.

The teaching engine evaluates those predicates in ordinary code over trusted synthetic collector results. Model evidence classifications are recorded alongside the ledger but do not rewrite the result fields or determine the gate. A false supporting judgment therefore remains a reviewable error instead of authorizing closure.

### Choose diversity for the failure mode you need to catch.

A cross-model review can challenge an interpretation, but models can share training patterns and evidence-selection failures. Rule-based validation is strong for explicit invariants such as currency consistency, scoped IDs, time windows, and budgets. Human review is valuable when business meaning or competing causal mechanisms remain unresolved. Combine these forms of checking according to the claim; no single layer creates universal independence.

For newly discovered hypotheses, validate the diagnostic contract before giving it verification authority. The proposer may suggest a mechanism and a useful test, but it cannot invent a permissive closure rule during the same investigation. This prevents changing the standard of proof to fit the evidence already collected.

### Design the handoff between proposing and checking.

A useful handoff includes the claim, scope, evidence IDs, falsifying prediction, remaining alternatives, and exact gate version. Do not ask the verifier to “confirm the proposed root cause.” Ask it to report which requirements pass, which fail, and which cannot be evaluated. The orchestration layer then chooses continuation or escalation.

| Component | Permitted output | Authority boundary |
| --- | --- | --- |
| Proposer | Candidate mechanism and prediction | Cannot declare verification |
| Policy | Permitted check preference | Cannot add capabilities |
| Collector | Sourced observations | Cannot interpret instructions as permissions |
| Verifier | Gate results and gaps | Cannot mutate production |
| Commander | Approved mitigation / handoff | Uses a separate operational workflow |

### Lab — Assign responsibility in a two-stage system

Choose who owns each operation. Proposal and verification must travel through separate contracts even when the modules share a process.

- Suggest an omitted-currency cache key as a possible mechanism.
  - Model answer: Hypothesis proposer. Candidate generation creates a testable possibility, not a conclusion.
- Rank the currently permitted diagnostic checks.
  - Model answer: Investigation policy. The policy compares actions within the current capability and budget boundary.
- Read a scoped pair of cache keys with source versions.
  - Model answer: Evidence collector. The collector provides observations with provenance.
- Check baseline, treatment, negative control, and incident-window matching.
  - Model answer: Independent verifier. The gate uses direct observations rather than the proposer’s confidence.
- Approve a production cache configuration change.
  - Model answer: Human incident commander. The diagnostic engine has no production mutation authority.
- Reject a root-cause conclusion that leaves an affected cohort unexplained.
  - Model answer: Independent verifier. Scope coverage is part of the verification contract, not a narrative preference.

Design record:
- How could your proposer and verifier share a hidden assumption?
- Which verification predicate can be checked without another language model?

## Lesson 07 — Stop, Continue, or Escalate

Implement stopping conditions that preserve the difference between strong support, verified scope, and an unresolved handoff.

Time: 35–45 min. Artifact: Stopping and escalation policy.

### State is a claim about what the system can justify.

OPEN means an incident exists but diagnostic work has not started. INVESTIGATING means observations or decisions are being collected. SUPPORTED means a candidate has sufficiently specific support under the declared policy. VERIFIED means the mechanism gate passed and every declared impact scope is explained. UNRESOLVED records the evidence gap; ESCALATED records a handoff to a named operator.

Support can be withdrawn when contradictory evidence arrives. A verified mechanism for EU quotes does not close an incident that also includes unexplained APAC prices. Keep per-hypothesis status separate from incident status. In the engine, an individual mechanism may pass while the incident remains SUPPORTED until coverage is complete.

### Define closure before seeing the answer.

The verification policy checks provenance, freshness, mechanism evidence, controlled contrast, and scope coverage. A model confidence score is not one of those predicates. If the gate fails, report the failing requirements. A manual “verify” request evaluates the gate and leaves the investigation open when the evidence is insufficient.

Budgets control continuation. Stop requesting more evidence when the permitted next check would exceed time, tool calls, or cost. Evaluate existing evidence first; a gate already satisfied can still close the diagnostic workflow. Otherwise preserve the best-supported account and escalate. Exhaustion must not convert support into verification.

### Abstention needs an operational next step.

An unresolved report should identify what is supported, what remains unexplained, which evidence was unavailable, who owns the next action, and which diagnostic request could close the gap. “The model is uncertain” gives an operator very little to work with. “The cache replay lacks the affected time window; the quote-platform owner must provide snapshot key-v12 and request pair q-17/q-18” is a practical handoff.

A system can escalate early when permitted checks cannot resolve a material gap. During severe impact, the commander may prioritize containment while diagnosis remains incomplete. These decisions belong to the response workflow; investigation evidence should inform them without silently granting the model mitigation authority.

### Another check should have a reason to exist.

Marginal value is the expected improvement from one more observation given the current state. It declines when remaining checks are redundant or cannot change any relevant decision. A cheap check may still be worthwhile if it challenges the leading mechanism, while repeated supportive samples may add little. Use explicit continuation criteria, and record why a run stops even when its final answer sounds convincing.

| Condition | Policy outcome |
| --- | --- |
| Fresh controlled evidence + all scopes covered | Verify |
| Strong support + missing mechanism evidence + useful affordable check | Continue |
| One verified mechanism + unexplained second cohort | Continue or escalate; incident stays open |
| Insufficient evidence + budget exhausted | Unresolved → escalated |
| No permitted check can close the gap | Unresolved → escalated |

### Lab — Implement the stopping table

Choose the policy outcome for each state snapshot. A high confidence score cannot substitute for a failed gate.

- Two sources support cache omission; a matching controlled replay is still available and affordable.
  - Model answer: Continue. Collect the mechanism evidence before closure.
- A cache replay passes, but the APAC price cohort remains unexplained and its supplier check is affordable.
  - Model answer: Continue. Per-mechanism verification does not close a wider incident.
- The model reports 99% confidence; the matching replay window is unavailable and no permitted check can retrieve it.
  - Model answer: Escalate. Confidence cannot replace unavailable incident evidence.
- Every scope has passing baseline, treatment, control, freshness, and provenance checks.
  - Model answer: Verify. Software can close the declared diagnostic scope using the predeclared gate.
- The next necessary check exceeds the remaining budget, and the verification gate still fails.
  - Model answer: Escalate. Preserve support, the exact gap, and an owner. Do not lower the verification standard.

Design record:
- State your stop condition as machine-checkable predicates.
- What exact evidence request and owner should appear in an unresolved handoff?

## Lesson 08 — Designing a Production-Grade Investigation Engine

Assemble state, evidence, permissions, budgets, and a bounded judge into one replayable diagnostic workflow.

Time: 55–75 min. Artifact: Runnable investigation engine and report.

### The engine owns the lifecycle.

The complete flow begins with a user-impact contract, gathers scoped signals, proposes candidates, selects a permitted check, collects evidence, updates hypotheses, and evaluates verification. Each iteration receives a new view of the same explicit state. The model sees what it needs for its bounded judgment; the engine resolves action IDs, reserves budget, and controls transitions.

This implementation uses native JavaScript modules that run in the browser and in Node. It has no database migration and no new frontend framework dependency. The site simulator uses synthetic collectors and a transparent fixture judge. The optional CLI model adapter can replace the judge while the same engine continues to own permissions and closure.

```javascript
const engine = new InvestigationEngine("cache");
await engine.requestJudgment(async (view, signal) => {
  // Call an authorized provider from a server-side adapter.
  // Return a bounded envelope; never execute a model-written command.
  return {kind:"choose_check", checkId:"contract",
          reason:"Identify the violated user contract first."};
});
const gate = engine.verify(); // evaluates evidence; does not force closure
```

### Store authoritative evidence separately from memory.

An evidence ledger preserves observations and provenance. Investigation memory can summarize which paths were explored and which checks failed, but it should point to ledger IDs. A compressed model summary must not overwrite raw observations or make eliminated alternatives disappear. The engine returns immutable snapshots; private state prevents the judge from setting status or editing collector results through the public interface.

A capability registry maps a known check ID to a typed collector. Prerequisites constrain when each collector is useful. This simulator allows reads and isolated synthetic replays. It contains no shell command, arbitrary URL fetch, deployment, or payment capability. Production credentials belong to collectors with separately scoped service permissions, outside model context.

### Audit both decisions and refusals.

The event log records accepted decisions, rejected proposals, evidence, verification attempts, and unresolved handoffs. Model calls have deadline and cost metering. Rejected decisions consume decision budget, while a repeated check cannot consume a second collector call. Recorded decisions let replay reconstruct the same fixture observations and gates without invoking a model again.

Replay consistency is different from source authenticity. Exported browser reports are unsigned training artifacts, and a consistent fabricated event history is not proof that a production collector actually ran. A deployed system would add authenticated source references, access controls, integrity protection, retention policy, and durable versioned storage.

### Observe the investigator as a production service.

Track rejected decisions, repeated-check attempts, candidate expansions, unexplained scopes, verifier refusals, model timeouts, budget consumption, and human handoff time. Connect these metrics to run IDs and policy versions. Keep high-cardinality incident details out of broad metric labels and redact sensitive payloads at the collector boundary.

Use the workbench below to step through the same flow that a model adapter would use. Request verification before and after a replay, attempt an unrestricted proposal, and export the event report. A denied proposal is a useful result: it demonstrates that uncertain judgment cannot expand the execution surface.

### Lab — Build and inspect the bounded engine

Step through collection, attempt early verification, run the fixture judge, and replay an exported report. The UI calls the same engine as the CLI and test harness.

Design record:
- Which collector and verification contracts would you replace for your own service?
- Which investigation metrics would reveal a recurring policy failure?

## Lesson 09 — When the Root Cause Is Missing

Detect candidate-set failure and expand through scoped topology and effective-change evidence.

Time: 45–60 min. Artifact: Hypothesis expansion record.

### Eliminating candidates can expose a broken search space.

The first candidate set includes a binary regression, database contention, supplier throttling, and packet loss. Each has a plausible relationship to the incident, but none explains the wrong currency on cache hits. Once predictions fail, selecting the least implausible survivor can turn an incomplete search into a confident mistake. Treat persistent unexplained observations as a signal that candidate coverage may have failed.

Candidate-set failure does not require eliminating every hypothesis. A strong observation can expose a domain absent from the tree while other theories remain open. Preserve those observations as gaps rather than forcing them into the nearest existing explanation.

### Expand along real dependencies.

A service topology read identifies a cache with a configuration owner separate from the application deployment. That observation justifies expanding the investigation to effective key dimensions and the configuration change log. A repository diff alone might miss the value actually running during impact. Record the discovered edge, owner, observed version, and time relationship before creating a new candidate.

Recent-change analysis should include runtime flags, infrastructure policy, cache behavior, dependency versions, network routing, credentials, and external providers. Search beyond the first visible blast radius through a bounded process: identify a relevant edge, obtain scoped read authority, inspect the effective state, and attach a falsifiable prediction.

### Expansion must preserve permission boundaries.

A model suggesting “inspect every service” does not authorize unrestricted discovery. The engine accepts expansion only for candidate IDs exposed by an approved topology observation. In a real environment, newly discovered services may require a human owner to authorize access and approve a diagnostic contract. Dynamic discovery and dynamic authority are separate changes.

Keep an expansion budget and avoid generating endlessly refined variations of an unsupported theory. If the topology is incomplete, the relevant configuration history is unavailable, or expansion would require prohibited data access, escalate with the missing artifact and owner.

### Measure recovery from a missing candidate.

Evaluation should record initial hypothesis recall, final recall after expansion, steps spent before discovery, and false elimination of true mechanisms. A final correct answer can conceal a weak discovery process if it arrived only after exhaustive unrelated checks. The lab starts without the cache candidate, then uses a sourced discovery event to add it and replay the final investigation.

### Lab — Recover from a deliberately incomplete candidate set

The cache hypothesis is absent initially. Collect trace and topology evidence, add only a discovered candidate, then pursue a scoped verification. Try adding cache before discovery to inspect the refusal.

Design record:
- What observation exposed the missing domain?
- What prevents hypothesis expansion from also expanding tool authority?

## Lesson 10 — Ambiguous and Misleading Incidents

Follow causal ordering and cohort boundaries through symptoms, partial outages, and simultaneous faults.

Time: 50–65 min. Artifact: Multi-cohort causal account.

### The loudest signal may describe a consequence.

Database latency becomes the strongest visible signal after users retry incorrect quotes. The spike is operationally real, but its temporal order challenges the theory that it caused the first wrong quote. An investigation needs a timeline of effects and interventions, not only a ranked list of abnormal metrics. A retry storm can also become a contributing cause of additional failures, so keep onset and later amplification distinct.

HTTP 200 measures a transport-level outcome. It cannot guarantee that currency, price freshness, entitlement, or response meaning satisfies the user contract. Define black-box checks that compare expected and actual behavior at the application boundary. Green infrastructure dashboards can coexist with a broken experience when the dashboard never measures the relevant semantic invariant.

### Partition the incident before choosing a single story.

In the advanced fixture, EU requests receive a wrong cached currency and APAC requests receive stale prices during supplier throttling. The cache mechanism explains one cohort; it does not explain the other. Group observations by user contract, region, dependency path, version, and onset before deciding whether one mechanism covers the incident.

The engine keeps per-hypothesis verification distinct from incident closure. After the cache replay passes, the incident remains open until the APAC supplier mechanism also passes its gate. This blocks the tempting shortcut of declaring an incident solved because one elegant explanation is already verified.

### Time and sampling can hide intermittent mechanisms.

A healthy sample outside the affected interval does not eliminate a transient failure. For intermittent incidents, record sampling coverage, expected occurrence rate, and how the observation overlaps the affected cohort. A delayed effect may arise from TTLs, queues, eventual propagation, scheduled tasks, or cached configuration. “The change was earlier” can be compatible with the mechanism rather than evidence against it.

Partial outages often disappear inside aggregate averages. Compare affected and unaffected cohorts and preserve negative controls. A system-wide success rate can remain high when a small region fails, while a broad metric spike can lead investigators away from the specific request path.

### Report layered causes without overstating certainty.

A useful account distinguishes the triggering mechanism, contributing conditions, downstream amplification, and evidence gaps. Multiple faults can coexist without a single shared root. Keep causal statements scoped, and state what would be needed to link separate failures. In the lab, verify the cache mechanism first, inspect the remaining scope gap, and continue toward the supplier route replay.

### Lab — Investigate two faults behind one incident banner

Explain EU currency errors and APAC stale prices separately. A database spike is a downstream distraction. Verify one mechanism and inspect why the incident still cannot close.

Design record:
- Which signal was a consequence, and what timeline supports that interpretation?
- Which affected cohort remained after your first verified mechanism?

## Lesson 11 — Evaluating Investigation Systems

Score candidate coverage, unsafe conclusions, diagnostic behavior, abstention, and replay alongside final answers.

Time: 50–65 min. Artifact: Deterministic evaluation report.

### A correct final answer can conceal an unreliable process.

An investigator may name the right cause after skipping contradictory evidence, leaking tool authority, or relying on an unsourced guess. A final-answer metric would reward that run. Evaluate the process by recording the candidates considered, checks selected, eliminations, source quality, verification attempts, budget usage, and unresolved handoff. Unsafe verification deserves a separate metric because it can make a convincing report operationally dangerous.

Ground truth belongs to the evaluator, outside the policy and model context. The fixtures specify which mechanisms are true and which incidents have enough available evidence for closure. The engine sees only authored observations and contracts; it does not import the evaluator’s answer map.

### Define every denominator.

Hypothesis recall measures inclusion of true mechanisms. Initial recall and final recall after expansion answer different questions. Hypothesis precision here uses the supported or verified shortlist; it does not score every early possibility as a mistake. False elimination divides true mechanisms incorrectly falsified by the number of true mechanisms. False verification divides incorrectly verified mechanisms by verified mechanisms, with zero when none are verified.

Evidence provenance coverage measures the fraction of ledger entries that retain required source fields. This is a schema metric, not proof that the source is authentic or the interpretation correct. Tool-call efficiency in this harness measures unique collected checks per collector call; it catches duplication but does not claim that every unique check was useful.

### Make deterministic evaluation a controlled experiment.

Pin fixture and policy versions, input snapshots, tool observations, recorded judgments, tie-breaking rules, and metric definitions. The fixture judge uses stable ordering, and replay consumes recorded model decisions without calling a provider. A fresh live model run may choose different checks even with similar decoding settings; evaluate its repeated outcomes separately from deterministic replay.

Measure virtual fixture latency and cost for the offline suite, and mark them clearly. A production benchmark needs actual collector latency, model usage and price assumptions, sampling policies, and repeated noisy incidents. Four authored scenarios are useful regression checks but cannot estimate general production reliability.

### Reward a useful unresolved outcome.

The insufficient-evidence fixture intentionally lacks a matching replay window. Correct behavior is escalation with a supported account, evidence references, a named owner, and unresolved scope. Closing that fixture would be a false incident-level verification even if the hypothesized cache mechanism happened to be the hidden truth.

Run the suite below and inspect initial recall in the missing-cause case, complete scope coverage in the two-fault case, and abstention quality in the insufficient case. The accompanying test harness also attacks malformed judgments, repeated checks, budget exhaustion, stale observations, premature verification, deadlines, and tampered replay reports.

| Metric group | What it reveals |
| --- | --- |
| Diagnosis | Correct mechanisms and correctly unresolved incidents |
| Candidates | Initial / final recall, supported-shortlist precision |
| Safety | False elimination, false mechanism verification, false incident closure |
| Evidence | Required provenance-field coverage |
| Efficiency | Depth, decisions, calls, duplicate-call ratio, fixture cost and time |
| Handoff | Correct abstention, usable escalation, replay consistency |

### Lab — Score answers and investigation behavior

Run all four fixed scenarios with the same policy, inspect the per-case metrics, and export the machine-readable evaluation. Repeated suite runs should return the same report.

Design record:
- Which failure would final-answer accuracy miss in your environment?
- Which dataset must you keep outside the investigation policy?

## Capstone — Build an AI Investigation System

Investigate a simulated distributed quote application using bounded actions, sourced evidence, explicit state, and verifiable stopping conditions.

- Generate or expand multiple testable hypotheses.
- Maintain explicit incident and per-hypothesis state.
- Select only permitted diagnostic actions with typed envelopes.
- Record structured evidence, source versions, scope, and time windows.
- Falsify candidates when scoped evidence contradicts their predictions.
- Keep unrestricted execution and production mutation outside the model interface.
- Enforce decision, tool, model-call, cost, and elapsed-time budgets.
- Distinguish a supported theory, a verified mechanism, and verified incident coverage.
- Abstain when matching evidence is unavailable and escalate to an owner.
- Export a replayable JSON ledger and a readable investigation report.

## Further reading

[Engineering references for this track](https://deploytovalue.com/resources/investigation-systems/)