Skip to content
LESSON 02 · CORE

Building the Hypothesis Set

Create a candidate space that covers services, dependencies, infrastructure, and changes without hiding the possibility of a missing cause.

Open this lesson’s lab
TIME40–50 min
LEVELAdvanced systems
LABBounded design exercise
ARTIFACTHypothesis tree and prediction table
Candidate root causesService, dependency, infrastructure, and change hypothesesHypothesis treesCoverage and precisionMissing root causesKnown and unknown gaps

The correct choice needs a place in the candidate set.

An investigator can rank every available hypothesis perfectly and still fail when the causal mechanism is absent. Begin with the affected user contract and a scoped dependency graph. For incorrect quotes, candidate domains include application arithmetic, database inputs, supplier responses, cache behavior, network delivery, and effective configuration. A recent deployment is one branch of that tree, rather than the definition of the whole search space.

Make leaves concrete enough to predict an observation. “The database is broken” does not distinguish a stale replica, lock contention, or an incorrect query. “The quote reads a replica that is twenty minutes behind” predicts a version difference between primary and replica for the same record. Keep the scope, proposed mechanism, expected observation, and falsifying observation together.

Coverage and precision create different engineering pressures.

Coverage asks whether the candidate set includes the causal mechanisms needed to explain the incident. Precision asks how much of the candidate space is useful. A huge list can hide a real cause under dozens of irrelevant theories, while a tidy list of three popular failure modes can omit a configuration change. Start with a small, diverse set, record why each candidate is in scope, and reserve an explicit expansion path.

Do not assign the absence of a candidate the meaning “impossible.” In this track, a service-catalog check can expose a separately managed cache dependency. The engine then permits adding the cache hypothesis. This is a limited teaching mechanism with an authored registry; a production expansion process needs topology ownership, scoped discovery permissions, and validation of any newly proposed diagnostic contract.

Build a tree that preserves useful distinctions.

Use the top level for failure domains and the leaves for mechanisms. Put the new-binary hypothesis under service changes; put omitted key dimensions under configuration; put supplier throttling under an external dependency. A change can cross domains, so the tree is an organizing aid rather than proof of independence. If two leaves describe the same mechanism, merge them before they compete for budget.

Attach expected evidence to every leaf. This makes candidate generation reviewable and gives the check selector a discriminative target. When all surviving candidates predict something different from the user experience, mark candidate coverage as suspect. Revisit topology, recent effective configuration, and the original contract before inventing a more elaborate story.

Preserve the boundary between gaps and explanations.

A known unknown is a gap you can name, such as an unavailable configuration history. An unknown unknown becomes visible when an unexpected observation exposes a missing dependency or failure mode. Record both without assigning invented probabilities. Your hypothesis tree should include an “unexplained / expand” condition so that elimination of current candidates leads to discovery or escalation instead of forced selection.

DomainMechanismDiscriminating prediction
Service / changeNew binary changes arithmeticOnly the new build fails a matched replay
DependencyDatabase causes initial incorrect totalsThe relevant database failure precedes wrong quotes
InfrastructurePacket loss corrupts this pathScoped network evidence and a replay match impact
External dependencySupplier throttles APAC pricesCurrent APAC calls and a route replay reproduce failure
ConfigurationKey omits currencyDifferent currencies collide and a key-only change fixes the replay
DESIGN PRINCIPLE

LLMs make uncertain judgments. Software controls state, authority, verification, and stopping.

INTERACTIVE LAB · HYPOTHESIS TREE AND PREDICTION TABLE

Build a hypothesis tree from five signal families

For each leaf, choose its domain and its first discriminating check. The generated tree preserves your predictions, including an explicit unexplained branch.

01 · A new quote binary deployed near incident onset.

02 · Database wait time rose after user retries began.

03 · One network zone may be unhealthy.

04 · APAC supplier calls may return throttling responses.

05 · Quote caching has its own configuration owner.

Generated hypothesis tree
Your tree appears after evaluation.

DESIGN RECORD

Keep the reasoning that should survive the run.

Write the assumption or boundary you would carry into your own implementation.

Engineering references for this track · Troubleshooting, observability, and bounded workflow design.