Skip to content
LESSON 07 · CORE

Stop, Continue, or Escalate

Implement stopping conditions that preserve the difference between strong support, verified scope, and an unresolved handoff.

Open this lesson’s lab
TIME35–45 min
LEVELAdvanced systems
LABBounded design exercise
ARTIFACTStopping and escalation policy
Confidence and verificationSupported and verified claimsStopping conditionsLatency and resource budgetsMarginal valueAbstention and escalationUnresolved incidents

State is a claim about what the system can justify.

OPEN means an incident exists but diagnostic work has not started. INVESTIGATING means observations or decisions are being collected. SUPPORTED means a candidate has sufficiently specific support under the declared policy. VERIFIED means the mechanism gate passed and every declared impact scope is explained. UNRESOLVED records the evidence gap; ESCALATED records a handoff to a named operator.

Support can be withdrawn when contradictory evidence arrives. A verified mechanism for EU quotes does not close an incident that also includes unexplained APAC prices. Keep per-hypothesis status separate from incident status. In the engine, an individual mechanism may pass while the incident remains SUPPORTED until coverage is complete.

Define closure before seeing the answer.

The verification policy checks provenance, freshness, mechanism evidence, controlled contrast, and scope coverage. A model confidence score is not one of those predicates. If the gate fails, report the failing requirements. A manual “verify” request evaluates the gate and leaves the investigation open when the evidence is insufficient.

Budgets control continuation. Stop requesting more evidence when the permitted next check would exceed time, tool calls, or cost. Evaluate existing evidence first; a gate already satisfied can still close the diagnostic workflow. Otherwise preserve the best-supported account and escalate. Exhaustion must not convert support into verification.

Abstention needs an operational next step.

An unresolved report should identify what is supported, what remains unexplained, which evidence was unavailable, who owns the next action, and which diagnostic request could close the gap. “The model is uncertain” gives an operator very little to work with. “The cache replay lacks the affected time window; the quote-platform owner must provide snapshot key-v12 and request pair q-17/q-18” is a practical handoff.

A system can escalate early when permitted checks cannot resolve a material gap. During severe impact, the commander may prioritize containment while diagnosis remains incomplete. These decisions belong to the response workflow; investigation evidence should inform them without silently granting the model mitigation authority.

Another check should have a reason to exist.

Marginal value is the expected improvement from one more observation given the current state. It declines when remaining checks are redundant or cannot change any relevant decision. A cheap check may still be worthwhile if it challenges the leading mechanism, while repeated supportive samples may add little. Use explicit continuation criteria, and record why a run stops even when its final answer sounds convincing.

ConditionPolicy outcome
Fresh controlled evidence + all scopes coveredVerify
Strong support + missing mechanism evidence + useful affordable checkContinue
One verified mechanism + unexplained second cohortContinue or escalate; incident stays open
Insufficient evidence + budget exhaustedUnresolved → escalated
No permitted check can close the gapUnresolved → escalated
DESIGN PRINCIPLE

LLMs make uncertain judgments. Software controls state, authority, verification, and stopping.

INTERACTIVE LAB · STOPPING AND ESCALATION POLICY

Implement the stopping table

Choose the policy outcome for each state snapshot. A high confidence score cannot substitute for a failed gate.

01 · Two sources support cache omission; a matching controlled replay is still available and affordable.

02 · A cache replay passes, but the APAC price cohort remains unexplained and its supplier check is affordable.

03 · The model reports 99% confidence; the matching replay window is unavailable and no permitted check can retrieve it.

04 · Every scope has passing baseline, treatment, control, freshness, and provenance checks.

05 · The next necessary check exceeds the remaining budget, and the verification gate still fails.

DESIGN RECORD

Keep the reasoning that should survive the run.

Write the assumption or boundary you would carry into your own implementation.

Engineering references for this track · Troubleshooting, observability, and bounded workflow design.