Approved safety procedures
Highest authority for hazardous work. Answers may not contradict these controls.
BindingMISSION 03 · PRODUCTION AI
IN PRODUCTION
Titan
Industrial
Deploy a field-service knowledge assistant that remains useful under operational pressure without producing unsupported or unsafe repair guidance.
01 / CORE QUESTION
How can technicians get useful answers without unsafe shortcuts?
Titan maintains complex industrial equipment at customer sites. Experienced technicians solve many issues from memory, while newer technicians search across manuals, service bulletins, parts catalogs, and old work orders.
The mission asks the learner to deploy an AI-assisted retrieval workflow that clearly exposes evidence, respects document authority, handles conflicting sources, and escalates when the system cannot answer safely.
DECISION TO IMPROVEWhat procedure or next diagnostic step should the technician consider, and when must the system abstain or escalate?
02 / SOURCE SYSTEM
Authority, usefulness, and recency do not align.
The corpus is designed so the most readable source is not always the safest and the most authoritative source is not always the most current.
Current service bulletins
May supersede older manuals for specific equipment models and serial ranges.
Authoritative and time-sensitiveEquipment manuals
Comprehensive but potentially outdated when newer bulletins exist.
Approved with version checksWork-order histories
Operationally useful observations with inconsistent quality and incomplete verification.
Contextual evidenceTechnician notes
Concise, practical, and sometimes unsafe. Cannot silently override approved procedure.
Unverified field knowledgeEXPECTED CORPUS DEFECTSDuplicate versionsModel aliasesMissing datesConflicting instructionsUnsafe shortcutsScanned documents
03 / REQUIRED BUILD
A governed retrieval and answer workflow.
The system is evaluated as an operational product, not merely on whether an LLM can produce fluent answers.
Document ingestion
Parse, classify, version, deduplicate, and attach equipment applicability metadata.
Equipment identification
Resolve model numbers, serial ranges, aliases, and component relationships before retrieval.
Retrieval hierarchy
Rank by authority, recency, equipment match, safety relevance, and semantic relevance.
Cited answer interface
Expose source passages, version dates, authority level, uncertainty, and conflicts.
Safety escalation
Block unsupported procedural synthesis and route hazardous or ambiguous cases to human review.
Feedback workflow
Collect corrections without allowing unreviewed feedback to alter approved procedure.
04 / EVALUATION FRAMEWORK
The system must be rewarded for abstaining.
A strong evaluation set includes answerable, ambiguous, outdated, conflicting, and safety-critical cases.
| Metric | What it tests | Target behavior |
|---|---|---|
| Citation accuracy | Whether claims are supported by the displayed source passages. | Unsupported claims are treated as failures. |
| Retrieval relevance | Whether the returned sources match equipment, issue, and procedure context. | Authority and applicability outweigh generic semantic similarity. |
| Unsupported-answer rate | How often the system invents or overextends beyond evidence. | Near zero for procedural claims. |
| Safety escalation accuracy | Whether hazardous or conflicting cases reach human review. | High recall with documented false-positive tradeoffs. |
| Outdated-source usage | Whether superseded documents influence answers improperly. | Newer bulletins visibly override older manuals. |
| Abstention quality | Whether the system refuses clearly and usefully when evidence is insufficient. | Refusal includes missing information and next action. |
| User correction rate | How often technicians identify a wrong mapping, source, or answer. | Corrections enter a governed review queue. |
| Latency and cost | Whether the workflow remains usable in field conditions. | Targets reflect limited connectivity and device constraints. |
05 / HIDDEN COMPLICATION
The preferred source is the least governed.
The complication is released after the first retrieval and answer-policy checkpoint.
REVEAL PRODUCTION COMPLICATION+
Pilot technicians consistently prefer answers grounded in informal work-order notes because they are shorter and more practical. Several of those notes describe unofficial shortcuts that bypass safety checks or use obsolete parts.
Required response: revise source ranking, answer composition, warnings, feedback handling, and escalation. Preserve the useful field context without allowing unapproved notes to become procedural authority.
Authority first
Safer but may reduce usefulness and technician adoption.
Context with controls
Use notes as supporting context while preventing them from generating instructions.
Human review queue
Promote useful field practices only after expert validation and document governance.
Explicit conflict display
Show disagreements among sources rather than silently selecting one answer.
06 / PRODUCTION STATUS
What is being built now.
The public specification is complete. The gated corpus, evaluation cases, and starter implementation are the next production work.
Mission and governance model
Decision, source hierarchy, required workflow, complication, and evaluation dimensions.
Document corpus
Manuals, bulletins, work orders, technician notes, versions, and model mappings.
Starter implementation
Ingestion pipeline, vector retrieval, authority filters, citations, and evaluation harness.
Assessment package
Test cases, pilot metrics, hidden complication, reviewer anchors, and defense questions.
EARLY ACCESS
Pilot the governed AI mission.
Help test whether the evidence hierarchy, safety cases, and evaluation workload reflect real production AI deployment.