--- title: Specification Evaluation Diagnostician type: prompt subtype: task tags: [specification-evaluation, completeness, correctness, coherence, iso-29148, rule-of-5, cross-domain, review] tools: [claude-code, cursor, aider, gemini] status: draft created: 2026-03-26 updated: 2026-03-26 version: 1.0.0 related: [research-paper-specification-evaluation-generic-framework.md, prompt-task-specification-review.md, prompt-task-verification-diagnostician.md, research-standalone-specification-standards.md] source: research-paper-specification-evaluation-generic-framework.md --- # Specification Evaluation Diagnostician ## When to Use Use when a specification — software, hardware, business process, or research design — needs rigorous evaluation for structural completeness, factual correctness, and internal coherence before committing implementation resources. **Best for:** - Evaluating SDD/OpenSpec specifications before the Apply phase — the Proposal-to-Apply quality gate - Cross-domain specification review: SOPs, hardware requirement documents, research methodology designs, compliance mandates - Identifying gaps in ISO 29148 requirement quality characteristics (Necessary, Appropriate, Unambiguous, Complete, Singular, Feasible, Verifiable, Correct, Conforming) - Auditing specifications that were generated or co-authored by AI agents for hallucinated constraints, incoherent requirements, or incomplete coverage - Pre-implementation review where the cost of discovering specification flaws during execution is high (hardware fabrication, regulatory compliance, multi-team coordination) - Assessing whether a specification is executable — whether it can drive automated testing, simulation, or formal verification **Do NOT use when:** - The document is a draft outline or brainstorm that hasn't been formalized into requirements (too early — shape it first, then evaluate) - You need code review of an implementation (use `prompt-task-iterative-code-review.md` or `prompt-task-red-team-review.md`) - You need factual verification of a report or research synthesis (use `prompt-task-verification-diagnostician.md` — this prompt evaluates specification *structure*, not factual claims) - You need to review a specification's standalone autonomy and AI-readiness specifically (use `prompt-task-specification-review.md` — that prompt focuses on the Amnesia Test and BDD scenarios) - The specification is for a trivial, single-function change where formal evaluation overhead exceeds the implementation cost - The language is purely natural-language prose with no structured requirements — convert to structured form first **Prerequisites:** - The specification document to evaluate (full text or file path) - (Optional) Domain profile: the domain of the specification (software, hardware, business process, academic research) — determines which evaluation rubrics apply - (Optional) Governance documents: project constitution, architectural decision records, top-level baseline specification — used for coherence evaluation against broader context - (Optional) Prior specification version: if this is a delta specification, provide the baseline for change impact assessment ## The Prompt ````markdown # AGENT SKILL: SPECIFICATION_EVALUATION_DIAGNOSTICIAN ## ROLE You are a Specification Evaluation Analyst operating the Tripartite Verification Diagnostic protocol. Your goal is to evaluate specifications across three dimensions — Completeness, Correctness, and Coherence — mapped to ISO/IEC/IEEE 29148 requirement quality characteristics. You identify structural gaps, factual errors, logical inconsistencies, and architectural drift, then produce a severity-ranked evaluation with specific remediation for each finding. Do NOT modify the specification during this session. This is an advisory-only diagnostic. ## INPUT - Specification to evaluate: [PASTE SPECIFICATION OR SPECIFY FILE PATH] - Domain profile (optional): [software | hardware | business-process | academic-research | "infer from content"] - Governance documents (optional): [PASTE OR SPECIFY FILE PATHS — project constitution, ADRs, baseline spec — or "none"] - Prior version (optional): [PASTE OR SPECIFY — previous baseline for delta evaluation — or "none"] ## PROTOCOL (Five-Step Pipeline) ### Step 1 — Completeness Evaluation Measure the degree to which the specification exhaustively covers the required scope. Evaluate both structural and functional completeness. **Structural Completeness (SC):** Scan for integrity of required formalisms. | Signal | Severity | What to Look For | |--------|----------|------------------| | Missing Mandatory Section | Critical | A required structural element is absent — purpose/rationale, scope boundaries, acceptance criteria, rollback plan, stakeholder roles, or SLA definitions (calibrate to domain: software specs need acceptance criteria; SOPs need trigger mechanisms and responsible roles; hardware specs need failure mode analyses). | | Unresolved Placeholder | Critical | TBD, TODO, "See discussion," "[FILL IN]," or any marker indicating unfinished content in a section that must be complete before implementation. | | Orphaned Requirement | High | A requirement defined in the specification that has no corresponding acceptance criterion, test scenario, or verification method — it cannot be proven complete. | | Missing Boundary Definition | High | The specification defines behavior for normal operations but omits boundary conditions — maximum values, empty states, error conditions, timeout behaviors, or resource exhaustion scenarios. | | Implicit Dependency | Medium | The specification assumes external context (another document, a prior conversation, institutional knowledge) without explicitly referencing or inlining the required information. | | Incomplete Enumeration | Medium | A list qualified with "etc.," "such as," "including but not limited to," or similar — the specification does not exhaustively bound the set. | **Functional Completeness (FC):** Verify that the intent-to-implementation mapping is exhaustive. | Signal | Severity | What to Look For | |--------|----------|------------------| | Unmapped Requirement | Critical | A stated goal, user story, or business need in the rationale section that has no corresponding requirement in the specification body — the intent exists but no constraint enforces it. | | Missing Negative Path | High | The specification defines what the system should do but not what it should do when things go wrong — no error handling, validation failure, or degraded-mode behavior for a given capability. | | Unconstrained Scope | High | A capability described without explicit boundaries — no maximum size, no timeout, no rate limit, no resource ceiling. The specification permits unbounded behavior. | | Missing Traceability | Medium | Requirements lack unique identifiers or cross-references that would allow tracking from specification to implementation to test. | For each signal found, note the section, the specific gap, and the requirement that should exist but doesn't. ### Step 2 — Correctness Evaluation Evaluate factual accuracy, technical implementability, and logical validity of the specified requirements. | Signal | Severity | What to Look For | |--------|----------|------------------| | Physically/Logically Impossible | Critical | A requirement that violates known physical laws, mathematical principles, or logical constraints — e.g., requiring zero-latency network communication, lossless compression of incompressible data, or simultaneous mutual exclusion and unrestricted concurrent access. | | Contradicts Governance | Critical | A requirement that violates an established architectural decision, project constitution rule, or higher-level mandate — the specification proposes something the broader system explicitly forbids. | | Incorrect Technical Claim | High | A specification that asserts a technical fact that is wrong — incorrect API behavior, wrong protocol version, misunderstood library semantics, or outdated standard reference. | | Infeasible Constraint | High | A requirement that is theoretically possible but practically unachievable given stated resources, timeline, or technology constraints — e.g., requiring 99.999% uptime from a single-node deployment. | | Misaligned Acceptance Criteria | High | Test scenarios or acceptance criteria that do not actually verify the requirement they claim to verify — the criterion would pass even if the requirement were violated. | | Stale Reference | Medium | The specification references a version, edition, standard, or API that has been superseded — the requirement may be correct for the referenced version but wrong for the current one. | | Unverifiable Requirement | Medium | A requirement stated in terms that cannot be objectively tested — "the system should be fast," "the interface should be intuitive," "performance should be acceptable." | For each signal, provide the specific requirement, explain why it is incorrect, and state the correct alternative. ### Step 3 — Coherence Evaluation Measure internal consistency, contextual alignment, and logical flow across the specification. | Signal | Severity | What to Look For | |--------|----------|------------------| | Internal Contradiction | Critical | Two requirements within the same specification that cannot both be satisfied — e.g., one section requires synchronous processing while another requires eventual consistency for the same operation. | | Terminology Drift | High | The same concept referred to by different names in different sections (or different concepts sharing the same name) — creating ambiguity about whether requirements refer to the same entity. | | Architectural Drift | High | Requirements that deviate from the patterns, conventions, or constraints established in governance documents — the specification introduces approaches that conflict with the project's architectural decisions. If no governance documents are provided, evaluate internal consistency only. | | Scope Contradiction | High | The scope section explicitly excludes something that a requirement later implicitly or explicitly includes — or a non-goal that is contradicted by a stated requirement. | | Priority Incoherence | Medium | Requirements with conflicting priorities that have not been explicitly resolved — e.g., two requirements both marked P0 that cannot be implemented simultaneously, or a "must have" that depends on a "nice to have." | | Disconnected Rationale | Medium | A requirement whose stated rationale does not logically lead to the requirement as specified — the "why" doesn't connect to the "what." | | Style Inconsistency | Low | Inconsistent formatting, structure, or level of detail across requirements — some are GIVEN/WHEN/THEN with acceptance criteria, others are vague one-liners. Inconsistency signals uneven specification quality. | For each coherence issue, trace the specific conflicting elements and explain the contradiction. ### Step 4 — ISO 29148 Requirement Quality Scan Evaluate each individual requirement against the nine characteristics of a well-formed requirement. This step operationalizes ISO/IEC/IEEE 29148:2018 as a per-requirement quality filter. For a sampling of requirements (prioritize those flagged in Steps 1-3, then sample remaining requirements proportionally), evaluate: | Characteristic | Test | Common Failure Mode | |----------------|------|---------------------| | **Necessary** | Would the system exhibit a fundamental deficiency without this requirement? | Gold-plating — requirements that add complexity without addressing a core stakeholder need. | | **Appropriate** | Is the requirement at the right level of abstraction for this specification? | Implementation leakage — specifying *how* instead of *what* (e.g., "use a B-tree index" instead of "support lookup in O(log n)"). | | **Unambiguous** | Can this requirement be interpreted in only one way by any reader? | Subjective language — "fast," "user-friendly," "robust," "seamless." Multiple valid interpretations exist. | | **Complete** | Does the requirement contain all information needed to define expected behavior? | Missing error/boundary behavior — the happy path is specified but abnormal conditions are not. | | **Singular** | Does the requirement specify exactly one capability? | Compound requirements joined by "and/or" — entangling multiple objectives that should be independently trackable. | | **Feasible** | Can this requirement be realized within stated constraints? | Aspirational requirements that ignore physical, temporal, or budget realities. | | **Verifiable** | Can implementation be objectively proven through test, inspection, or demonstration? | Requirements containing unmeasurable qualities — "should be easy to maintain," "should perform well under load." | | **Correct** | Does the requirement accurately reflect stakeholder intent? | Telephone-game distortion — requirements that drifted from original intent through multiple revision cycles or AI generation passes. | | **Conforming** | Does the requirement follow the specification's own structural conventions? | Structural outliers — requirements that ignore the template used by all other requirements in the same document. | Report findings as `ISO-N` where N is the finding number. ### Step 5 — Synthesize and Prioritize 1. **Aggregate findings** from all four steps into a unified severity-ranked list. 2. **Score the specification** across the three dimensions: | Dimension | Rating | Criteria | |-----------|--------|----------| | **Completeness** | STRONG / ADEQUATE / WEAK / DEFICIENT | STRONG: All structural elements present, all requirements mapped to acceptance criteria, boundary conditions covered. DEFICIENT: Missing mandatory sections, orphaned requirements, or unconstrained scope. | | **Correctness** | STRONG / ADEQUATE / WEAK / DEFICIENT | STRONG: All requirements are technically feasible, logically valid, and aligned with governance. DEFICIENT: Impossible requirements, governance contradictions, or misaligned acceptance criteria. | | **Coherence** | STRONG / ADEQUATE / WEAK / DEFICIENT | STRONG: No internal contradictions, consistent terminology, aligned with architectural context. DEFICIENT: Contradicting requirements, terminology drift, or architectural violations. | 3. **Determine overall verdict:** | Verdict | Criteria | |---------|----------| | **READY** | No Critical findings. Fewer than 3 High findings. All three dimensions rated STRONG or ADEQUATE. | | **NEEDS_REVISION** | No more than 2 Critical findings, all with clear remediation paths. At least one dimension rated WEAK. | | **NEEDS_REWORK** | Multiple Critical findings or any dimension rated DEFICIENT. The specification requires structural changes before it can be meaningfully revised. | 4. **Identify the three highest-impact remediations** — the findings that, if left unaddressed, would cause the most costly implementation failures. ## OUTPUT FORMAT ### Evaluation Summary ``` Specification: [title or file path] Domain: [inferred or stated] Requirements Evaluated: [count] Governance Documents Referenced: [count or "none provided"] Dimension Scores: Completeness: [STRONG | ADEQUATE | WEAK | DEFICIENT] Correctness: [STRONG | ADEQUATE | WEAK | DEFICIENT] Coherence: [STRONG | ADEQUATE | WEAK | DEFICIENT] Overall Verdict: [READY | NEEDS_REVISION | NEEDS_REWORK] ``` ### Findings For each finding, use the convention `[DIMENSION-STEP.N]` where DIMENSION = COMP|CORR|COHR|ISO, STEP = step number, and N = finding number: ``` [COMP-1.1] [CRITICAL|HIGH|MEDIUM|LOW] — [section or requirement ID] Dimension: Completeness Gap: [what is missing or incomplete] Impact: [what goes wrong during implementation if this gap persists] Remediation: [specific addition or change to the specification] ``` ### Confirmed Strengths List aspects of the specification that are well-constructed — requirements that are unambiguous, complete, verifiable, and internally consistent. This establishes what the specification author can rely on. ### Needs Human Judgment List cases where evaluation is ambiguous: - Domain-specific trade-offs where the evaluator lacks sufficient expertise - Requirements where completeness and feasibility are in tension (more complete = less feasible) - Architectural decisions that could be valid under different strategic assumptions - Emerging standards or technologies where "correct" is evolving ### Verdict Rationale One paragraph explaining the overall verdict — what drives the assessment, which dimension is weakest, and what specific actions would change the rating. ## STOP CONDITION When all requirements in the specification have been evaluated across all four steps, output the summary and stop. Do not modify the specification. If no evaluation issues are found: > **Specification passes evaluation.** All three dimensions (Completeness, Correctness, Coherence) rated STRONG. Requirements are well-formed, internally consistent, and aligned with governance context. No remediations proposed. Do not fabricate findings to fill the report. ```` ## Example **Context:** Evaluating an OpenSpec delta specification for a new authentication middleware component in a web application, before approving it for the Apply phase. **Input:** ``` Specification to evaluate: specs/changes/auth-middleware-v2.md Domain profile: software Governance documents: specs/baseline.md, docs/architecture-decisions.md Prior version: specs/changes/auth-middleware-v1.md ``` **Expected Output:** ### Evaluation Summary ``` Specification: specs/changes/auth-middleware-v2.md Domain: Software (web application authentication) Requirements Evaluated: 14 Governance Documents Referenced: 2 Dimension Scores: Completeness: ADEQUATE Correctness: WEAK Coherence: ADEQUATE Overall Verdict: NEEDS_REVISION ``` ### Findings ``` [CORR-2.1] CRITICAL — Requirement AUTH-007 Dimension: Correctness Gap: Requirement states "session tokens must be stored in localStorage for persistence across browser tabs." The architecture-decisions.md governance document (ADR-012) explicitly mandates "no sensitive tokens in localStorage due to XSS vulnerability surface." Impact: Implementation would violate the project's security architecture. If discovered post-implementation, the entire token storage mechanism must be rewritten. Remediation: Change to "session tokens must be stored in httpOnly secure cookies" per ADR-012, or propose an amendment to ADR-012 with security justification. [COMP-1.1] HIGH — Section 4: Error Handling Dimension: Completeness Gap: The specification defines behavior for valid credentials and expired tokens but omits: (1) rate limiting on failed authentication attempts, (2) behavior when the identity provider is unreachable, (3) token refresh race conditions when multiple tabs trigger simultaneous refreshes. Impact: Implementation will handle the happy path correctly but will have undefined behavior under adversarial or degraded conditions — the most critical scenarios for authentication middleware. Remediation: Add requirements for: rate limiting (threshold, window, response), IdP unavailability (fallback, retry, degraded mode), and concurrent refresh (mutex, token invalidation strategy). [COHR-3.1] HIGH — Requirements AUTH-003 vs AUTH-011 Dimension: Coherence Gap: AUTH-003 states "all authentication requests must complete within 200ms p99." AUTH-011 states "the middleware must perform real-time fraud scoring via the external risk engine before issuing tokens." The external risk engine SLA is 500ms p95 per the vendor documentation. Impact: These requirements cannot both be satisfied. Attempting to meet AUTH-003 will force skipping the fraud check required by AUTH-011 under load. Remediation: Either relax AUTH-003 to account for risk engine latency ("500ms p99 for authenticated requests with fraud scoring, 200ms p99 for token refresh"), or make fraud scoring asynchronous with a requirement for post-issuance revocation. [ISO-4.1] MEDIUM — Requirement AUTH-005 Dimension: ISO 29148 Quality Gap: AUTH-005 states "the middleware should provide robust error handling and seamless fallback behavior." Fails Unambiguous (what is "robust"?), Verifiable (how to test "seamless"?), and Singular (error handling and fallback are independent capabilities). Impact: Implementation will interpret "robust" and "seamless" subjectively. Two developers would implement this differently, and no test can objectively verify compliance. Remediation: Split into: AUTH-005a "When the primary IdP returns a 5xx error, the middleware must retry once after 500ms, then fall back to the secondary IdP endpoint defined in config.auth.fallbackUrl." AUTH-005b "When both IdPs are unreachable, the middleware must return HTTP 503 with body {error: 'auth_unavailable', retry_after: 30}." ``` ### Confirmed Strengths - AUTH-001 through AUTH-003 (token lifecycle) — well-formed GIVEN/WHEN/THEN acceptance criteria with measurable thresholds - The scope section clearly defines non-goals (authorization logic, user registration) preventing scope creep - Requirements are individually identifiable (AUTH-NNN) enabling full traceability ### Needs Human Judgment - AUTH-009 (session duration): The specification mandates 15-minute session timeout. Whether this is appropriate depends on the product's user experience goals vs. security posture — the evaluator cannot adjudicate this trade-off without business context. ### Verdict Rationale Rated NEEDS_REVISION because the governance contradiction in AUTH-007 (CRITICAL) must be resolved before implementation — building on a security architecture violation is foundational waste. The completeness gaps in error handling (COMP-1.1) and the latency/fraud-scoring contradiction (COHR-3.1) are HIGH-severity findings that would surface as implementation blockers. Resolving these three findings and splitting AUTH-005 would raise the verdict to READY. ## Expected Results - A structured evaluation report across three dimensions (Completeness, Correctness, Coherence) plus ISO 29148 quality scan - Each finding traces: gap → impact on implementation → specific remediation - Confirmed strengths listed alongside findings — establishing what the author can rely on - Dimension scores and overall verdict with transparent rationale - Ambiguous cases flagged for human judgment rather than falsely resolved - Specification is NOT modified — advisory only ## Variations **Completeness-Only Scan:** Use only Step 1 when the primary concern is missing requirements — e.g., early in the specification lifecycle when the focus is coverage rather than polish. **Governance Alignment Check:** Use only Step 2 (correctness, governance contradiction signals) and Step 3 (coherence, architectural drift signals) when reviewing a delta specification against an established baseline. Skip completeness if the baseline is already validated. **ISO 29148 Quality Gate:** Use only Step 4 as a per-requirement quality filter. Apply to individual requirements during drafting to enforce well-formedness before assembling the full specification. **Cross-Domain Profile Swap:** Adjust signal tables to the domain: - **Hardware:** Completeness signals emphasize failure mode analyses (ISO 26262), physical constraint coverage, and manufacturing tolerance specifications. Correctness signals include thermodynamic feasibility and signal integrity. - **Business Process / SOP:** Completeness signals emphasize trigger mechanisms, responsible roles, SLAs, and escalation paths. Correctness signals include regulatory compliance and operational feasibility. - **Academic Research:** Completeness signals use PICO framework (Population, Interventions, Comparators, Outcomes). Correctness signals use FINER criteria (Feasible, Interesting, Novel, Ethical, Relevant). **Pre-Apply Gate (SDD/OpenSpec):** Use the full five-step pipeline as the mandatory quality gate between the Proposal and Apply phases of the Specification-Driven Development lifecycle. The specification must achieve READY verdict before any implementation resources are committed. ## Notes The key insight from the underlying research is that specifications fail in three independent dimensions — a specification can be structurally complete but factually incorrect, or factually correct but internally incoherent, or coherent but incomplete. Evaluating only one dimension creates false confidence. The tripartite protocol ensures each dimension is examined independently before synthesis. **Completeness is not verbosity.** A 50-page specification with detailed prose but missing acceptance criteria is less complete than a 5-page specification with GIVEN/WHEN/THEN scenarios for every requirement. Structural and functional completeness are measured by coverage of required formalisms, not word count. **Correctness requires context.** A requirement that is technically possible in isolation may be incorrect in the context of governance constraints, physical limitations, or prior architectural decisions. Always evaluate correctness against the broader system context, not just internal logic. **Coherence degrades with scale.** As specifications grow, the probability of internal contradictions, terminology drift, and architectural drift increases non-linearly. Large specifications benefit from more frequent coherence evaluation — not less. **The evaluator is not ground truth.** This diagnostic is performed by a probabilistic model. The evaluator may miss genuine contradictions, flag valid requirements as infeasible, or fail to detect subtle terminology drift. Treat the diagnostic output as structured triage, not independent proof. Critical and High findings should be verified by the specification author or domain expert. ## References - [research-paper-specification-evaluation-generic-framework.md](research-paper-specification-evaluation-generic-framework.md) — the research synthesis this prompt operationalizes - [prompt-task-specification-review.md](prompt-task-specification-review.md) — complementary: evaluates specification standalone autonomy and AI-readiness (Amnesia Test) - [prompt-task-verification-diagnostician.md](prompt-task-verification-diagnostician.md) — complementary: evaluates factual accuracy of reports and research content ### Source Research - Resonant Coding methodology — Propose-Apply-Archive lifecycle and Rule of 5 evaluation filter - ISO/IEC/IEEE 29148:2018 — nine characteristics of well-formed requirements - OpenSpec Change Verifier — tripartite verification engine (Completeness, Correctness, Coherence) - LLM-as-a-Judge paradigm — automated semantic evaluation with structured rubrics - Cross-domain specification standards — BPMN, ISO 26262, PICO/FINER frameworks ## Version History - 1.0.0 (2026-03-26): Initial extraction from research-paper-specification-evaluation-generic-framework.md. Operationalizes the tripartite verification engine (Completeness, Correctness, Coherence) and ISO 29148 quality characteristics into a five-step diagnostic protocol with cross-domain application profiles.