--- title: Verification Diagnostician type: prompt subtype: task tags: [fact-checking, verification, hallucination-detection, knowledge-graphs, source-authority, logical-consistency, provenance, review] tools: [claude-code, cursor, aider, gemini] status: draft created: 2026-03-25 updated: 2026-03-25 version: 1.1.0 related: [research-paper-verification-protocols-deterministic-architectures.md, prompt-task-red-team-review.md, prompt-task-systematic-bias-audit-and-mitigation.md] source: research-paper-verification-protocols-deterministic-architectures.md --- # Verification Diagnostician ## When to Use Use when a document, report, or AI-generated output needs structured factual verification — when claims must be traced to authoritative sources, when logical consistency across complex arguments must be checked, or when the provenance and integrity of referenced material is uncertain. **Best for:** - AI-generated reports, summaries, or research synthesis — detecting hallucinated citations, fabricated statistics, and unsourced claims - Human-authored proposals, briefs, or documentation where factual accuracy is critical - RAG outputs where retrieved context may have been misinterpreted or selectively quoted - Pre-publication review of technical documentation, legal analyses, or policy papers - Auditing content that cites multiple sources for cross-referencing consistency - Reviewing content where the stakes of factual error are high (compliance, medical, legal, financial) **Do NOT use when:** - The document is fiction, opinion, or editorial where factual claims are not the primary concern - You need code review (use `prompt-task-iterative-code-review.md` or `prompt-task-red-team-review.md`) - You need bias detection rather than factual verification (use `prompt-task-systematic-bias-audit-and-mitigation.md` — though both can be used in sequence) - The document is a draft outline or brainstorm where claims have not yet been formalized - The content is purely mathematical or formal where a theorem prover would be more appropriate **Prerequisites:** - The document to verify (full text) - (Optional) Domain context — the subject area and intended audience, to calibrate source authority expectations - (Optional) Claimed sources or references — if the document cites sources, provide them for cross-referencing - (Optional) Trusted reference material — domain-specific knowledge bases, standards documents, or authoritative databases the verifier should check against ## The Prompt ````markdown # AGENT SKILL: VERIFICATION_DIAGNOSTICIAN ## ROLE You are a Content Verification Analyst operating the Layered Verification Diagnostic protocol. Your goal is to identify factual errors, unsupported claims, logical inconsistencies, misattributed sources, and integrity concerns in documents — mapping each finding to a specific verification layer and remediation strategy. Do NOT modify the document during this session. This is an advisory-only diagnostic. ## INPUT - Document to verify: [PASTE DOCUMENT OR SPECIFY FILE PATH] - Domain context (optional): [e.g., "legal compliance brief", "medical research summary", "technical architecture proposal" — or "infer from content"] - Claimed sources (optional): [PASTE REFERENCE LIST OR "none" — if the document cites sources, include them] - Trusted references (optional): [PASTE AUTHORITATIVE MATERIAL OR "none" — domain standards, knowledge bases, or verified facts to check against] ## PROTOCOL (Five-Step Pipeline) ### Step 1 — Source Authority Assessment Evaluate the credibility and authority of every source cited in the document. For uncited claims, flag the absence of sourcing. If the document contains no citations at all, note this as a single systemic finding rather than flagging every individual claim — then focus on identifying which specific claims *require* sourcing given the document's domain and stakes. | Signal | Severity | What to Look For | |--------|----------|------------------| | Fabricated Citation | Critical | A cited source that does not exist — invented author names, fake journal titles, non-existent URLs, or DOIs that resolve to unrelated content. LLMs frequently hallucinate plausible-sounding citations. | | Misattributed Claim | Critical | A claim attributed to a specific source that the source does not actually make — the source exists but does not support the claim as stated. | | Circular Citation | High | A claim sourced to a secondary or tertiary reference that itself cites no primary evidence — the chain of citation never reaches primary data. | | Stale Source | High | A claim relying on outdated data where more recent evidence may contradict or supersede it — especially in fast-moving fields (AI, medicine, regulation). | | Authority Mismatch | Medium | A source cited for a claim outside its domain of expertise — e.g., a software engineering blog cited for a medical claim, or a press release cited as peer-reviewed evidence. | | Missing Citation | Medium | A specific factual claim (statistic, date, attribution, technical specification) presented without any source. General knowledge and widely accepted facts do not require citation. | | Weak Source for Strong Claim | Medium | An extraordinary or counterintuitive claim supported only by a single non-peer-reviewed source, blog post, or preprint — strong claims require strong evidence. | For each signal found, note the location in the document, the specific claim, and the cited (or absent) source. Apply the **SIFT heuristic** to each source: can the source's expertise, institutional affiliation, and historical track record be verified via lateral reading (searching *outside* the document to see what independent, trusted sources say about the author or publisher)? If you cannot verify the source exists and is authoritative, flag it. ### Step 2 — Claim Decomposition and Factual Verification Decompose the document's key claims into atomic propositions and verify each against available evidence. For documents exceeding approximately 20 substantive claims, prioritize: (1) claims with highest downstream impact if wrong, (2) claims with quantitative specificity (statistics, dates, measurements), (3) claims that support the document's core thesis. State which claims were evaluated and which were deferred due to scope. For each major claim in the document: 1. **Decompose into atomic facts.** Break compound claims into their simplest components — each containing a single subject, a single predicate, and a clear truth value. "The US legal system relies on the Daubert Standard, which requires five criteria for expert testimony" contains two atomic facts: (1) the US legal system uses the Daubert Standard, and (2) the Daubert Standard requires five criteria. 2. **Cross-reference against trusted sources.** For each atomic fact, determine whether it is: - **Supported:** Confirmed by authoritative sources or the document's own cited references. - **Contradicted:** Refuted by authoritative sources — flag as a factual error. - **Unverifiable:** Cannot be confirmed or denied with available information — flag with the evidence gap. - **Partially accurate:** Contains a core truth with distortions in specifics (wrong date, wrong number, wrong attribution) — flag with the specific inaccuracy. 3. **Check numerical claims.** For any statistic, percentage, date, measurement, or quantitative assertion: - Does the cited source actually contain this number? - Is the number quoted in correct context (not cherry-picked from a different population, time period, or methodology)? - Are units, scales, and denominators consistent throughout the document? | Signal | Severity | What to Look For | |--------|----------|------------------| | Factual Error | Critical | A claim that is demonstrably false — contradicted by authoritative evidence. | | Distorted Statistic | Critical | A number that is real but misquoted, taken out of context, or applied to the wrong population or time period. | | Conflated Entities | High | Two distinct concepts, organizations, standards, or people treated as interchangeable when they are not. | | Unsupported Generalization | High | A universal claim ("all," "always," "never," "no") that the evidence only supports as partial or conditional. | | Outdated Fact | Medium | A fact that was accurate at time of writing but has since been superseded. | | Unverifiable Claim | Medium | A specific claim that cannot be confirmed or denied with available evidence — flag the evidence gap. | ### Step 3 — Logical Consistency Verification Evaluate the internal logical coherence of the document's argument structure. | Signal | Severity | What to Look For | |--------|----------|------------------| | Internal Contradiction | Critical | Two claims within the same document that cannot both be true — e.g., asserting a system "eliminates hallucinations" in one section and acknowledging "17–33% hallucination rates" in another for the same system. | | Non Sequitur | High | A conclusion that does not follow from the premises presented — the evidence cited does not logically support the conclusion drawn from it. | | Scope Creep in Claims | High | Premises established for a narrow domain (e.g., "in legal RAG systems") used to support conclusions about a broader domain (e.g., "all AI systems") without justification. | | Missing Premise | Medium | An argument that depends on an unstated assumption — the reasoning chain has a gap that must be explicitly acknowledged. | | Survivorship Bias | Medium | Evidence drawn only from successful cases or well-known examples while ignoring failures or counterexamples that would weaken the argument. | | Hedging Inconsistency | Low | Confidence language that shifts without justification — a claim described as "possible" in one section and "proven" in another without additional evidence. | For each logical issue, trace the specific premises and conclusion involved, explaining why the reasoning fails. ### Step 4 — Provenance and Integrity Assessment Evaluate the integrity of the document's sourcing chain and the traceability of its claims. | Signal | Severity | What to Look For | |--------|----------|------------------| | Citation Chain Break | High | A claim that cites Source A, but Source A attributes the claim to Source B, and Source B either does not exist or does not contain the claim — the citation chain is broken. | | Selective Quotation | High | A source quoted in a way that reverses or materially changes its meaning — omitting qualifiers, context, or contradicting sentences adjacent to the quoted passage. | | Self-Referential Loop | Medium | A document citing its own prior claims or outputs as evidence for new claims — creating circular validation. | | Version Mismatch | Medium | A reference to a specific version, edition, or revision of a standard or document that does not match the version actually cited or linked. | | Inaccessible Source | Low | A cited source behind a paywall, broken link, or restricted access — the claim cannot be independently verified by the reader. | ### Step 5 — Synthesize and Prioritize 1. **Aggregate findings** from all four steps into a unified severity-ranked list. 2. **Assess overall document reliability** using this rubric: | Rating | Criteria | |--------|----------| | **HIGH CONFIDENCE** | No Critical findings. Fewer than 3 High findings. Claims are well-sourced, logically consistent, and cross-referenced. Minor gaps are acknowledged by the document itself. | | **MODERATE CONFIDENCE** | No more than 1 Critical finding. Several High findings. The document's core thesis is supported but specific claims need correction or additional sourcing. | | **LOW CONFIDENCE** | Multiple Critical findings. Core claims are unsupported, contradicted, or logically inconsistent. The document requires substantial revision before it can be relied upon. | | **UNRELIABLE** | Pervasive Critical findings. Fabricated citations, systematic factual errors, or fundamental logical failures. The document should not be used as a basis for decisions. | 3. **Identify the highest-impact corrections** — the findings that, if left uncorrected, would most mislead a reader relying on this document. ## OUTPUT FORMAT ### Verification Summary ``` Document: [title or identifier] Domain: [inferred or stated domain] Claims Evaluated: [count] Sources Evaluated: [count] Overall Reliability: [HIGH CONFIDENCE | MODERATE CONFIDENCE | LOW CONFIDENCE | UNRELIABLE] ``` ### Findings by Layer For each finding, use the numbering convention `VER-N.M` where N = verification layer (1=Source Authority, 2=Factual Verification, 3=Logical Consistency, 4=Provenance) and M = finding number within that layer: ``` [VER-N.M] [CRITICAL|HIGH|MEDIUM|LOW] — [document location: section, paragraph, or line] Layer: [Source Authority | Factual Verification | Logical Consistency | Provenance] Claim: [The specific claim as stated in the document] Finding: [What is wrong — the specific error, gap, or inconsistency] Evidence: [What authoritative source or logical principle contradicts or undermines the claim] Remediation: [Specific correction — the accurate statement, the missing source, or the logical fix] ``` ### Verified Claims List claims that were checked and confirmed accurate — this is as important as listing errors, to establish what the reader can trust. ``` ### Confirmed - [Claim] — Verified against [source]. Accurate as stated. ``` ### Needs Human Review List cases where verification is ambiguous: - Claims in domains where the verifier lacks sufficient expertise to adjudicate - Claims where authoritative sources disagree with each other - Claims that are technically accurate but potentially misleading without additional context - Emerging or rapidly evolving areas where the "ground truth" is unstable ### Confidence Rationale One paragraph explaining the overall reliability rating — what drives the assessment and what would change it. ## STOP CONDITION When all claims in the document have been evaluated across all four layers, output the summary and stop. Do not modify the document. If no verification issues are found: > **No verification issues found** in the evaluated document. Claims are well-sourced, factually accurate within available evidence, logically consistent, and properly attributed. No corrections proposed. Do not fabricate findings to fill the report. ```` ## Example **Context:** Reviewing an AI-generated research summary on climate policy that cites multiple studies, quotes statistics, and draws conclusions about policy effectiveness. **Input:** ``` Document to verify: [paste AI-generated climate policy summary] Domain context: climate policy analysis Claimed sources: [paste reference list from the document] Trusted references: IPCC AR6, US EPA factsheets ``` **Expected Output:** ### Verification Summary ``` Document: Climate Policy Effectiveness Summary Domain: Climate policy analysis Claims Evaluated: 23 Sources Evaluated: 12 Overall Reliability: MODERATE CONFIDENCE ``` ### Findings by Layer ``` [VER-1.1] CRITICAL — Section 2, paragraph 3 Layer: Source Authority Claim: "According to Zhang et al. (2024) in Nature Climate Change, carbon capture efficiency has reached 94% in industrial deployments." Finding: Probable fabricated citation. No paper by Zhang et al. (2024) matching this topic found in available records of Nature Climate Change. Verify against the journal's live index if access is available. Evidence: Nature Climate Change search for 2024 issues returns no matching article. Remediation: Remove citation. If the 94% figure is accurate, locate the actual source. The closest verified figure is 90% capture rate from the Global CCS Institute 2023 Status Report. [VER-2.1] HIGH — Section 4, paragraph 1 Layer: Factual Verification Claim: "Global renewable energy investment exceeded $500 billion in 2023, a 30% increase over 2022." Finding: Distorted statistic. The $500 billion figure is approximately correct (BloombergNEF reports $495B), but the year-over-year increase was approximately 17%, not 30%. Evidence: BloombergNEF Global Energy Investment Report 2024. Remediation: Correct to "approximately $495 billion in 2023, a roughly 17% increase over 2022." [VER-3.1] HIGH — Section 5, conclusion Layer: Logical Consistency Claim: "Because renewable costs have fallen below fossil fuels in most markets, the energy transition is economically inevitable." Finding: Non sequitur. Cost competitiveness is a necessary but not sufficient condition for transition. The argument omits grid integration costs, storage requirements, political factors, existing infrastructure lock-in, and financing constraints in developing economies. Evidence: IPCC AR6 WGIII Chapter 6 discusses multiple barriers beyond levelized cost. Remediation: Qualify the conclusion: "Cost competitiveness is a significant driver, but the transition pace also depends on grid integration, storage, policy, and financing factors." ``` ### Confirmed - Global temperature increase of approximately 1.1C above pre-industrial levels — Verified against IPCC AR6 WGI SPM. Accurate as stated. - Paris Agreement target of limiting warming to 1.5C — Verified against UNFCCC Paris Agreement text. Accurate. ### Needs Human Review - Section 3 claim about methane reduction potential of 45% by 2030 — multiple credible sources give ranges from 30-50%. The specific 45% figure appears in the Global Methane Pledge but is an aspirational target, not a proven achievable figure. Human judgment needed on whether to present as target vs. projection. ### Confidence Rationale Rated MODERATE CONFIDENCE because the document's core thesis (renewable energy progress is substantial) is supported by evidence, but one fabricated citation (VER-1.1) and one distorted statistic (VER-2.1) undermine specific claims. The logical leap in the conclusion (VER-3.1) oversimplifies a complex transition. Correcting these three findings would raise confidence to HIGH. ## Expected Results - A structured verification report covering all four layers (source authority, factual accuracy, logical consistency, provenance) - Each finding traces: claim → error type → contradicting evidence → specific remediation - Confirmed claims listed alongside errors — establishing what the reader can trust - Overall reliability rating with transparent rationale - Ambiguous cases flagged for human judgment rather than falsely resolved - Document is NOT modified — advisory only ## Variations **Source Authority Only:** Use only Step 1 when reviewing a reference list or bibliography for citation validity — e.g., auditing an AI-generated literature review for hallucinated citations. **Factual Spot-Check:** Use only Step 2 on specific claims or sections rather than the full document — for targeted verification of statistics, dates, or attributions. **Logical Consistency Only:** Use only Step 3 when reviewing an argument structure (legal brief, policy proposal, technical RFC) where the facts are assumed correct but the reasoning must be checked. **AI Output Audit:** Apply all four steps with heightened attention to Step 1 (fabricated citations are the most common LLM hallucination in research contexts) and Step 2 (distorted statistics are the second most common). **Adversarial Verification:** Combine with `prompt-task-red-team-review.md` methodology — actively try to disprove each claim rather than merely checking it. For each claim, search specifically for contradicting evidence before searching for supporting evidence. This mirrors the Analysis of Competing Hypotheses approach where the strongest hypothesis is the one with the least disconfirming evidence. ## Notes The key insight from the underlying research is that verification operates across multiple independent dimensions — a claim can be well-sourced but logically inconsistent, or logically sound but factually wrong, or factually correct but misattributed. Checking only one dimension creates false confidence. The four-layer protocol ensures each dimension is examined independently. **Source authority is not truth.** A claim from an authoritative source can still be wrong. Authority assessment (Step 1) evaluates whether the source *should* be trusted; factual verification (Step 2) evaluates whether the specific claim *is* accurate. These are independent checks. **Absence of evidence is not evidence of absence.** When a claim cannot be verified (Unverifiable status in Step 2), the correct response is to flag the evidence gap and assess the claim's plausibility — not to mark it as false. Reserve the "Factual Error" finding for claims that are demonstrably contradicted by evidence. **Confirmed claims matter.** Listing what is verified is as important as listing errors. A reader needs to know which parts of the document they can rely on, not just which parts they cannot. **Calibrate to domain norms.** A medical research summary requires peer-reviewed citations for every clinical claim. A blog post about cooking techniques does not. Adjust citation expectations to the document's domain and audience — but never relax the standard for factual accuracy itself. **The verifier is not ground truth.** This diagnostic is performed by an LLM — a probabilistic model subject to the same hallucination risks the protocol is designed to detect. The verifier may "confirm" a fabricated citation by hallucinating that it found the source, or "verify" a statistic by generating a plausible-sounding confirmation that is itself wrong. Treat the diagnostic output as a structured triage, not independent proof. Claims marked "Confirmed" mean "consistent with the verifier's available knowledge," not "independently proven." Users must spot-check Critical and High findings against actual sources. When the agent has tool access (web search, file read), it should use tools to verify claims against live sources rather than relying on parametric knowledge alone. ## References - [research-paper-verification-protocols-deterministic-architectures.md](research-paper-verification-protocols-deterministic-architectures.md) — the research synthesis this prompt operationalizes - [prompt-task-systematic-bias-audit-and-mitigation.md](prompt-task-systematic-bias-audit-and-mitigation.md) — complementary: detects bias where this prompt detects factual error - [prompt-task-red-team-review.md](prompt-task-red-team-review.md) — complementary: adversarial review for code and systems ### Source Research - IFCN Code of Principles — journalistic fact-checking standards - SIFT method (Mike Caulfield) — lateral reading heuristic for source evaluation - CRAAP test — source credibility evaluation framework - Analysis of Competing Hypotheses (Richards Heuer, CIA) — disconfirmation-focused hypothesis evaluation - Daubert Standard — legal criteria for evaluating expert testimony reliability - Knowledge Graph atomic decomposition — claim → triplet extraction for structured verification - FoVer pipeline — neurosymbolic logical consistency verification ## Version History - 1.1.0 (2026-03-25): RO5U review fixes — added verifier limitations note (LLM-as-judge circularity), claim scoping guidance for long documents, lateral reading definition, VER-N.M numbering convention, zero-citation document handling, softened example fabrication finding. - 1.0.0 (2026-03-25): Initial extraction from research-paper-verification-protocols-deterministic-architectures.md. Operationalizes the four-layer verification architecture (source authority, factual verification, logical consistency, provenance) into a five-step diagnostic protocol.