--- name: kb-validate-citations description: Spot-check citations in library files for obvious hallucinations. Resolves DOIs against the DOI registry, validates arXiv IDs, checks the format of journal references. Soft-fails on transient lookup failures rather than blocking. Recommended after kb-ingest and as part of periodic library hygiene. disable-model-invocation: false argument-hint: "[]" --- # Validate Knowledge Base Citations Catch obvious citation hallucinations in library files. Deep research engines occasionally fabricate DOIs, attribute findings to the wrong author, or invent journal names. Manual spot-checking 10-20% of citations remains necessary; this skill automates the obvious failure cases so manual review can focus on subtler issues. ## Argument Optional file path or glob to scope validation. Default: all library files in `library/**/*.md` excluding the index files. Examples: - `library/dora-metrics.md` — single file - `library/dora-*.md` — files matching a pattern - (no argument) — all library files ## Preflight - Verify the knowledge base is configured. - Verify network access (DOI resolution requires HTTPS to https://doi.org). ## Steps ### 1. Discover library files Use Glob to find files matching the scope. Exclude: - `library/_shelf-index.md` - `library/_index.md` - `library/log.md` - `library/raw/**` ### 2. Extract citations from each file For each library file, parse the `## Key References` section and the inline citations in `## Core Findings`. Identify three citation types: **DOI citations** — strings matching the DOI regex: `10\.\d{4,9}/[-._;()/:A-Z0-9]+` (case-insensitive). DOIs are the strongest signal because they're machine-resolvable. **arXiv IDs** — strings matching `arXiv:\d{4}\.\d{4,5}` or `arxiv\.org/abs/\d{4}\.\d{4,5}`. **Journal references** — anything matching the pattern `. (). "." <Journal Name>, <volume>(<issue>), <pages>.` or similar bibliographic format. Best-effort extraction; not all citations follow a standard format. ### 3. Validate each citation type #### DOI resolution For each DOI, attempt to resolve it via: ```bash curl -sLI "https://doi.org/<doi>" -o /dev/null -w "%{http_code}" ``` Expected: HTTP 302 or 200 (redirected to the publisher landing page). Failure: HTTP 404 (DOI does not exist) or 500 (DOI registry error). Classify each result: - **OK**: 200/302 - **NOT FOUND**: 404 — flag as likely hallucinated - **TRANSIENT**: 500/timeout/network failure — soft-fail with "could not verify" Rate limit: pause 100ms between DOI lookups to avoid hammering doi.org. If rate-limited (429), back off and retry once with a longer pause. #### arXiv ID validation For each arXiv ID: 1. Validate format: must be `YYMM.NNNNN` (4-5 digit suffix, 2007 or later) 2. Optionally verify existence: ```bash curl -sI "https://arxiv.org/abs/<id>" -o /dev/null -w "%{http_code}" ``` 3. Expected: HTTP 200 4. NOT FOUND: HTTP 404 — flag as likely hallucinated #### Journal reference format For each journal reference, perform structural checks only (do not attempt to verify the journal exists in any registry — too many legitimate journals to maintain a list): - Year is in the past or current year (no 2099 typos) - Volume and issue, when present, are integers - Page numbers, when present, follow `<start>-<end>` format - Author list looks plausible (not a single word, not all caps unless it's an organisation) These are weak checks — they catch obvious typos but not real hallucinations. The strong signals are DOI and arXiv resolution. ### 4. Produce the report Output format: ```markdown # Citation Validation Report **Date:** YYYY-MM-DD **Scope:** all files (or pattern) **Files scanned:** N **Citations checked:** M ## Summary | Result | Count | |---|---| | OK (resolved) | A | | NOT FOUND (likely hallucinated) | B | | TRANSIENT (could not verify) | C | | Format issues | D | **Total issues:** B + D (transient failures are not counted as issues — they're informational) --- ## NOT FOUND citations ### library/dora-metrics.md - DOI 10.1109/TSE.2010.99999 — DOI registry returned 404 Context: "Madeyski (2010) reports defect reduction of..." Recommendation: verify the citation manually; consider whether the DOI was hallucinated (repeat for each NOT FOUND) ## TRANSIENT failures (informational) ### library/dora-metrics.md - DOI 10.1109/TSE.2010.5544067 — registry timeout, not validated Recommendation: re-run kb-validate-citations later; do not act on this (repeat for each TRANSIENT) ## Format issues ### library/dora-metrics.md - "Forsgren (3024)" — year appears to be a typo Recommendation: check the original source (repeat) --- ## Recommended actions 1. Manually verify B citations marked NOT FOUND — these are the highest-confidence hallucination candidates 2. Re-run validation once C transient failures should have cleared 3. Fix D format issues ``` ### 5. Append to log.md (if present) ```markdown ## [YYYY-MM-DD] validate-citations | <scope> Files scanned: N Citations checked: M NOT FOUND: B TRANSIENT: C Format issues: D ``` ### 6. Exit code - Exit 0 if no NOT FOUND results AND no format issues (transient failures don't count) - Exit 1 if any hard failures (NOT FOUND or format issues) This lets the skill be wired into pre-push validation as an opt-in gate. ## What this skill does NOT do - It does not verify that the cited paper actually says what the library file claims it says (LLM hallucination of *content*, not metadata — beyond automated reach; manual spot-checking remains necessary) - It does not modify library files to fix issues - It does not block on transient failures — soft-fail and report - It does not validate non-academic citations (blog posts, GitHub repos, white papers without DOIs) ## Errors - **No knowledge base configured** — run `/sdlc-knowledge-base:kb-init` first - **No network access** — fail with explanation; the skill cannot work offline - **DOI registry persistently down** — soft-fail all DOIs as TRANSIENT and recommend re-running later - **Rate limited persistently** — back off, then partial-fail with a "ran out of attempts" note for unresolved DOIs ## Example ``` /sdlc-knowledge-base:kb-validate-citations # Citation Validation Report **Date:** 2026-04-08 **Scope:** all files **Files scanned:** 22 **Citations checked:** 187 ## Summary | Result | Count | |---|---| | OK | 178 | | NOT FOUND | 2 | | TRANSIENT | 5 | | Format issues | 2 | Total issues: 4 (report continues...) ``` --- Source attribution: SteveGJones Original project: https://github.com/SteveGJones/ai-first-sdlc-practices/blob/631a626ff044f069e445d40777a0951e491a8132/plugins/sdlc-knowledge-base/skills/kb-validate-citations/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2024 Steve Jones Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: SteveGJones Original project: https://github.com/SteveGJones/ai-first-sdlc-practices/blob/631a626ff044f069e445d40777a0951e491a8132/plugins/sdlc-knowledge-base/skills/kb-validate-citations/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Education & Research Roles: Academic Writer, Data Analyst, Researcher Specialties: API Tools: Doi Registry Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: fe385f53ee0aa5cdb0870d745a1a1a0db0cba8c79a1283bd2ac3e486977006d7