Ai Rag

---
name: ai-rag
description: "Designs RAG retrieval for LLM grounding: chunking, BM25+vector hybrid, reranking, GraphRAG. Use when adding embeddings to SQLite FTS5 search or running held-out retrieval eval."
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.3"
last_validated: 2026-07-11
---

# RAG & Retrieval Engineering

**Operating posture**

- Choose the retrieval mode before tuning chunk size. A vector index is not the default answer to every knowledge problem.
- Separate retrieval quality from answer quality and evaluate both.
- Treat retrieved text, tool responses, and MCP resources as untrusted input.
- Prefer primary sources for vendor or framework recommendations; volatile facts must be verified live.
- Treat OpenTelemetry GenAI semantic conventions as useful but still evolving.
- **Context budgets:** measure chunks with the production tokenizer and reserve space for instructions, evidence, and output; model releases may share a tokenizer, but that does not establish the same retrieval-quality budget.
- **Managed retrieval is a real option:** provider server-side web-search tools and hosted file-search are API-native retrieval surfaces — evaluate them before building a self-hosted RAG stack. Tool identifiers are versioned and change; look up the current tool name and version in the provider's API docs at use time. See [references/managed-retrieval-vs-self-hosted.md](references/managed-retrieval-vs-self-hosted.md).
- **Retrieval may not be the right answer at all:** if the corpus is small and stable, CAG / long-context / fine-tune may be cheaper and more reliable than RAG. Run the decision rubric in [`../ai-context-layer/references/retrieve-vs-preload-vs-finetune.md`](../ai-context-layer/references/retrieve-vs-preload-vs-finetune.md) before building a RAG pipeline.

**Scope note**: For generation-prompt structure and output contracts after retrieval, use [ai-prompt-engineering](../ai-prompt-engineering/SKILL.md).

**Implementation note**: This skill owns retrieval theory and evaluation concepts. For vector-brain builds with paste-ready SQL, pgvector assets, manifests, ingest scripts, and agent retrieval tool contracts, use [ai-vector-brain](../ai-vector-brain/SKILL.md).

## Workflow

1. **Pin the authority source and freshness budget first.** Before touching a chunker or embedder, decide whether truth lives in documents, live tools, or a structured store, and how stale an answer is allowed to be. This decision, not chunk size, determines the architecture.
2. **Run the retrieval mode decision** (see Retrieval Choice Framework below); use [system design](assets/design/rag-system-design.md) to record the chosen mode and alternatives against freshness, traceability, and latency.
3. **Write the corpus contract before indexing**: use the [metadata schema](assets/indexing/template-metadata-schema.md) and [index configuration](assets/indexing/template-index-config.md) for tenant/ACL scope, source, freshness, and deletion propagation. For ingestion, load [basic](assets/chunking/template-basic-chunking.md), [code](assets/chunking/template-code-chunking.md), or [long-document chunking](assets/chunking/template-long-doc-chunking.md) for the actual corpus shape.
4. **Build a lexical baseline and golden eval before adding infrastructure.** Use [golden retrieval cases](assets/eval/golden-retrieval-cases.jsonl) and `scripts/retrieval_eval.py` to measure recall@k, MRR, and nDCG. `scripts/exact_search_baseline.py` is exhaustive vector search for checking ANN loss; its hash embedder is a lexical smoke fixture, not a learned embedding benchmark.
5. **Route retrieval legs and pick the post-retrieval mode.** Legs: semantic, lexical (BM25 plus exact match, so document, clause and ticket IDs hit by string), relational (graph traversal), structured (registry/SQL for "current version / owner / who approved / in force"). Output modes: return cited evidence (dedup + rerank), compute a result (aggregates, temporal predicates), or synthesise an answer. Default for compliance/regulated corpora is evidence plus a citation-support check, with synthesis left to the calling agent; deviate only when the consumer cannot reason over evidence, and then add a citation-support gate. Detail, including the lexical-only FTS5/BM25 upgrade path: [retrieval-choice-framework.md](references/retrieval-choice-framework.md#8-retrieval-legs-and-post-retrieval-mode).
6. **Add one layer for the measured failure.** Start from [retrieval](assets/retrieval/template-retrieval-pipeline.md); load [hybrid search](assets/retrieval/template-hybrid-search.md) for lexical/semantic misses, [reranking](assets/retrieval/template-reranking.md) for ranking misses, or [query rewriting](assets/query/template-query-rewrite.md) for query mismatch. Use the [template index](references/quick-start-guide.md#template-index) only for another selected mode.
7. **Keep retrieval and answer eval as separate gates.** Use [context packing](assets/context/template-context-packing.md) and [grounding](assets/context/template-grounding.md) to preserve evidence IDs, then [answer evaluation](assets/eval/template-rag-eval.md) to check faithfulness and abstention. Citation-ID checks prove target existence; they cannot prove that a claim follows from the cited text.
8. **Instrument before you scale**: evidence IDs, retrieval-mode traces, and confidence scores must exist end to end so a bad answer can be traced back to a specific retrieval decision (see observability-tracing-contract.md).
9. **Treat the eval set as a regression gate, not a one-time report.** Re-run it before every chunker, embedder, index, or reranker change, and before any vendor swap (embedding model, reranker, vector DB) — silent regressions are the most common way "working" RAG systems degrade.

## Quick Reference

| Need | Recommended path | Use when | Avoid when |
|------|------------------|----------|------------|
| Small corpus fits in context | Long-context prompt or lightweight search | Low update rate, low audit burden | Corpus is large, fast-changing, or citation-critical |
| Provider-managed retrieval | Provider web-search server tool (look up the current versioned tool ID) or hosted file search | Fresh web data with citations, fast start, no infra to operate | You need residency controls, custom ranking internals, or corpus isolation beyond provider defaults |
| Provider-managed file/doc retrieval | OpenAI file-search (Responses API) | Standard docs/Q&A, limited infra appetite | You need custom ranking, strict residency controls, or deep retrieval tuning |
| Provider-managed doc retrieval on AWS | AWS Bedrock Knowledge Bases | AWS-native, want managed ingestion + embedding + retrieval with IAM/residency controls | You need cross-cloud portability or custom ranking internals → references/aws-bedrock-knowledge-bases.md |
| Source of truth is tools/APIs | Tool-first or MCP retrieval | Data lives in SQL, CRM, ticketing, SaaS APIs, or internal tools | You actually need semantic retrieval over large prose corpora |
| Website or research ingestion | Crawl/extract first, then index | The source starts as websites, reports, or broad web research | You only need one-off browsing instead of a maintained retrieval system |
| Whole-document or thematic questions | RAPTOR-style summary tree (recursive cluster + summarise, retrieve across levels) | Answers need synthesis across a long doc or many sections, no entity graph exists | Questions are passage-level lookups, or the corpus churns faster than the tree can be rebuilt → references/retrieval-patterns.md §6 |
| Exact joins or aggregations | SQL/graph retrieval | Questions need filters, joins, counts, or relationship traversal | Natural-language corpus lookup is the main problem |
| Standard knowledge retrieval | Hybrid search + rerank | Mixed lexical + semantic queries, high recall, controllable latency | A simpler hosted or tool-first option already solves the problem |
| Generated repo/context hub corpus | Graph-bounded hybrid (graph scopes, vector recalls, rerank) | A compiled multi-repo/context hub with an existing knowledge/code graph | The corpus has no graph, or simple semantic lookup already passes eval |
| High-precision retrieval | Late interaction / multivector retrieval | Near-duplicate docs, subtle wording differences, PDF pages, multilingual precision | Latency budget is tight and BM25+dense+reranking is already sufficient |
| PDFs, tables, diagrams | Multimodal document retrieval | OCR loses structure or layout meaning matters | Plain text extraction is already high quality |
| Production readiness proof | Golden eval + exact baseline + traces | You need to prove quality, not just describe architecture | One-off exploratory research with no durable corpus |

## Retrieval Choice Framework

```text
Need external knowledge?
  ├─ No -> Use direct prompting / standard prompt engineering
  └─ Yes
      ├─ Data already lives behind tools, APIs, SQL, or SaaS?
      │   └─ Start with tool-first or MCP retrieval
      │
      ├─ Corpus fits comfortably in model context and changes slowly?
      │   └─ Try long-context or hosted file search before custom indexing
      │
      ├─ Need joins, counts, or relationship traversal?
      │   └─ Use SQL, graph, or graph+vector hybrid retrieval
      │       (paradigm choice: see ../software-database-design/SKILL.md#technology-selection)
      │
      ├─ Need fresh web data or provider-managed file search without self-hosting?
      │   └─ Evaluate managed retrieval first: provider web-search server tools,
      │      hosted file-search, or other provider-native tools before building a custom stack
      │      (see references/managed-retrieval-vs-self-hosted.md for decision criteria)
      │
      ├─ Need retrieval over prose or mixed documents?
      │   └─ Use sparse+dense hybrid as the default baseline
      │
      ├─ Retrieval misses subtle matches or page-level structure?
      │   └─ Add late interaction, multivector, or multimodal retrieval
      │
      ├─ Precision still poor?
      │   └─ Add reranking, filters, query rewriting, and stronger evals
      │
      └─ Same queries keep re-running synthesis? Need a reviewable knowledge asset?
          └─ Switch from retrieval to knowledge compilation (P7 in ai-context-layer)
              Route to ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md
              Blocks A15 (RAG re-run per turn instead of compiled knowledge)
```

## Core Concepts

- **Authority source**: define whether truth comes from retrieved documents, live tools, structured databases, or a hybrid of them.
- **ACL invariant (this skill owns it)**: enforce the caller's ACL/tenant scope at candidate generation — before fusion, before the reranker, and before any cache. Never filter after the reranker or after generation: a post-reranker filter still sends unauthorized text through the reranker, the response cache, and any third-party reranker API, and it silently collapses recall because top-k was chosen without the scope in mind. Every retrieval pipeline diagram, SQL example, and checklist in this skill and in `ai-vector-brain` must show the ACL filter at or before candidate generation.
- **Trust boundary**: retrieved chunks, uploaded files, MCP resources, tool results, and web pages are untrusted until validated.
- **Freshness model**: set staleness budget, invalidation triggers, and deletion propagation rules.
- **Generated-context corpus**: a compiled multi-repo/context hub is a valid corpus; graph-bounded build path is [../ai-vector-brain/references/dev-context-hub-vector-recipe.md](../ai-vector-brain/references/dev-context-hub-vector-recipe.md).
- **Evidence contract**: return stable evidence IDs, source metadata, and enough context for later citation verification.
- **Two eval planes**: retrieval relevance and answer faithfulness are separate systems and must be measured separately.

## When NOT to Use This Skill

| Need | Route to |
|------|----------|
| Product or site search for end users (engine choice, facets, autocomplete, merchandising, reindexing, search analytics) | [`../software-search/SKILL.md`](../software-search/SKILL.md) |
| Personal note collection where an LLM-maintained `INDEX.md` passes retrieval eval | [`../docs-notes-retrieval/SKILL.md`](../docs-notes-retrieval/SKILL.md) |
| Paste-ready SQL, DDL, pgvector indexes, or backend loader scripts | [`../ai-vector-brain/SKILL.md`](../ai-vector-brain/SKILL.md) |
| Building a repo/docs/compliance vector brain end to end | [`../ai-vector-brain/SKILL.md`](../ai-vector-brain/SKILL.md) |
| App context architecture, memory lifecycle, and grounding boundaries | [`../ai-context-layer/SKILL.md`](../ai-context-layer/SKILL.md) |
| Agent topology and tool orchestration | [`../ai-agents/SKILL.md`](../ai-agents/SKILL.md) |
| Bot UX, conversation flows, and escalation | [`../ai-bot-builder/SKILL.md`](../ai-bot-builder/SKILL.md) |
| On-device iOS retrieval-stitch composer (Apple Foundation Models, `@Generable`, retrieval over local chunks with no-cloud guarantee) | [`../software-ios-ai-engine/SKILL.md`](../software-ios-ai-engine/SKILL.md) |
| End-to-end natural conversational iOS surface composed with this skill + `ai-context-layer` + `ai-vector-brain`, Path A (Foundation Models) and Path B (vector-DB-only) | [`../software-ios-ai-engine/references/composition-with-rag-context-vector.md`](../software-ios-ai-engine/references/composition-with-rag-context-vector.md) |
| Cross-platform natural conversation (iOS / Android / web / Telegram-Discord-WhatsApp-Slack bots / voice / backend) with or without on-device models | [`../ai-context-layer/references/conversational-surfaces-cross-platform.md`](../ai-context-layer/references/conversational-surfaces-cross-platform.md) |

## Retrieval Error Localization Gate

Before changing chunking, embeddings, reranking, or the generator, label failures as corpus absence, ingest loss, candidate miss, ranking miss, context packing loss, citation mismatch, or answer-generation error. Run the generator once with oracle evidence and run retrieval against gold evidence IDs. If oracle context still fails, fix the answer contract or model path; if gold evidence never enters the candidate set, fix ingestion or retrieval. Tune only the stage that owns the measured loss and retain the previous stage as the control.

## Known Traps

- tuning chunk size, embedder, or reranker before deciding whether retrieval is even the right architecture
- reusing chunk-size or token-budget heuristics measured on a different model without re-measuring
- assuming iterative/agentic multi-hop retrieval improves every query distribution; compare against the fixed retrieval baseline on the same multi-hop cases before adding loop overhead
- shipping vector/full-text migrations before verifying the target database supports the required extensions, text-search configs, generated columns, and index operators
- mixing DDL, retrieval-function changes, and large content seeds in one migration so a search-feature failure blocks safe corpus rollout
- upgrading a small passing lexical corpus to embeddings or search infrastructure just because it is on the long-term roadmap
- reindexing synthesized reports or summaries as if they were authoritative primary evidence
- picking an embedding or reranker vendor purely off a leaderboard snapshot without checking license terms (many open-weight rerankers ship non-commercial licenses) and ownership stability (reranker/embedding vendors are consolidating via acquisition; confirm the model's roadmap and support commitment survive a vendor change before building reindex-heavy dependencies on it)
- treating retrieval recall problems and answer-faithfulness problems as one metric
- relying on hosted retrieval defaults while assuming ACL, residency, freshness, and deletion semantics are handled
- filtering ACL/tenant scope after the reranker or after generation instead of at candidate generation (pre-fusion) — see the ACL invariant in Core Concepts
- mixing tool outputs, crawled pages, uploaded files, and MCP resources without normalizing provenance and trust boundaries

## Common Anti-Patterns

See [references/quick-start-guide.md](references/quick-start-guide.md) for detailed root-cause analysis of each anti-pattern below.

- vector database first, source-of-truth model later (A2 in [`../ai-context-layer/references/anti-patterns-catalog.md`](../ai-context-layer/references/anti-patterns-catalog.md))
- agentic retrieval loops for straightforward lookup tasks
- no freshness or invalidation model for mutable corpora
- citation formatting without evidence-ID verification (A13 — provenance as optional metadata)
- cross-tenant or cross-sensitivity indexing with retrieval-time filtering bolted on later (A10)
- **RAG re-run per turn instead of compiled knowledge (A15)** — fix: switch to knowledge compilation (P7 in [`../ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md`](../ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md))

## Vendor Recommendation Protocol

When users ask for "best" tools, models, or frameworks:

1. Read [data/sources.json](data/sources.json) and start from sources marked `add_as_web_search: true`.
2. Verify claims against current primary docs, release notes, or official benchmarks.
3. Prefer durable guidance: retrieval mode choice, operational tradeoffs, integration constraints, evaluation and rollback plan.
4. If live browsing is unavailable, state that rankings, prices, and benchmark claims are unverified.

## Research & Ingestion Pipelines

See [references/research-and-ingestion-patterns.md](references/research-and-ingestion-patterns.md) for crawl/extract (Firecrawl-style), research loops (GPT Researcher-style), recurring data pipelines (dlt-style), and CLI evaluation workflows (simonw/llm-style). Core rule: separate crawl/extract from rank/retrieve, keep raw capture and chunks as distinct artifacts.

## Scripts

`check_sources.py` — validate sources.json · `retrieval_eval.py` — recall@k/MRR/nDCG · `check_citation_support.py` — evidence-ID verification · `generate_synthetic_rag_testset.py` — testset scaffolds · `late_interaction_eval.py` — ColBERT/ColPali offline eval (no inference runner) · `exact_search_baseline.py` — exact cosine/dot baseline · `hybrid_rrf_demo.py` — BM25 (k1/b) + vector + RRF smoke test

## Navigation

### References (this skill)

Architecture & choice: [retrieval-choice-framework.md](references/retrieval-choice-framework.md) · [managed-retrieval-vs-self-hosted.md](references/managed-retrieval-vs-self-hosted.md) · [pipeline-architecture.md](references/pipeline-architecture.md) · [aws-bedrock-knowledge-bases.md](references/aws-bedrock-knowledge-bases.md)

Corpus & chunking: [chunking-strategies.md](references/chunking-strategies.md) · [chunking-patterns.md](references/chunking-patterns.md) · [index-selection-guide.md](references/index-selection-guide.md) · [embedding-model-guide.md](references/embedding-model-guide.md) · [research-and-ingestion-patterns.md](references/research-and-ingestion-patterns.md) · [pdf-heavy-retrieval-playbook.md](references/pdf-heavy-retrieval-playbook.md)

Retrieval patterns: [retrieval-patterns.md](references/retrieval-patterns.md) · [vector-search-patterns.md](references/vector-search-patterns.md) · [hybrid-fusion-patterns.md](references/hybrid-fusion-patterns.md) · [bm25-tuning.md](references/bm25-tuning.md) · [graph-rag-patterns.md](references/graph-rag-patterns.md) · [contextual-retrieval-guide.md](references/contextual-retrieval-guide.md) · [advanced-rag-patterns.md](references/advanced-rag-patterns.md) · [agentic-rag-patterns.md](references/agentic-rag-patterns.md)

Ranking & query: [ranking-pipeline-guide.md](references/ranking-pipeline-guide.md) · [query-rewriting-patterns.md](references/query-rewriting-patterns.md) · [backend-comparison-fixtures.md](references/backend-comparison-fixtures.md)

Grounding & eval: [grounding-checklists.md](references/grounding-checklists.md) · [confidence-scoring.md](references/confidence-scoring.md) · [abstention-recipe.md](references/abstention-recipe.md) · [rag-evaluation-guide.md](references/rag-evaluation-guide.md)

Ops & debugging: [observability-tracing-contract.md](references/observability-tracing-contract.md) · [retrieval-debugging-runbook.md](references/retrieval-debugging-runbook.md) · [rag-troubleshooting.md](references/rag-troubleshooting.md) · [rag-caching-patterns.md](references/rag-caching-patterns.md) · [multilingual-domain-patterns.md](references/multilingual-domain-patterns.md) · [security-red-team-cases.md](references/security-red-team-cases.md)

Product-search eval, learning to rank, click models, and search SLOs live in software-search: [search-evaluation-guide.md](../software-search/references/search-evaluation-guide.md) (nDCG/MRR judged sets, reusable for retrieval eval) · [learning-to-rank-pipeline.md](../software-search/references/learning-to-rank-pipeline.md) · [click-models-and-bias-correction.md](../software-search/references/click-models-and-bias-correction.md) · [user-feedback-learning.md](../software-search/references/user-feedback-learning.md) · [distributed-search-slos.md](../software-search/references/distributed-search-slos.md) · [search-debugging.md](../software-search/references/search-debugging.md)

Onboarding: [quick-start-guide.md](references/quick-start-guide.md) - workflow, full template index, detailed anti-patterns · [wiki-grounded-retrieval.md](references/wiki-grounded-retrieval.md)

### Templates (this skill)

Eval data: [backend comparison template](assets/eval/backend-comparison-template.json) · [golden retrieval predictions example](assets/eval/golden-retrieval-predictions.example.jsonl) · [security red-team cases](assets/eval/security-redteam-cases.jsonl) · [RAG testset](assets/eval/template-rag-testset.jsonl) · [search testset](assets/eval/template-search-testset.jsonl) · [search evaluation template](assets/eval/template-search-eval.md)

Ranking: [ranking pipeline](assets/ranking/template-ranking-pipeline.md) · [reranker configuration](assets/ranking/template-reranker.md)

Retrieval variants: [graph + vector](assets/retrieval/template-graph-hybrid-retrieval.md) · [multimodal document](assets/retrieval/template-multimodal-document-retrieval.md) · [multivector](assets/retrieval/template-multivector-retrieval.md) · [tool-first](assets/retrieval/template-tool-first-retrieval.md)

Search index configs: [BM25](assets/search/template-bm25-config.md) · [HNSW](assets/search/template-hnsw-config.md) · [hybrid search](assets/search/template-hybrid-config.md) · [IVF](assets/search/template-ivf-config.md)
### Cross-Skill References

- [`../ai-context-layer/references/anti-patterns-catalog.md`](../ai-context-layer/references/anti-patterns-catalog.md) - A2, A10, A13, A15 anti-patterns referenced throughout this skill
- [`../ai-context-layer/references/retrieve-vs-preload-vs-finetune.md`](../ai-context-layer/references/retrieve-vs-preload-vs-finetune.md) - Decision rubric: RAG vs long-context vs fine-tune
- [`../ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md`](../ai-context-layer/references/knowledge-compilation-and-wiki-pattern.md) - P7 pattern for compiled knowledge (A15 fix)
- [`../ai-vector-brain/references/dev-context-hub-vector-recipe.md`](../ai-vector-brain/references/dev-context-hub-vector-recipe.md) - Graph-bounded build path for generated-context corpora
- [`../software-ios-ai-engine/references/composition-with-rag-context-vector.md`](../software-ios-ai-engine/references/composition-with-rag-context-vector.md) - iOS conversational RAG composition

### External Sources

- [data/sources.json](data/sources.json) - Primary-source catalog for live verification

## Related Skills

Gate before invoking: each foundation has a `When to Apply` / `When to Skip` section.

- **[ai-evals](../ai-evals/SKILL.md)** - LLM-judge bias control and reproducibility for faithfulness/answer evals
- **[ai-prompt-engineering](../ai-prompt-engineering/SKILL.md)** - Retrieval-grounded prompt contracts and structured outputs
- **[ai-agents](../ai-agents/SKILL.md)** - Agent orchestration and tool workflows
- **[ai-mlops](../ai-mlops/SKILL.md)** - Deployment, monitoring, and governance
- **[ai-llm-inference](../ai-llm-inference/SKILL.md)** - Latency, batching, caching, and cost controls
- **[docs-notes-retrieval](../docs-notes-retrieval/SKILL.md)** - Local note-vault and notebook-export packaging for retrieval
- **[foundations-information-theory](../foundations-information-theory/SKILL.md)** - Entropy, mutual information, and KL divergence for chunk scoring, MMR diversity, and drift detection

## Learnings Loop

When prior decisions or pitfalls are relevant, consult `learnings.consolidated.md` if present; use `learnings.md` only for needed history or as the available fallback. Otherwise skip both. After applying it, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.