Build a production rag pipeline from chunking through generation. Covers embedding model selection, vector databases, hybrid retrieval, reranking, and securing the generation layer.
Most RAG tutorials end at "it works in a notebook." Production is where the problems start: retrieval returns plausible but wrong passages, costs scale unpredictably, and the generation layer leaks data nobody intended to expose.
This guide walks through each phase of a rag pipeline, from document chunking to a secured generation layer, with the specific numbers and trade-offs that matter when you ship to real users.
A production rag pipeline has four sequential phases:
Each phase constrains the next. Bad chunks produce bad embeddings. Bad embeddings produce irrelevant retrieval. Irrelevant retrieval produces hallucinated answers. The pipeline metaphor is literal: garbage propagates forward.
Weaviate's 2025 benchmark measured up to a 9% recall gap between the best and worst chunking methods on the same corpus, using the same embedding model and retriever. Chunking is the most underestimated step. Teams spend days evaluating embedding models and seconds choosing a chunk size.
Chunk size depends on query patterns:
| Query type | Optimal chunk size | Example |
|---|---|---|
| Factoid (names, dates, specific values) | 256–512 tokens | "What is the API rate limit?" |
| Analytical (comparisons, explanations) | 512–1,024 tokens | "How does pricing compare across tiers?" |
| Mixed workloads | 400–512 tokens (starting point) | General knowledge base |
A 500-token chunk should use 50–100 tokens of overlap with adjacent chunks. Without overlap, information that falls on a chunk boundary gets split across two segments, and neither segment contains enough context for the retriever to match it. Overlap creates redundancy, but the alternative is silent retrieval misses.
Chroma's research showed recursive character splitting delivers 85–90% recall at 400 tokens. Semantic chunking, which splits by meaning rather than character count, reaches 91–92% recall. That 2–3 percentage point improvement costs an embedding call for every sentence during the chunking step itself.
For most teams, recursive splitting is the right starting point. It respects structural boundaries (paragraphs, headings, line breaks) without the overhead of running an embedding model during ingestion.
Production systems rarely use one chunking strategy. A common pattern:
The embedding model converts each chunk into a vector. Model choice determines both the quality of your vector representations and the cost of indexing your corpus.
| Model | Price per 1M tokens | MTEB score | MIRACL score | Dimensions |
|---|---|---|---|---|
| text-embedding-ada-002 | $0.10 | 61.0% | 31.4% | 1,536 |
| text-embedding-3-small | $0.02 | 62.3% | 44.0% | 1,536 |
| text-embedding-3-large | $0.13 | 64.6% | 54.9% | 3,072 |
text-embedding-3-small costs 5x less than ada-002 and scores higher on both benchmarks. There is no reason to use ada-002 for new projects. text-embedding-3-large costs 6.5x more than text-embedding-3-small for a 2.3 percentage point MTEB improvement and a meaningful 10.9 point MIRACL improvement, which matters primarily for multilingual retrieval.
text-embedding-3-large supports native dimension reduction. You can request fewer than the full 3,072 dimensions via the API, trading some accuracy for smaller vectors and lower storage costs.
If data cannot leave your infrastructure, open-source models like nomic-embed-text and BGE run on-premise with full data control. The trade-off is hosting and maintaining GPU infrastructure for inference. For teams already running GPU nodes, the marginal cost of embedding is near zero.
The vector database stores your embedded chunks and handles similarity search at query time. The market has consolidated around a few options.
pgvector runs as a PostgreSQL extension. Vectors live in the same table as your relational data. No new service to deploy, authenticate against, or monitor. With HNSW indexes, pgvector handles up to 2 million vectors without special tuning.
The limitation is scale. Past 2 million vectors on a single Postgres instance, index build times grow, and you need read replicas or partitioning. For most RAG applications, 2 million vectors covers a large corpus.
Qdrant is a purpose-built vector database written in Rust. Published benchmarks show approximately 850 queries per second at p95 latency of around 8ms on 1 million vectors. Filtering performance (searching vectors that match metadata conditions) is a particular strength.
Pinecone delivers sub-20ms p95 latency at 5 million or more vectors with no infrastructure management. The cost is 3–8x higher than a self-hosted Postgres or Qdrant instance at equivalent vector counts. Teams that want to avoid ops overhead and can absorb the cost difference get a fast, stable service.
| Factor | pgvector | Qdrant | Pinecone |
|---|---|---|---|
| Already using Postgres? | Use it | Skip | Skip |
| Need >2M vectors? | Requires tuning | Good fit | Good fit |
| Self-hosted requirement? | Yes | Yes | No (managed only) |
| Budget-sensitive? | Cheapest | Moderate | Most expensive |
Naive vector search (embed the query, find the nearest vectors) works in demos. Production retrieval requires combining multiple signals.
Vector search excels at semantic similarity. Keyword search (BM25) excels at exact matches: product names, error codes, regulatory references. Hybrid search runs both and merges results using Reciprocal Rank Fusion (RRF), which combines the ranked lists without requiring score normalization between different search methods.
A query like "GDPR Article 17 data deletion requirements" benefits from BM25 matching "Article 17" exactly while vector search captures the semantic meaning of "data deletion requirements."
Complex queries often fail because the query embedding does not resemble the passage embedding. Two techniques help:
The retriever returns a top-K set (typically 20–50 candidates). A cross-encoder reranker then scores each candidate against the original query with full attention over both texts, producing a more accurate ranking than the initial vector similarity.
Anthropic's contextual retrieval research quantified this: adding document-level context to chunks before embedding reduced retrieval failures by 49%. Combining contextual embeddings with reranking reduced failures by 67%. Reranking is not optional in production. It is the step that separates "usually finds something relevant" from "consistently finds the right passage."
The generation step is where security risks concentrate. The LLM sees the full prompt: the user's query, the retrieved context, and any system instructions. If the retrieved context contains sensitive data, it flows through the model provider's API.
RAG systems pull from internal documents. Those documents contain customer names, email addresses, API keys, and financial data. Standard chunking does not filter sensitive content. It splits text mechanically and embeds whatever it finds.
When a user queries the system, retrieved chunks containing PII get sent to the LLM provider as part of the prompt. The model may then surface that PII in its response, or the data sits in the provider's request logs.
Rather than building PII detection into every application, a gateway layer can intercept requests before they reach the model provider. SHIM supports native OpenAI, Anthropic, and Google routes and replaces recognized sensitive values with typed placeholders before provider execution. Measure the end-to-end latency for your own inputs, provider, and deployment rather than assuming a fixed overhead.
This matters specifically for RAG because you cannot control what the retriever pulls from your corpus. Even if your ingestion pipeline tries to filter PII, edge cases slip through: a customer name embedded in a contract clause, an API key in a configuration snippet, a phone number in meeting notes. Gateway-level redaction catches what application-level filtering misses. Read more about PII redaction for AI applications.
LLMs produce unstructured text. When RAG answers feed downstream systems (dashboards, APIs, automated workflows), inconsistent formatting breaks integrations. Response normalization at the gateway layer enforces strict JSON schemas on model outputs, so every response matches the shape your application expects, regardless of which model generated it.
Production RAG systems see repetitive queries. "How do I reset my password?" and "I forgot my password, help" are semantically identical. A semantic cache at the gateway fingerprints prompts using embeddings and serves cached responses for equivalent queries, cutting both latency and cost for repeated patterns. See how smart caching cuts AI costs.
You cannot improve what you do not measure. Three metrics matter:
Of all relevant passages in your corpus, what fraction appears in the top-K retrieved results? Low recall means the retriever misses relevant content.
Of the top-K retrieved passages, what fraction is relevant? Low precision means the LLM's context window fills with noise.
Does the generated answer match a ground-truth answer? This end-to-end metric captures failures across all four pipeline phases.
Build an evaluation dataset of 50–100 question/answer/source-passage triples from your actual documents. Run retrieval and generation against this set after every pipeline change. Without this, you are tuning blind.
A concrete stack for a team starting today:
This is not the only valid architecture. It is a specific, defensible starting point that avoids premature optimization while covering the failure modes that matter most in production.