Most RAG implementations work great in demos and fail spectacularly in production. The tutorials show you how to split documents by character count, embed everything with the first model you find, and hope vector similarity returns something useful. Then you deploy it and discover that your AI assistant confidently tells users that your documentation says the exact opposite of what it actually says.
The problem isn’t that RAG is hard. The problem is that RAG is a system, not a single algorithm. And like any system, it fails at the weakest link.
Chunking Is Not a Text Processing Problem#
The most common RAG failure mode starts with chunking. You take your documentation, split it every 500 characters, overlap by 50 characters, and call it done. This approach treats chunking like a text processing problem when it’s actually a knowledge representation problem.
When you chunk by character count, you break sentences in the middle. You separate code examples from their explanations. You split numbered lists across chunk boundaries. The embedding model sees fragments, not concepts.
BenBot uses semantic chunking that respects document structure. Each chunk represents a complete thought—a section, a code example with its explanation, or a complete procedure. The chunking logic looks for natural breakpoints: markdown headers, code fences, paragraph boundaries.
Here’s what semantic chunking preserves that character-based chunking destroys:
- Technical procedures stay together with their context
- Code examples remain connected to explanatory text
- Lists and tables stay intact
- Section headers travel with their content
The tradeoff is variable chunk sizes. Some chunks are 200 tokens. Others are 800. Your embedding model handles both fine, but your retrieval logic needs to account for the size variation when selecting context for the LLM.
Embedding Models Are Not Commodities#
Your choice of embedding model affects everything downstream. Most implementations grab text-embedding-ada-002 because it’s the first result in the OpenAI docs, but that model wasn’t designed for technical documentation retrieval.
You need to test at least three approaches:
OpenAI’s text-embedding-3-small performs better than Ada-002 on technical content and costs less per token. At 512 dimensions, it finds semantic matches that Ada-002 misses while using less storage space.
Cohere’s embed-english-v3.0 outperforms OpenAI on domain-specific technical queries in many benchmarks, but costs more per request. The quality difference matters if your corpus contains specialized terminology that general-purpose models struggle with.
Open-source alternatives like all-MiniLM-L6-v2 cost almost nothing to run but require hosting infrastructure. The embedding quality is acceptable for many use cases, but you lose the semantic sophistication of commercial models.
The only way to choose is to benchmark against your actual queries and content. Build a test set of 50-100 real user questions with known correct answers. Measure which embedding model returns the right chunks in the top 5 results most often.
Pure Vector Search Is Not Enough#
The dirty secret of production RAG systems is that pure vector similarity often returns garbage. Your embedding model thinks “HTTP status codes” and “HTTP caching” are semantically similar, so when someone asks about 404 errors, they get explanations of cache-control headers.
Hybrid search solves this by combining vector similarity with keyword matching. You run both searches in parallel and fuse the results using a ranking algorithm that considers both semantic relevance and exact term matches.
BenBot’s hybrid approach:
- Vector search finds semantically similar chunks
- Keyword search finds exact term matches
- Ranking fusion combines results, boosting chunks that appear in both lists
- Fallback logic activates pure keyword search when vector search returns low-confidence results
The ranking fusion algorithm weights vector results higher for abstract queries (“how do I troubleshoot networking issues”) and keyword results higher for specific queries (“what is BGP AS path prepending”).
flowchart TD
Query[User Query] --> Vector[Vector Search
Semantic Match]
Query --> Keyword[Keyword Search
Exact Term Match]
Vector --> Fusion{Ranking Fusion}
Keyword --> Fusion
Fusion -->|High Confidence| TopK[Top-K Results]
Fusion -->|Low Confidence| Fallback[Fallback to Pure Keyword]
Fallback --> TopK
TopK --> LLM[LLM Generation]
This catches cases where embeddings miss obvious matches. If someone asks about “VPC peering,” you want results that contain those exact terms, not just semantically similar concepts about network connectivity.
Quality Assessment Prevents Silent Failures#
RAG systems fail silently. The retrieval returns irrelevant chunks, the LLM generates a confident-sounding answer based on wrong information, and the user gets misleading guidance. You don’t know this happened unless you’re actively measuring quality.
You need automated relevance scoring for retrieved chunks and answer validation for LLM outputs. Build these checks into your pipeline, not as an afterthought.
For chunk relevance, implement a scoring function that considers:
- Semantic similarity scores
- Keyword match density
- Document recency and authority
- Historical user interaction data
For answer validation, use a separate LLM call to verify that the generated answer is supported by the retrieved chunks. If the validation fails, return “I cannot find sufficient information in the available documentation” instead of a hallucinated response.
This adds latency and cost, but prevents the system from confidently delivering wrong answers. The user experience of “I don’t know” is vastly superior to confident misinformation.
Edge Computing Solves the Latency Problem#
RAG pipelines have a latency problem. You’re making multiple API calls—embedding generation, vector search, LLM inference—and each adds round-trip time. Users abandon requests that take longer than 3 seconds, which doesn’t leave much budget for network calls.
Cloudflare Workers solve this by running your retrieval logic at the edge. The vector database query happens from the same data center that serves your users, eliminating transcontinental round trips. The embedding generation and LLM calls still hit external APIs, but you’ve cut the worst latency sources.
Edge deployment also enables request coalescing. Multiple users asking similar questions within a short time window can share embedding calculations and retrieved chunks, reducing both cost and latency for subsequent requests.
The architecture looks like this:
- Static site assets served from Cloudflare CDN
- RAG pipeline logic runs in Cloudflare Workers
- Vector index stored in Cloudflare Vectorize
- LLM calls routed to the nearest OpenAI edge endpoint
This setup delivers sub-second response times for most queries, which makes the difference between a useful tool and an abandoned experiment.
Context Window Optimization Is Critical#
LLMs have finite context windows, and retrieved chunks often exceed that limit. You cannot just concatenate the top 10 results and hope for the best. You need a strategy for selecting and ordering chunks to maximize relevance within token constraints.
The naive approach ranks chunks by similarity score and includes them until you hit the token limit. This often includes redundant information while excluding important details that appeared in lower-ranked chunks.
Better approaches:
Diversity-based selection ensures retrieved chunks cover different aspects of the topic. If the first chunk explains basic concepts and the second chunk also explains basic concepts, skip the second and include a chunk that covers advanced usage or troubleshooting.
Context-aware ordering places the most directly relevant chunk closest to the prompt, where the LLM pays the most attention. Supporting information goes in the middle. Background context goes at the beginning where it establishes foundation knowledge.
Dynamic chunk sizing adjusts retrieval based on query complexity. Simple factual questions get fewer, more focused chunks. Complex how-to questions get more comprehensive context even if individual chunks are less precisely matched.
Track which ordering strategies produce the best answers for different query types. This requires ongoing measurement, not just initial optimization.
Debugging Production RAG Systems#
RAG systems break in creative ways. The chunking logic miscategorizes content. The embedding model returns unexpected similarities. The LLM hallucinates despite having good source material. You need visibility into each component to debug failures effectively.
Essential logging:
- Query text and processed query embeddings
- Retrieved chunk IDs, similarity scores, and full text
- LLM prompt construction and token usage
- Generated response and confidence scores
- User feedback when available
Structure logs so you can trace a specific user query through the entire pipeline. When someone reports a wrong answer, you need to see exactly which chunks were retrieved and why the LLM generated its response.
Build debug endpoints that expose pipeline internals for testing. Create a query interface that shows retrieved chunks before LLM processing. Add switches to disable hybrid search components so you can isolate vector vs. keyword performance.
Monitor key metrics:
- Average retrieval latency by component
- Chunk relevance score distributions
- LLM token usage and cost per query
- User satisfaction ratings when available
Set up alerts for quality degradation. If average chunk relevance scores drop or user abandonment rates spike, you need to investigate before the problem affects more users.
Cost Optimization Reality#
RAG systems cost more than you expect. Embedding generation, vector storage, and LLM inference charges add up quickly. Optimize for cost efficiency without sacrificing quality.
Embedding costs: Cache embeddings aggressively. User queries often repeat, and document embeddings never change unless content updates. Store query embeddings with TTL expiration and document embeddings permanently.
Vector storage: Use lower-dimensional embeddings when quality allows. 512 dimensions usually perform as well as 1536 for domain-specific retrieval while using 70% less storage space.
LLM costs: Optimize prompt engineering to minimize token usage. Remove unnecessary retrieval context. Use shorter, more direct prompts. Consider smaller models for simple queries that don’t require advanced reasoning.
Infrastructure costs: Edge computing reduces API call volume through request coalescing and caching. The edge infrastructure cost is often offset by reduced upstream API charges.
Monitor cost per query and set budgets with alerts. Track cost efficiency metrics like successful answers per dollar spent. This data guides optimization decisions and helps justify infrastructure investments.
The goal is sustainable operation, not just functional operation. A RAG system that costs $500 per day to answer 100 queries is not viable long-term, regardless of answer quality.
RAG systems work when you treat them as systems. Focus on the inputs and outputs of each component. Measure everything. Optimize for the user experience you actually want, not the demo you can build in an afternoon.
Featured image by Microsoft Copilot on Unsplash
Recommended Reading#
- CCNP Enterprise Certification Study Guide: 350-401 ENCOR by Ben Piper & David Clinton
- The LLM Engineer's Handbook by Paul Iusztin and Maxime Labonne
- AI Engineering: Building Applications with Foundation Models by Chip Huyen