Reader mode

Building a Production-Ready Retrieval-Augmented Generation (RAG) Application

Rajdeep Sengupta

Published on : Mar 23, 2025

Retrieval-Augmented Generation (RAG) is a powerful AI paradigm that enhances language models with external knowledge, enabling them to produce more accurate and contextually relevant responses. This blog will cover the end-to-end process of building a production-ready RAG system, including best practices, optimizations, and architectural considerations.

What is RAG?

RAG enhances language models by retrieving relevant information from an external knowledge base (vector database) and incorporating it into the response. This helps reduce hallucinations and improves factual accuracy.

High-Level Architecture of a RAG System

(Replace with an actual image URL)

Key Steps to Build a Production-Ready RAG System

1. Data Preparation & Indexing

Before querying a vector database, we need to prepare and index our documents.

Preprocessing Steps:

Text Cleaning: Remove unnecessary formatting, stopwords, and non-relevant data.
Chunking: Split long documents into smaller, meaningful chunks (~200-500 tokens per chunk).
Embedding Generation: Convert text into vector embeddings using a model like OpenAI’s text-embedding-ada-002 or Cohere’s embeddings.
Storing in a Vector Database: Use Weaviate, Pinecone, or FAISS to store and manage embeddings.

Example Code to Generate Embeddings & Store in Weaviate:

python

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Weaviate
import weaviate

client = weaviate.Client("http://localhost:8080")
embeddings = OpenAIEmbeddings()

# Example document
document = "Machine learning enables computers to learn from data."
vector = embeddings.embed(document)

client.data_object.create({
    "text": document,
    "vector": vector
}, class_name="Document")

2. Query Expansion for Better Retrieval

Why? Users often ask ambiguous queries. Expanding queries ensures better recall by adding synonyms and related terms.

Techniques for Query Expansion:

LLM-Based Expansion: Ask an LLM to generate related terms.
Embedding Similarity Expansion: Find similar phrases in the existing vector database.
Ontology-Based Expansion: Use predefined taxonomies for domain-specific queries.

Example Using LLM for Query Expansion:

python

from openai import OpenAI

query = "How does machine learning work?"
prompt = f"Generate alternative search queries for: {query}"
response = openai.ChatCompletion.create(
    model="gpt-4-turbo",
    messages=[{"role": "system", "content": prompt}]
)
expanded_queries = response["choices"][0]["message"]["content"].split("\n")

3. Retrieval from a Vector Database

Why? Retrieves documents relevant to the expanded query from the vector database.

Search Techniques:

Semantic Search: Finds documents using cosine similarity between vectors.
Hybrid Search: Combines vector-based search with keyword-based search for better accuracy.
BM25 + Embeddings: Uses a traditional BM25 ranking with vector similarity.

Example: Hybrid Search in Weaviate

python

retriever = client.query.get("Document", ["text", "vector"])
retriever = retriever.with_hybrid(search="machine learning techniques")
results = retriever.do()

4. Reranking Retrieved Documents

Why? The first retrieval step may return many results, but not all are equally relevant. Reranking ensures the best ones are prioritized.

Using Cohere’s Reranker API:

python

from cohere import Client

cohere_client = Client("COHERE_API_KEY")
response = cohere_client.rerank(
    model="rerank-english-v2.0",
    query="How does machine learning work?",
    documents=[doc["text"] for doc in results],
    top_n=5
)
ranked_results = [doc["text"] for doc in response["results"]]

5. Context Injection & Prompt Engineering

Why? The top-ranked documents are formatted into a structured prompt for the LLM to generate an informed response.

Best Practices:

Include only the most relevant 3-5 documents to avoid LLM token limits.
Format the context in a clear way (e.g., bullet points or paragraphs).
Use instructions in the prompt to guide the model.

Example Prompt:

python

prompt = """
Context:
"""
 + "\n".join(ranked_results) + """

Answer the following question based on the context:
How does machine learning work?
"""

6. Generating the Final Response Using an LLM

Why? The LLM generates a final response based on retrieved and reranked data.

Example Using OpenAI’s Chat Model:

python

response = openai.ChatCompletion.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": prompt}]
)
print(response["choices"][0]["message"]["content"])

7. Evaluation & Monitoring

Why? Ensure the RAG system performs well and provides high-quality responses.

Key Metrics to Track:

Retrieval Accuracy: Measure the relevance of retrieved documents.
Response Quality: Evaluate LLM responses using human annotators or automated scoring.
Latency & Scalability: Ensure fast response times under different loads.

Logging & Observability:

Use monitoring tools like Prometheus + Grafana to track system performance and response quality.

8. Deploying and Scaling Your RAG System

Deployment Considerations:

Containerization: Use Docker to package the app.
Serverless or Kubernetes: Deploy using AWS Lambda, GCP Cloud Run, or Kubernetes.
Load Balancing: Use Nginx or API Gateway to distribute traffic.
Auto-Scaling: Adjust based on user demand.

Conclusion

Building a scalable, accurate, and production-ready RAG system involves:

Preprocessing and Indexing Data
Expanding Queries for Better Recall
Retrieving Data Efficiently
Reranking for Higher Precision
Context Injection and Prompt Optimization
Generating Responses from an LLM
Evaluating and Monitoring Performance
Deploying and Scaling for Production

With these steps, your RAG-based AI can provide more factually accurate, context-aware, and high-quality responses to users. 🚀