Build a RAG System with Vertex AI + LangChain: Vector Search & Gemini

A hands-on tutorial for building a production-grade RAG (Retrieval-Augmented Generation) system with Vertex AI Vector Search + LangChain + Gemini. We cover embedding generation, chunking, ANN search, and injection into Gemini, all with working code samples.

Architecture

Documents (PDF/MD/HTML)
       |
    Parse (Document AI / Unstructured)
       |
    Chunking (512-1024 tok, overlap 100)
       |
    Embedding (text-embedding-005)
       |
       v
Vertex AI Vector Search Index
       ^
       | (search)
User Query → Embedding → Vector Search → Top-K → Gemini Pro → Answer

Step 1: Document Preprocessing

# pip install langchain langchain-google-vertexai google-cloud-aiplatform
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("doc.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1024,
    chunk_overlap=100,
    separators=["\n\n", "\n", "。", "！", "？", " "],
)
chunks = splitter.split_documents(docs)
print(f"{len(chunks)} chunks")

Step 2: Generate Embeddings

from langchain_google_vertexai import VertexAIEmbeddings

embeddings = VertexAIEmbeddings(model_name="text-embedding-005")

texts = [chunk.page_content for chunk in chunks]
vectors = embeddings.embed_documents(texts)
print(f"Vector dim: {len(vectors[0])}")  # 768

Step 3: Create a Vertex AI Vector Search Index

from google.cloud import aiplatform as ai

ai.init(project="my-project", location="asia-northeast1")

# Upload embeddings to GCS in JSONL format
# {"id": "doc1-chunk1", "embedding": [0.1, 0.2, ...]}

# Create the index
index = ai.MatchingEngineIndex.create_tree_ah_index(
    display_name="my-rag-index",
    contents_delta_uri="gs://my-bucket/embeddings/",
    dimensions=768,
    approximate_neighbors_count=10,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
)

# Deploy the endpoint
endpoint = ai.MatchingEngineIndexEndpoint.create(
    display_name="my-rag-endpoint",
    public_endpoint_enabled=True,
)
endpoint.deploy_index(index=index, deployed_index_id="my_rag_v1")

Step 4: Retrieve and Generate Answers with Gemini

from langchain_google_vertexai import ChatVertexAI
from langchain.prompts import ChatPromptTemplate

llm = ChatVertexAI(model_name="gemini-2.0-pro-001", temperature=0.3)

def search_and_answer(query: str) -> str:
    # 1. Embed the query
    query_emb = embeddings.embed_query(query)

    # 2. Vector Search
    response = endpoint.find_neighbors(
        deployed_index_id="my_rag_v1",
        queries=[query_emb],
        num_neighbors=5,
    )
    top_chunks = [chunks[int(n.id.split("-chunk")[1])] for n in response[0]]

    # 3. Generate the answer with Gemini
    context = "\n\n".join([c.page_content for c in top_chunks])
    prompt = ChatPromptTemplate.from_template("""
Answer the user's question based on the context below.
If the answer is not in the context, reply "I don't know."

# Context
{context}

# Question
{query}

# Answer
""")
    chain = prompt | llm
    result = chain.invoke({"context": context, "query": query})
    return result.content

# Usage
print(search_and_answer("How do I request time off?"))

Step 5: Hybrid Search (Vector + Keyword)

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# Keyword search
bm25 = BM25Retriever.from_documents(chunks)

# Vector search (Vertex AI Vector Search wrapper)
from langchain_google_vertexai import VectorSearchVectorStore
vector_retriever = VectorSearchVectorStore.from_components(
    project_id="my-project",
    region="asia-northeast1",
    gcs_bucket_name="my-bucket",
    index_id="my-rag-index",
    endpoint_id="my-rag-endpoint",
).as_retriever()

# Ensemble (weighted)
ensemble = EnsembleRetriever(
    retrievers=[vector_retriever, bm25],
    weights=[0.7, 0.3],
)
results = ensemble.invoke("time off request")

Step 6: Evaluation

• Evaluate accuracy with RAGAS, TruLens, or DeepEval
• Key metrics: Faithfulness (no hallucination), Answer Relevance, Context Precision/Recall
• Managed evaluation via Vertex AI Gen AI Evaluation Service (available since 2024)

Production Operations

• When a new document is added → automatically embed and update the index (Cloud Functions + Pub/Sub)
• Serve the API on Cloud Run
• Build a UI with Vertex AI Agent Builder (optional)
• Include citations (source references) in the answer
• Configure the Safety Filter (BLOCK_MOST)
• Log every query and response with Cloud Logging (for quality improvement)

Cost Example (1,000 documents, 10K queries/month)

* Google Cloud and Gemini are property of Google LLC; LangChain is property of LangChain, Inc.