Azure RAG Pattern Implementation Guide: Chunking, Embedding, Azure AI Search, Hallucination Reduction

RAG retrieves relevant information from external knowledge bases (organization-specific data, latest information) and includes it in the prompt to generate responses. Standalone LLMs (Closed-book) can only answer based on general knowledge up to their training cutoff, while RAG (Open-book) enables responses grounded in organization-specific data and current news. Representative use cases: 1) Internal Q&A bots (referencing internal documents and policies), 2) Customer Support (FAQ and product manuals), 3) Legal Research (case law and contracts), 4) Medical Q&A (papers and guidelines), 5) E-commerce product recommendations (product catalog). RAG benefits: 1) lower cost and faster than fine-tuning, 2) easy data updates (just re-index), 3) source citation for accountability, 4) reduced hallucination. This is the most important topic in AI-103 Domain 2 (Generative AI).

Standard RAG pipeline: 1) Document Ingestion (ingest PDFs, Word, SharePoint, web pages; structured extraction via Azure AI Document Intelligence), 2) Chunking (split documents into 200-1,000 token units; Sentence / Paragraph / Recursive Character Splitter), 3) Embedding (vectorize with Azure OpenAI text-embedding-3-large; 3,072 dimensions; batch processing recommended), 4) Indexing (store in Azure AI Search Vector Index; Vector Field + Filterable Metadata Field), 5) Retrieval (user query -> embed -> top-K search via cosine similarity; Hybrid Search = Vector + Full-text + BM25 for higher accuracy), 6) Generation (include retrieved chunks in prompt; generate response with GPT-4o; include source citations). Production deployments require continuous improvement at every step. Chunking strategy, retrieval accuracy, and prompt design especially drive response quality.

Azure AI Search is the most recommended Azure-native vector store for RAG. Key features: 1) Vector Search (fast ANN search via HNSW Algorithm; supports up to hundreds of millions of vectors), 2) Hybrid Search (Vector + Full-text + BM25 for higher recall), 3) Semantic Ranking (re-ranking via Microsoft's proprietary BERT model), 4) Filter (filter by metadata fields), 5) Faceted Navigation (category aggregation), 6) Integrated Vectorization (auto-generate embeddings during document ingestion; Azure OpenAI integration), 7) Index Replicas + Partitions (high availability + scale). Pricing tiers: Basic (dev), Standard S1/S2/S3 (production), Storage Optimized (large volume). For production RAG, Standard S1 + 2 Replicas + 2 Partitions is the standard configuration, starting at tens of thousands of yen per month. Cosmos DB Vector and PostgreSQL pgvector are alternatives, but Azure AI Search leads on features and integration.

Chunking is the most important design decision affecting RAG quality. Common strategies: 1) Fixed-size Chunking (fixed 200, 500, 1,000 tokens; simplest; downside: token boundaries cut mid-sentence), 2) Sentence Splitter (split by sentence; natural breaks; uneven chunk sizes), 3) Paragraph Splitter (by paragraph; most natural; depends on PDF structure), 4) Recursive Character Splitter (provided by LangChain; hierarchical split Paragraph -> Sentence -> Word; recommended), 5) Semantic Chunking (group semantically by embedding similarity; highest quality; expensive), 6) Document-structure-aware Chunking (preserves heading, table, list structure; integrated with Microsoft AI Document Intelligence). Overlap (50-100 token overlap) prevents context loss. Typical designs: general document -> 500 tokens + 100 overlap + Recursive Character; long-form articles -> 1,000 tokens + 200 overlap; structured PDFs -> Document-structure-aware Chunking. Production deployments standardly A/B test multiple strategies for optimization.

Azure OpenAI embedding models: 1) text-embedding-3-large (3,072 dimensions; highest accuracy; medium cost), 2) text-embedding-3-small (1,536 dimensions; fast; low cost; about 1/5 the price of 3-large), 3) text-embedding-ada-002 (1,536 dimensions; legacy; backward compatibility only). Decision: high-accuracy requirements (legal, medical, precise search) -> text-embedding-3-large; high-speed/low-cost requirements (internal FAQ, large document volumes) -> text-embedding-3-small. Dimension reduction (e.g., 3-large from 3,072 -> 1,024 dimensions) can reduce cost at a slight accuracy penalty (API parameter). Microsoft / OSS alternatives: BAAI/bge-large-en, E5 series; for multilingual requirements, multilingual-specialized models like multilingual-e5-large. Changing the embedding model mid-production requires regenerating all vectors, so initial selection is critical. Comparing multiple models in a pilot is recommended.

Accuracy improvement techniques: 1) Hybrid Search (combined score via Reciprocal Rank Fusion of Vector + Full-text + BM25; 10-30% better accuracy than vector alone), 2) Semantic Ranking (retrieve top 50 -> re-rank with Microsoft Semantic Ranker -> top 5 -> LLM; balances recall and precision), 3) Query Rewriting (user query -> LLM generates multiple variants -> parallel search -> result fusion; Hypothetical Document Embeddings (HyDE) pattern), 4) Multi-step Retrieval (first search -> generate additional queries from results -> second search; agent pattern), 5) Filter (narrow domain by metadata field; date range, author, etc.), 6) Re-ranker (Cohere Rerank API, cross-encoder models), 7) Chunk Enhancement (add heading, summary, questions to chunks to diversify embeddings). Production deployments require measuring each technique's effectiveness via evaluation and optimizing combinations. Continuous improvement via Azure AI Foundry's Prompt Flow + Evaluation is standard.

Hallucination (the phenomenon where LLMs confidently generate factually incorrect responses) reduction: 1) Grounding (instruct in prompt: 'answer only based on the provided context; respond with "I do not know" if not in context'), 2) Source Citation (instruct in prompt: 'cite source IDs in responses' so users can verify), 3) Confidence Score (have LLM output response confidence; treat low confidence as uncertain), 4) Groundedness Detection (Azure AI Foundry's built-in metric; automatically verify generated content conforms to context; regenerate below threshold), 5) Multi-step Reasoning (Chain of Thought makes reasoning process explicit; easier to spot bad reasoning), 6) RAG result verification (after generation, have LLM self-check: 'is this response based solely on the provided context?'), 7) lower temperature toward 0 (deterministic responses suppress creative fabrication). Production deployments require combining multiple techniques + continuous evaluation. Microsoft's Groundedness Detection is a standard component of production RAG.

AI-103 (Developing AI Apps and Agents on Azure; GA 2026-06) Domain 2 (Generative AI 30-35%) deeply tests RAG patterns and is the flagship certification for this area. AI-901 (AI Fundamentals; GA 2026-06) covers RAG basics; AZ-204 (Developer Associate; retiring 2026-07, beware) covers RAG implementation from a developer perspective; SC-100 (Cybersecurity Architect Expert) covers RAG security and Responsible AI; DP-700 (Fabric Data Engineer) covers data preparation pipelines for RAG; DP-420 (Cosmos DB Specialty) covers RAG with Cosmos DB Vector. Understanding RAG is essential for Azure AI engineers and is the most important AI-103 study topic.