Azure AI Foundry Complete Guide: Hub/Project, Prompt Flow, Agent Service, Model Catalog & Fine-tuning

Azure AI Foundry (formerly Azure AI Studio) is Microsoft's unified AI development platform. It centrally manages Azure OpenAI, Azure AI Search, Azure AI Speech, Azure AI Vision, Azure AI Document Intelligence, Phi models, and OSS models (Mistral, Llama 3, Hugging Face models). It uses a two-tier Hub (organization level) / Project (team or project level) structure to centrally manage AI development across multiple teams. Key features: 1) Model Catalog (1,800+ model directory), 2) Playground (model experimentation), 3) Prompt Flow (LLM workflow design), 4) Agent Service (AI agent construction), 5) Evaluation (evaluation metrics, LLM-as-a-Judge), 6) Fine-tuning (model fine-tuning), 7) Content Safety (harmful content filtering), 8) Tracing (LLM call tracking). It is at the core of the AI-103 exam and the standard platform for modern Azure AI development.

Azure AI Foundry Hub-Project hierarchy. Hub (organization level): one Hub per organization or department, centrally manages shared infrastructure (Storage Account, Key Vault, Application Insights, Container Registry), and shares Compute Resources, Model Deployments, and Connections at the Hub level. Project (team/project level): multiple Projects sit under a Hub, isolated per team or project, run Prompt Flow, Agents, and Evaluation inside the Project, and inherit shared resources from the Hub. Typical layout: 1 organization -> 1-2 Hubs (production/non-production separation) -> 10-50 Projects (per team) inside each Hub. Access control: hierarchical RBAC roles like Hub Owner, Project Manager, and Project Contributor. Similar in design to the Microsoft Fabric Workspace Hub-Workspace model, enabling centralized AI governance for large organizations.

Prompt Flow is Azure AI Foundry's LLM workflow design tool. It uses a DAG (Directed Acyclic Graph) to define complex LLM pipelines with a mix of visual and code. Typical flow: 1) user query -> 2) query classification (LLM Node), 3) RAG retrieval (Python Tool calling Azure AI Search), 4) build LLM Prompt from Prompt Template (LLM Node), 5) generate response with GPT-4o, 6) format output (Python Tool). Each Node is selected from LLM Tool, Python Tool, Prompt Tool, Embedding Tool, Connection Tool, and others. Typical use cases: 1) RAG pipelines (retrieval + generation), 2) multi-step Agents (complex decisions), 3) document Q&A, 4) chatbot construction, 5) code generation + validation. Built on a Microsoft-developed OSS project, it supports local development (promptflow CLI), and production deployment to Managed Endpoint or Azure Functions / Container Apps.

Azure AI Foundry Agent Service (formerly Azure AI Assistant API) is a managed service for building complex LLM Agents. An Agent has Tools (functions/APIs), Knowledge (files/data sources), and Behavior (System Message/Instructions), and executes multi-step autonomous actions for a user query. Typical Tools: 1) Code Interpreter (Python code execution and data analysis), 2) File Search (full-text search over uploaded files), 3) Bing Search (web search), 4) Function Calling (custom APIs), 5) OpenAPI Tool (REST API calls), 6) Logic Apps Tool (SaaS integration via 300+ connectors), 7) Azure AI Search Tool (Vector + Hybrid search). Typical scenarios: 1) Customer Support Agent (FAQ search + ticket creation), 2) Data Analyst Agent (DB queries + chart generation), 3) Code Review Agent (PR analysis + comments), 4) Sales Agent (CRM lookup + draft email generation). Multi-Agent patterns (multiple Agents collaborating) are also supported. This is a core topic of AI-103 exam new Domain 3.

Azure AI Foundry's Evaluation continuously assesses the quality of LLM responses. Metrics: 1) Groundedness (whether generated content is grounded in source data; hallucination detection), 2) Relevance (relevance to the query), 3) Coherence (logical consistency of the response), 4) Fluency (natural writing), 5) Similarity (similarity to Ground Truth), 6) F1 Score (Precision + Recall), 7) Retrieval Score (Top-K retrieval accuracy in RAG), 8) Custom metrics (organization-specific evaluation via LLM-as-a-Judge). Implementation flow: 1) prepare Test Dataset (Query + Expected Answer + Context), 2) run Evaluation Run (Prompt Flow + Test Dataset), 3) metrics are computed automatically, 4) compare and analyze trends in the Results Dashboard. Continuous improvement pattern: 1) extract Failed Cases from Production Logs, 2) add to Test Dataset, 3) improve Prompt/Model, 4) rerun Evaluation to confirm improvement. It is the core quality-assurance function for production, and the standard design is to integrate it into the CI/CD pipeline for continuous evaluation.

Azure AI Foundry Model Catalog provides a unified catalog of 1,800+ AI models. By provider: 1) Azure OpenAI (gpt-4o, gpt-4o-mini, o1, o3, DALL-E, Whisper, Embeddings), 2) Microsoft (Phi-3, Phi-4 series, Florence, Speech models), 3) Meta (Llama 3.1, Llama 3.2/3.3), 4) Mistral (Mistral Large, Codestral), 5) Cohere (Command R+), 6) Hugging Face (BERT, T5, 1,500+ other OSS models), 7) partner models such as NVIDIA and Stability AI. Deployment options: 1) Pay-as-you-go API (Microsoft-managed, per-token pricing), 2) Managed Compute (dedicated VM, fine-tuning capable), 3) Serverless API (lightweight deployment), 4) local execution via container (for some OSS models). Typical selection: 1) general chat -> gpt-4o-mini, 2) lowest cost / on-prem execution -> Phi-4, 3) code generation -> Codestral, 4) Open License requirements -> Llama 3.3, 5) real-time voice -> GPT-4o Realtime. Model Catalog also provides model comparison and benchmarks.

Azure AI Foundry's Fine-tuning lets you fine-tune an existing LLM with your organization's own data. Supported models: gpt-4o-mini, gpt-3.5-turbo, Phi-3 series, and others (gpt-4o is Limited Preview at the time of writing). Steps: 1) prepare Training Dataset (JSONL format, minimum 50 examples, recommended 500-10,000), 2) prepare Validation Dataset (10-20% of Training), 3) create a job from the Fine-tuning UI in Azure AI Foundry, 4) configure Hyperparameters (Epochs, Learning Rate, Batch Size), 5) run training (hours to days depending on data volume), 6) compare the Fine-tuned model against the Base model with Evaluation, 7) deploy it for production. Typical use cases: 1) adapting organization-specific Tone/Style (Brand Voice), 2) improving understanding of industry terminology, 3) increasing accuracy of structured outputs, 4) strengthening multilingual support, 5) replacing few-shot examples with Fine-tuning to cut cost (shorter prompts). Cost: training time x unit price + monthly hosting of the Fine-tuned model (about 3-15 USD/h). RAG is often sufficient, so it is realistic to treat Fine-tuning as a last resort.

AI-103 (Developing AI Apps and Agents on Azure, GA 2026-06) is the headline certification where Azure AI Foundry is tested deeply at the core (Prompt Flow, Agent Service, Evaluation, Model Catalog, and Fine-tuning are all covered). AI-901 (AI Fundamentals, GA 2026-06, successor to AI-900) covers AI Foundry fundamentals, SC-100 (Cybersecurity Architect Expert) covers AI security and Responsible AI, and AZ-305 (Solutions Architect Expert) covers AI architecture. As the core platform of Microsoft's AI strategy, it is an essential skill for AI engineers and the most important area of AI-103 study.