What is Vector Search? A Complete Guide to Databricks Vector Search

Databricks Vector Search is a managed vector database integrated with Unity Catalog. It converts unstructured data such as text and images into vectors (embeddings) and provides search based on semantic similarity. It plays a central role as the retrieval component of RAG (Retrieval-Augmented Generation) pipelines, and the GenAI Engineer exam frequently tests index type selection and query design.

Vector Search Overview

Traditional keyword search (BM25, etc.) depends on lexical matching between query and document, so it cannot handle variations in wording such as "machine learning" vs "ML". Vector Search converts text into high-dimensional vectors and computes semantic closeness via cosine similarity or dot product, enabling search that handles synonyms, paraphrases, and multilingual content.

Key Features of Databricks Vector Search

• Fully managed: Databricks automatically handles index build, scaling, and availability
• Unity Catalog integration: Table-level access control and lineage tracking apply to Vector Search as well
• Delta Lake integration: Provides an auto-syncing index (Delta Sync Index) sourced from a Delta Table
• Foundation Model API integration: Embedding computation can stay entirely inside Databricks
• Metadata filtering: Hybrid search that combines vector similarity with metadata predicates

Index Type Comparison

Vector Search offers two index types that differ in how data is managed and which use cases they fit. The GenAI Engineer exam tests selecting the right index type for a given scenario.

In Delta Sync Index's auto-compute mode, you just point at a text column and Databricks computes the embeddings and stores them in the index. In precomputed-column mode, you compute the embeddings yourself, store them in a Delta Table column, and point the index at that column.

Creating an Endpoint

A Vector Search Endpoint is the compute resource that hosts indexes. You create the endpoint first, then attach indexes to it.

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()

# Create a Vector Search endpoint
vsc.create_endpoint(
    name="my_vs_endpoint",
    endpoint_type="STANDARD"
)

# Check endpoint status
endpoint = vsc.get_endpoint("my_vs_endpoint")
print(endpoint["endpoint_status"]["state"])  # "ONLINE"

Creating an Index

Delta Sync Index (auto embedding computation)

# Create a Delta Sync Index (auto-compute embeddings)
index = vsc.create_delta_sync_index(
    endpoint_name="my_vs_endpoint",
    index_name="catalog.schema.doc_index",
    source_table_name="catalog.schema.documents",
    primary_key="doc_id",
    pipeline_type="TRIGGERED",          # "TRIGGERED" or "CONTINUOUS"
    embedding_source_column="content",  # Specify the text column
    embedding_model_endpoint_name="databricks-gte-large-en"
)

# Check index status
print(index.describe()["status"]["ready"])  # True

Delta Sync Index (precomputed embeddings)

# Create an index from a Delta Table that already has a precomputed embedding column
index = vsc.create_delta_sync_index(
    endpoint_name="my_vs_endpoint",
    index_name="catalog.schema.doc_index_precomputed",
    source_table_name="catalog.schema.documents_with_embeddings",
    primary_key="doc_id",
    pipeline_type="TRIGGERED",
    embedding_dimension=1024,
    embedding_vector_column="embedding_vector"
)

Direct Vector Access Index

# Create a Direct Vector Access Index
index = vsc.create_direct_access_index(
    endpoint_name="my_vs_endpoint",
    index_name="catalog.schema.realtime_index",
    primary_key="item_id",
    embedding_dimension=1024,
    embedding_vector_column="embedding",
    schema={
        "item_id": "string",
        "embedding": "array<float>",
        "category": "string",
        "title": "string"
    }
)

# Insert vectors directly
index.upsert([
    {"item_id": "001", "embedding": [0.1, 0.2, ...], "category": "tech", "title": "..."},
    {"item_id": "002", "embedding": [0.3, 0.1, ...], "category": "science", "title": "..."}
])

Similarity Search Queries

Run a query against the index to retrieve the top-K most similar documents. You can use a text query (auto-embedded) or a vector query (precomputed).

# Text-based similarity search (Delta Sync Index + auto embeddings)
results = index.similarity_search(
    query_text="How to manage Databricks clusters",
    columns=["doc_id", "content", "source_url"],
    num_results=5
)

# Process the results
for doc in results["result"]["data_array"]:
    print(f"Score: {doc[-1]:.4f} | ID: {doc[0]} | Content: {doc[1][:100]}...")

# Vector-based similarity search (Direct Vector Access Index)
results = index.similarity_search(
    query_vector=[0.1, 0.2, 0.3, ...],  # precomputed vector
    columns=["item_id", "title", "category"],
    num_results=10
)

Metadata Filtering

Adding metadata predicates to a vector similarity search lets you restrict results to a specific category or time range. In RAG pipelines, this is used for permission-based document filtering and category narrowing.

# Search with metadata filtering
results = index.similarity_search(
    query_text="about security policy",
    columns=["doc_id", "content", "department", "updated_at"],
    num_results=5,
    filters={"department": "engineering", "updated_at >=": "2026-01-01"}
)

# Multi-predicate filtering
results = index.similarity_search(
    query_text="data pipeline design",
    columns=["doc_id", "content", "category"],
    num_results=5,
    filters={"category IN": ["architecture", "data-engineering"]}
)

Embedding Model Comparison

When using Delta Sync Index's auto embedding mode, specify a Foundation Model API model (such as GTE) via embedding_model_endpoint_name. To use an external model, create an endpoint with the External Model feature, precompute embeddings, store them in a Delta Table, and choose the precomputed-column mode.

GenAI Engineer Exam Focus Areas

• Index type selection: Choosing between Delta Sync and Direct Vector Access based on the scenario
• Embedding modes: When to use auto-compute mode vs precomputed-column mode
• pipeline_type: Difference between CONTINUOUS and TRIGGERED (real-time vs cost)
• Metadata filtering: Implementation patterns for security and category narrowing
• Unity Catalog integration: Applying ACLs to Vector Search indexes

Sample Question

Frequently Asked Questions