Databricks Feature Serving Guide: Online Inference, Feature Delivery, and Online Tables

When you run ML models online, you need a way to fetch the same features used during training with low latency. Databricks Feature Serving delivers Feature Store tables on Unity Catalog through dedicated endpoints, ensuring training-serving consistency (using the same feature definitions for both training and inference). This article walks through how Feature Serving works, how to configure Online Tables, how to create endpoints, and how to keep training and serving consistent.

Why You Need Feature Serving

Batch inference can read features directly from Delta tables, but for online inference (real-time API requests) querying Delta tables is too slow (seconds to tens of seconds). Feature Serving addresses the following challenges:

Feature Serving Architecture

[Training]
  │  Feature Engineering in UC
  │  fe.create_training_set(
  │      entity_df, feature_lookups=[...])
  │  → Looks up features at training time
  v
[Feature Table (Delta, UC)]
  │  Source tables
  │  ├─ customer_features (customer_id, avg_spend, ...)
  │  └─ product_features  (product_id, category, ...)
  v
[Online Table]
  │  Incremental sync from Delta (Snapshot / Triggered / Continuous)
  │  → Converted into a low-latency KV store
  v
[Feature Serving Endpoint]
  │  REST API (POST /serving-endpoints/{name}/invocations)
  │  → Returns the feature vector keyed by customer_id
  v
[Model Serving / Application]
  │  Features + real-time input → inference result

Configuring Online Tables

An Online Table syncs a Delta table into a low-latency KV store. Create one by specifying the source Delta table (the Feature Table) and a primary key.

-- Create an Online Table (SQL)
CREATE ONLINE TABLE ml.features.customer_features_online
SOURCE ml.features.customer_features
PRIMARY KEY (customer_id)
SYNC MODE TRIGGERED;

Match the primary key to the lookup key on the Feature Table. Composite keys are also supported (for example, PRIMARY KEY (customer_id, product_id)).

Creating a Feature Serving Endpoint

Create a Feature Serving Endpoint with the Python SDK (databricks-feature-engineering).

from databricks.feature_engineering import FeatureEngineeringClient
from databricks.feature_engineering.entities.feature_serving_endpoint import (
    EndpointCoreConfig,
    ServedEntity,
)

fe = FeatureEngineeringClient()

fe.create_feature_serving_endpoint(
    name="customer-features-endpoint",
    config=EndpointCoreConfig(
        served_entities=ServedEntity(
            feature_table_name="ml.features.customer_features",
            scale_to_zero_enabled=True,
        )
    )
)

Setting scale_to_zero_enabled to True scales the endpoint down to zero during idle periods, reducing cost. The trade-off is added latency on cold start — for strict SLAs, set it to False and keep the endpoint warm at all times.

Making Requests to the Endpoint

import requests
import json

endpoint_url = "https://<workspace-url>/serving-endpoints/customer-features-endpoint/invocations"
headers = {"Authorization": "Bearer <token>", "Content-Type": "application/json"}

payload = {
    "dataframe_records": [
        {"customer_id": "C001"},
        {"customer_id": "C002"}
    ]
}

response = requests.post(endpoint_url, headers=headers, json=payload)
features = response.json()
# => {"dataframe_records": [
#   {"customer_id": "C001", "avg_spend": 150.5, "total_orders": 42, ...},
#   {"customer_id": "C002", "avg_spend": 89.2, "total_orders": 15, ...}
# ]}

Achieving Training-Serving Consistency

Training-serving skew — when feature definitions or computations differ between training and inference — is one of the biggest pitfalls undermining ML system reliability. Databricks Feature Engineering in Unity Catalog resolves this structurally:

• Single feature definition: A Feature Table on UC is the single source for both batch training and online inference
• FeatureLookup: Use create_training_set to look up features when building the training dataset, and serve from the same table at inference time
• Model lineage: MLflow logs the feature tables and versions used by each model

from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup

fe = FeatureEngineeringClient()

training_set = fe.create_training_set(
    df=labels_df,  # Labels dataframe (customer_id, label)
    feature_lookups=[
        FeatureLookup(
            table_name="ml.features.customer_features",
            lookup_key=["customer_id"],
        ),
        FeatureLookup(
            table_name="ml.features.product_features",
            lookup_key=["product_id"],
        ),
    ],
    label="label",
)

training_df = training_set.load_df()

# Log the model (feature lineage is recorded automatically)
fe.log_model(
    model=trained_model,
    artifact_path="model",
    flavor=mlflow.sklearn,
    training_set=training_set,
    registered_model_name="ml.models.recommendation_model",
)

Models logged with fe.log_model embed references to their feature tables. When such a model is deployed to Model Serving, features are automatically looked up via Feature Serving on each inference request (Feature Function).

Design Considerations

Monitoring Feature Serving

• Endpoint latency: monitor P50/P99 and set an SLA (for example, P99 < 100ms)
• Online Table sync lag: monitor the delay between the last source Delta update and the corresponding Online Table refresh
• Feature drift: enable Lakehouse Monitor on the source feature tables to detect distribution shifts
• Endpoint error rate: monitor 4xx/5xx rates to catch key mismatches or schema changes

Check Your Understanding

Frequently Asked Questions