Hyperparameter Tuning: Complete Hyperopt and Optuna Guide

Hyperparameter tuning is the process of optimizing parameters that aren't learned during training — learning rate, tree depth, regularization strength, and so on. On Databricks, Hyperopt (distributed search via SparkTrials) and Optuna are the main tools, and every trial can be automatically recorded in MLflow Tracking. The ML Associate exam frequently tests the basic Hyperopt API, while ML Professional often asks when to use SparkTrials vs Trials.

Comparing Search Strategies

The algorithms used to find the best combination from a hyperparameter search space fall into three broad categories.

For both practical work and the exam, the most important strategy on Databricks is Bayesian Optimization (TPE: Tree-structured Parzen Estimator). Hyperopt and Optuna both default to TPE, and they can find strong solutions in high-dimensional spaces with roughly 50-200 trials.

Hyperopt Basics

Hyperopt is the Bayesian optimization library built into Databricks. You can run tuning simply by passing an objective function, a search space, an algorithm, and a maximum number of trials to fmin().

Defining the Search Space

Hyperopt + MLflow Integration Example

from hyperopt import fmin, tpe, hp, STATUS_OK, SparkTrials
import mlflow
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# Define the search space
search_space = {
    "n_estimators": hp.quniform("n_estimators", 50, 500, 50),
    "max_depth": hp.quniform("max_depth", 3, 15, 1),
    "min_samples_split": hp.quniform("min_samples_split", 2, 20, 1),
    "learning_rate": hp.loguniform("learning_rate", np.log(1e-4), np.log(1e-1)),
}

# Objective function (returns the value to minimize)
def objective(params):
    params["n_estimators"] = int(params["n_estimators"])
    params["max_depth"] = int(params["max_depth"])
    params["min_samples_split"] = int(params["min_samples_split"])

    clf = RandomForestClassifier(**params, random_state=42)
    score = cross_val_score(clf, X_train, y_train, cv=3, scoring="f1").mean()

    # Log metrics to MLflow
    mlflow.log_metrics({"f1_cv": score})

    # Return STATUS_OK and loss (loss is minimized)
    return {"loss": -score, "status": STATUS_OK}

# Run distributed with SparkTrials
spark_trials = SparkTrials(parallelism=8)

with mlflow.start_run(run_name="hyperopt_rf_tuning"):
    best_params = fmin(
        fn=objective,
        space=search_space,
        algo=tpe.suggest,       # TPE (Bayesian Optimization)
        max_evals=100,          # up to 100 trials
        trials=spark_trials,    # run in parallel on Spark executors
    )

The objective function must return a dict in the form {"loss": value, "status": STATUS_OK}. Because loss is the value fmin minimizes, return a negative value (-score) when you want to maximize accuracy.

SparkTrials vs Trials

The Trials class choice determines Hyperopt's execution mode. Whether you tap into the cluster's resources makes a huge difference in performance.

SparkTrials' parallelism involves a tradeoff. Higher values increase raw parallelism, but TPE uses past results to choose the next point. With parallelism too high, you end up choosing the next point while many in-flight trials haven't returned yet, which approaches random search. In practice, roughly the square root of max_evals is the recommended setting.

Optuna Basics

Optuna is a Bayesian optimization framework developed by Japan-based Preferred Networks. Unlike Hyperopt, it supports pruning (early termination) out of the box, letting you abort unpromising trials mid-flight to cut compute costs.

Key Optuna APIs

Optuna + MLflow Integration Example

import optuna
import mlflow
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 500),
        "max_depth": trial.suggest_int("max_depth", 3, 15),
        "learning_rate": trial.suggest_float("learning_rate", 1e-4, 1e-1, log=True),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "min_samples_split": trial.suggest_int("min_samples_split", 2, 20),
    }

    clf = GradientBoostingClassifier(**params, random_state=42)
    score = cross_val_score(clf, X_train, y_train, cv=3, scoring="f1").mean()

    # Log each trial to MLflow
    with mlflow.start_run(nested=True):
        mlflow.log_params(params)
        mlflow.log_metric("f1_cv", score)

    return score

with mlflow.start_run(run_name="optuna_gbm_tuning"):
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=100)

    # Record the best parameters
    mlflow.log_params(study.best_params)
    mlflow.log_metric("best_f1", study.best_value)

Hyperopt vs Optuna

Relationship with AutoML

Databricks AutoML automatically performs preprocessing, feature engineering, model selection, and hyperparameter tuning once you hand it data. Internally it uses Hyperopt's TPE algorithm to search hyperparameters for each model, and every trial is automatically logged to an MLflow experiment.

• Every notebook AutoML generates contains tuning code that uses Hyperopt
• You can edit the generated notebooks to expand the search space or add custom preprocessing
• A common production pattern is to use AutoML for a baseline and then fine-tune with Hyperopt
• Because AutoML results are logged to an MLflow experiment, you can directly compare them with manual tuning results

MLflow Tracking Integration

When you use Hyperopt with SparkTrials, each trial is automatically logged as a child run nested under the parent run. That lets you compare parameters and metrics across all trials in the MLflow UI.

• Parent run: created with mlflow.start_run(); records metadata for the entire tuning job
• Child runs: SparkTrials automatically logs each trial as a nested run (parameters, loss, status)
• Compare feature: plot metrics across child runs to analyze the impact of each parameter
• Register the best run's model in the Model Registry to move seamlessly to deployment

Best Practices for Distributed Tuning

• Setting parallelism: use the square root of max_evals as a rule of thumb; for 100 trials, parallelism of around 10 works well. Too high a value hurts TPE's search efficiency
• Early stopping: use the early_stop_fn parameter of fmin to stop the search once the target accuracy is reached
• Search-space types: use hp.loguniform for learning rate (parameters that span orders of magnitude) and hp.quniform for tree depth (integer parameters)
• Cluster sizing: with SparkTrials each trial runs as 1 executor = 1 trial; for GPU models, ensure one GPU per worker
• Data size and caching: for large training datasets, reduce data-transfer costs with spark.broadcast() or by writing the data to DBFS in advance
• Ensuring reproducibility: fix seeds with np.random.seed() and the rstate parameter, but note that distributed execution has non-deterministic ordering

Exam Focus Points

Check Your Understanding

Frequently Asked Questions