Cloud Monitoring SLO + Burn Rate Alert Tutorial

A complete tutorial for designing production alerts with the Cloud Monitoring SLO feature. Covers everything from SLI selection to Burn Rate Multi-window alerts, Terraform IaC, and Synthetic Monitoring.

SLO Design Flow

1. Define the user experience (e.g., API responses returned within 500ms)
2. Select SLIs (Availability / Latency / Throughput / Correctness)
3. Set the SLO (e.g., maintain 99.9% over a rolling 28-day window)
4. Compute the Error Budget (99.9% → about 43 minutes of allowed downtime per month)
5. Configure Multi-window Burn Rate alerts
6. Review the SLO regularly (quarterly)

Example SLIs

Cloud Run SLO Configuration (Terraform)

resource "google_monitoring_service" "api" {
  service_id   = "api-service"
  display_name = "Order API"

  basic_service {
    service_type = "CLOUD_RUN"
    service_labels = {
      service_name = "order-api"
      location     = "asia-northeast1"
    }
  }
}

resource "google_monitoring_slo" "availability" {
  service             = google_monitoring_service.api.service_id
  slo_id              = "availability-99-9"
  display_name        = "99.9% Availability over 28d"
  goal                = 0.999
  rolling_period_days = 28

  basic_sli {
    availability {}
  }
}

resource "google_monitoring_slo" "latency" {
  service             = google_monitoring_service.api.service_id
  slo_id              = "latency-p95-500ms"
  display_name        = "95% requests under 500ms"
  goal                = 0.95
  rolling_period_days = 28

  basic_sli {
    latency {
      threshold = "0.5s"
    }
  }
}

Burn Rate Alerts (Multi-window Multi-burn-rate)

resource "google_monitoring_alert_policy" "fast_burn" {
  display_name = "Fast burn: 14.4x in 1h"
  combiner     = "OR"

  conditions {
    display_name = "Fast burn"
    condition_threshold {
      filter = <<EOT
select_slo_burn_rate("${google_monitoring_slo.availability.name}", 3600s)
EOT
      threshold_value = 14.4
      duration        = "60s"
      comparison      = "COMPARISON_GT"
    }
  }

  notification_channels = [google_monitoring_notification_channel.pagerduty.id]
}

resource "google_monitoring_alert_policy" "slow_burn" {
  display_name = "Slow burn: 3x in 24h"
  combiner     = "OR"

  conditions {
    display_name = "Slow burn"
    condition_threshold {
      filter = <<EOT
select_slo_burn_rate("${google_monitoring_slo.availability.name}", 86400s)
EOT
      threshold_value = 3
      duration        = "300s"
      comparison      = "COMPARISON_GT"
    }
  }

  notification_channels = [google_monitoring_notification_channel.slack.id]
}

Notification Channel Configuration

# PagerDuty
resource "google_monitoring_notification_channel" "pagerduty" {
  display_name = "PagerDuty: SRE"
  type         = "pagerduty"
  labels = { service_key = var.pagerduty_key }
}

# Slack
resource "google_monitoring_notification_channel" "slack" {
  display_name = "Slack: #alerts"
  type         = "slack"
  labels = {
    channel_name = "#alerts"
    auth_token   = var.slack_token
  }
}

# Webhook
resource "google_monitoring_notification_channel" "webhook" {
  display_name = "Custom Webhook"
  type         = "webhook_tokenauth"
  labels       = { url = "https://api.example.com/alerts" }
}

OpenTelemetry Instrumentation Example (Python)

from opentelemetry import metrics, trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter

resource = Resource.create({"service.name": "order-api", "service.version": "1.0.0"})
reader = PeriodicExportingMetricReader(CloudMonitoringMetricsExporter())
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[reader]))

meter = metrics.get_meter(__name__)
order_counter = meter.create_counter("orders_total", description="Total orders")
order_latency = meter.create_histogram("order_latency_ms", unit="ms")

# Record from within the app
order_counter.add(1, {"status": "success"})
order_latency.record(123.4)

Synthetic Monitoring (E2E)

// Puppeteer Script (synthetic-script.js)
const synthetics = require('@google-cloud/synthetics-sdk');

synthetics.GenericSynthetic(async () => {
  const browser = await synthetics.launch();
  const page = await browser.newPage();
  await page.goto('https://app.example.com/login');
  await page.fill('#email', '[email protected]');
  await page.click('#login-button');
  await page.waitForSelector('#dashboard');
  return { success: true };
});

# Deploy (as a Cloud Function)
gcloud functions deploy synthetic-login \
  --gen2 --runtime=nodejs20 --trigger-http

Golden Signals Dashboard

• Latency: P50 / P95 / P99 (Time Series)
• Traffic: Requests/sec (Line Chart)
• Errors: Error rate (Scorecard + history)
• Saturation: CPU / memory / queue dwell
• SLO progress (Error Budget Gauge)

Operations Checklist

• ☐ Define SLIs (Cloud Run / GKE)
• ☐ Configure SLOs (99.9% / 28-day rolling)
• ☐ Fast burn alert → PagerDuty
• ☐ Slow burn alert → Slack
• ☐ Uptime Check (5 locations)
• ☐ Synthetic Monitoring (key UX flows)
• ☐ Document Error Budget Policy
• ☐ Prepare postmortem template
• ☐ On-call rotation

※ Google Cloud is a trademark of Google LLC. See the official SLO Monitoring docs for the latest information.