Cloud Monitoring SLO + Burn Rate アラート設定チュートリアル

Cloud Monitoring の SLO 機能を使った本番アラート設計を完全チュートリアル化します。 SLI 選定から Burn Rate Multi-window アラート、Terraform IaC 化、Synthetic Monitoring まで網羅します。

SLO 設計フロー

1. ユーザー体験を定義 (例: API レスポンスが 500ms 以内に返ること)
2. SLI 選定 (Availability / Latency / Throughput / Correctness)
3. SLO 設定 (例: 99.9% を Rolling 28 日で維持)
4. Error Budget 計算 (99.9% → 月 43 分のダウン許容)
5. Burn Rate アラート Multi-window 設定
6. 定期的に SLO レビュー (四半期)

SLI 例

Cloud Run の SLO 設定 (Terraform)

resource "google_monitoring_service" "api" {
  service_id   = "api-service"
  display_name = "Order API"

  basic_service {
    service_type = "CLOUD_RUN"
    service_labels = {
      service_name = "order-api"
      location     = "asia-northeast1"
    }
  }
}

resource "google_monitoring_slo" "availability" {
  service             = google_monitoring_service.api.service_id
  slo_id              = "availability-99-9"
  display_name        = "99.9% Availability over 28d"
  goal                = 0.999
  rolling_period_days = 28

  basic_sli {
    availability {}
  }
}

resource "google_monitoring_slo" "latency" {
  service             = google_monitoring_service.api.service_id
  slo_id              = "latency-p95-500ms"
  display_name        = "95% requests under 500ms"
  goal                = 0.95
  rolling_period_days = 28

  basic_sli {
    latency {
      threshold = "0.5s"
    }
  }
}

Burn Rate アラート (Multi-window Multi-burn-rate)

resource "google_monitoring_alert_policy" "fast_burn" {
  display_name = "Fast burn: 14.4x in 1h"
  combiner     = "OR"

  conditions {
    display_name = "Fast burn"
    condition_threshold {
      filter = <<EOT
select_slo_burn_rate("${google_monitoring_slo.availability.name}", 3600s)
EOT
      threshold_value = 14.4
      duration        = "60s"
      comparison      = "COMPARISON_GT"
    }
  }

  notification_channels = [google_monitoring_notification_channel.pagerduty.id]
}

resource "google_monitoring_alert_policy" "slow_burn" {
  display_name = "Slow burn: 3x in 24h"
  combiner     = "OR"

  conditions {
    display_name = "Slow burn"
    condition_threshold {
      filter = <<EOT
select_slo_burn_rate("${google_monitoring_slo.availability.name}", 86400s)
EOT
      threshold_value = 3
      duration        = "300s"
      comparison      = "COMPARISON_GT"
    }
  }

  notification_channels = [google_monitoring_notification_channel.slack.id]
}

通知チャネル設定

# PagerDuty
resource "google_monitoring_notification_channel" "pagerduty" {
  display_name = "PagerDuty: SRE"
  type         = "pagerduty"
  labels = { service_key = var.pagerduty_key }
}

# Slack
resource "google_monitoring_notification_channel" "slack" {
  display_name = "Slack: #alerts"
  type         = "slack"
  labels = {
    channel_name = "#alerts"
    auth_token   = var.slack_token
  }
}

# Webhook
resource "google_monitoring_notification_channel" "webhook" {
  display_name = "Custom Webhook"
  type         = "webhook_tokenauth"
  labels       = { url = "https://api.example.com/alerts" }
}

OpenTelemetry 計装例 (Python)

from opentelemetry import metrics, trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter

resource = Resource.create({"service.name": "order-api", "service.version": "1.0.0"})
reader = PeriodicExportingMetricReader(CloudMonitoringMetricsExporter())
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[reader]))

meter = metrics.get_meter(__name__)
order_counter = meter.create_counter("orders_total", description="Total orders")
order_latency = meter.create_histogram("order_latency_ms", unit="ms")

# アプリ内で記録
order_counter.add(1, {"status": "success"})
order_latency.record(123.4)

Synthetic Monitoring (E2E)

// Puppeteer Script (synthetic-script.js)
const synthetics = require('@google-cloud/synthetics-sdk');

synthetics.GenericSynthetic(async () => {
  const browser = await synthetics.launch();
  const page = await browser.newPage();
  await page.goto('https://app.example.com/login');
  await page.fill('#email', '[email protected]');
  await page.click('#login-button');
  await page.waitForSelector('#dashboard');
  return { success: true };
});

# デプロイ (Cloud Functions として)
gcloud functions deploy synthetic-login \
  --gen2 --runtime=nodejs20 --trigger-http

Golden Signal ダッシュボード

• Latency: P50 / P95 / P99 (Time Series)
• Traffic: Request/秒 (Line Chart)
• Errors: エラー率 (Scorecard + 履歴)
• Saturation: CPU / メモリ / Queue 滞留
• SLO 進捗 (Error Budget Gauge)

運用チェックリスト

• ☐ SLI 定義 (Cloud Run / GKE)
• ☐ SLO 設定 (99.9% / 28 日 Rolling)
• ☐ Fast burn アラート → PagerDuty
• ☐ Slow burn アラート → Slack
• ☐ Uptime Check (5 拠点)
• ☐ Synthetic Monitoring (主要 UX)
• ☐ Error Budget Policy 文書化
• ☐ Postmortem テンプレート準備
• ☐ On-call ローテーション

※ Google Cloud は Google LLC の商標です。最新は SLO Monitoring 公式 をご確認ください。