Keycloak on GKE with Terraform and GitHub Actions

Keycloak is one of those services where the interesting part is never the application. Pulling the container is trivial. What takes the time is everything around it: a database it can reach privately, a cluster whose nodes have no public IPs, a certificate that renews itself, a pipeline that can deploy without a service account key sitting in a secret, and alerts that fire when the thing is actually broken rather than when a probe blinked.

This is a write-up of a complete setup: Terraform for the infrastructure, Helm for the application, GitHub Actions for delivery, GCP as the platform. Everything below has been run end to end against a real project, and most of the detail here is the stuff I only learned by watching it fail.

The shape:

GitHub Actions (OIDC, no keys)
        │
        ├── terraform/bootstrap   → applied locally, once   (APIs, WIF pool, CI service account)
        └── terraform             → applied by CI           (VPC, Cloud SQL, GKE, monitoring)
                                          │
                                          └── helm upgrade --install keycloakx
                                                    │
                                          ingress-nginx + cert-manager + Cloud DNS

Two root modules, on purpose

The first decision that pays for itself is splitting Terraform into two root modules with separate state.

This is manual-by-design, not manual-by-omission. There are two hard ordering constraints:

• Terraform cannot create the bucket that stores its own state.
• CI cannot authenticate through a Workload Identity pool that Terraform has not created yet.

There is a third reason that matters more over time: blast radius. terraform destroy on the workload stack is a routine cost cleanup. If the WIF pool and the CI service account lived in that same state, every cleanup would also delete the identity CI authenticates with, and every restart would begin with a re-bootstrap.

The bonus is that once IAM and API enablement live outside the CI-applied stack, the deploy service account needs no projectIamAdmin, serviceAccountAdmin, workloadIdentityPoolAdmin or serviceUsageAdmin. It cannot escalate its own privileges, because the roles that would let it do so are granted by a human running the seed layer.

The seed layer itself is small:

module "project_services" {
  source     = "../modules/project_services"
  project_id = var.project_id
  services   = var.required_apis
}

# Enabling an API returns before it is usable project-wide.
# No triggers, so it sleeps once per state and is a no-op on every later apply.
resource "time_sleep" "wait_for_api_propagation" {
  depends_on      = [module.project_services]
  create_duration = "30s"
}

module "iam" {
  source          = "../modules/iam"
  project_id      = var.project_id
  github_owner    = var.github_owner
  github_repo     = var.github_repo
  wif_pool_id     = var.wif_pool_id
  wif_provider_id = var.wif_provider_id

  depends_on = [time_sleep.wait_for_api_propagation]
}

That time_sleep looks like superstition and is not. Enabling a GCP API returns success well before the API is usable project-wide, and the very next resource is the one that discovers it.

Networking: private nodes need more than a flag

The VPC is unremarkable except for three things that are easy to leave out and painful to debug.

resource "google_compute_subnetwork" "this" {
  project       = var.project_id
  name          = var.subnetwork_name
  ip_cidr_range = var.subnetwork_cidr
  region        = var.region
  network       = google_compute_network.this.id

  # Private nodes have no external IP. Without this they reach Google APIs --
  # registries, logging, monitoring -- the long way through Cloud NAT.
  private_ip_google_access = true

  log_config {
    aggregation_interval = "INTERVAL_10_MIN"
    flow_sampling        = 0.5
    metadata             = "INCLUDE_ALL_METADATA"
  }

  secondary_ip_range {
    range_name    = var.pods_secondary_range_name
    ip_cidr_range = var.pods_secondary_range_cidr
  }

  secondary_ip_range {
    range_name    = var.services_secondary_range_name
    ip_cidr_range = var.services_secondary_range_cidr
  }
}

Cloud NAT is not optional. Private GKE nodes have no external IPs. The first symptom is a Keycloak pod sitting in ImagePullBackOff with a quay.io timeout, which reads like a registry problem and is actually an egress problem. A router plus a NAT gateway fixes it:

resource "google_compute_router_nat" "nat" {
  project = var.project_id
  name    = "${var.network_name}-nat"
  region  = var.region
  router  = google_compute_router.nat_router.name

  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}

Private Service Access is what gives Cloud SQL a private IP. You reserve a range and peer it to the service networking API:

resource "google_compute_global_address" "private_service_range" {
  project       = var.project_id
  name          = var.private_service_range_name
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = var.private_service_range_prefix
  network       = google_compute_network.this.id
}

resource "google_service_networking_connection" "private_service_access" {
  network                 = google_compute_network.this.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_service_range.name]

  # Without this, `terraform destroy` blocks on the VPC peering teardown.
  deletion_policy = "ABANDON"
}

That deletion_policy = "ABANDON" is worth remembering the first time a destroy hangs for twenty minutes.

The webhook firewall rule. This one cost me an afternoon. On a private cluster, GKE automatically opens control-plane-to-node traffic on 443 and 10250 — and nothing else. The ingress-nginx admission webhook listens on 8443. So the cluster comes up healthy, everything looks fine, and the first Ingress you create fails with:

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io":
... context deadline exceeded

Nothing is misconfigured in ingress-nginx. The API server simply cannot reach it. The fix belongs in Terraform, next to the cluster:

resource "google_compute_firewall" "master_to_node_webhooks" {
  project   = var.project_id
  name      = "${var.network_name}-master-to-webhooks"
  network   = google_compute_network.this.id
  direction = "INGRESS"

  source_ranges = [var.master_ipv4_cidr_block]

  allow {
    protocol = "tcp"
    ports    = ["8443", "9443"]
  }
}

Cloud SQL: private IP, forced TLS, and a name that stays taken

resource "google_sql_database_instance" "this" {
  project             = var.project_id
  name                = "${var.instance_name}-${random_id.instance_suffix.hex}"
  region              = var.region
  database_version    = var.database_version   # e.g. POSTGRES_15
  deletion_protection = var.deletion_protection

  settings {
    tier              = var.tier
    disk_size         = var.disk_size_gb
    disk_type         = "PD_SSD"
    availability_type = var.availability_type  # ZONAL or REGIONAL

    backup_configuration {
      enabled                        = true
      start_time                     = var.backup_start_time
      point_in_time_recovery_enabled = var.point_in_time_recovery
    }

    ip_configuration {
      ipv4_enabled    = false
      private_network = var.private_network_self_link
      # The JDBC driver negotiates TLS by default, so Keycloak needs no extra config.
      ssl_mode = "ENCRYPTED_ONLY"
    }

    database_flags {
      name  = "log_min_error_statement"
      value = "error"
    }
    # plus log_checkpoints, log_connections, log_disconnections,
    # log_lock_waits, log_hostname, log_duration
  }
}

Three things worth calling out.

The random suffix on the instance name. Cloud SQL reserves a deleted instance name for up to a week. If you tear down and rebuild — which you will, repeatedly, while iterating — a fixed name makes the second run fail on a name that looks free and isn't. A random_id suffix is stable in state and only changes when the instance is genuinely recreated.

ssl_mode = "ENCRYPTED_ONLY" rather than client certificates. Policy scanners will ask for mutual TLS. That would mean issuing and rotating a client cert per Keycloak pod, to defend against an attacker who is already inside the VPC and holds valid database credentials. With no public IP and enforced transport encryption, that is not where the next hour is best spent.

Database flags are written out one by one, not generated from a map. A dynamic "database_flags" block is tidier and reads as no flags set to Checkov, which does not evaluate dynamic blocks. Being explicit means the scanner sees what you actually configured.

The password never leaves Terraform as a literal:

resource "random_password" "db_password" {
  length           = 24
  special          = true
  override_special = "_%@"
}

and reaches Kubernetes as a secret rendered at deploy time from the Terraform output — never through a values file, never through git:

kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -

DB_PASS="$(terraform -chdir=terraform output -raw keycloak_db_password)"

kubectl -n "${NAMESPACE}" create secret generic keycloak-db \
  --from-literal="password=${DB_PASS}" \
  --dry-run=client -o yaml | kubectl apply -f -

The --dry-run=client -o yaml | kubectl apply -f - pattern is what makes it idempotent: create-or-update, no error on the second run.

GKE: private cluster with Workload Identity

resource "google_container_cluster" "this" {
  project  = var.project_id
  name     = var.cluster_name
  location = var.zone

  network    = var.network_name
  subnetwork = var.subnetwork_name

  remove_default_node_pool = true
  initial_node_count       = 1

  release_channel { channel = var.release_channel }

  ip_allocation_policy {
    cluster_secondary_range_name  = var.pods_range_name
    services_secondary_range_name = var.services_range_name
  }

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = var.master_ipv4_cidr_block
  }

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  # Certificate-based auth is legacy and cannot be revoked per user.
  master_auth {
    client_certificate_config { issue_client_certificate = false }
  }

  network_policy { enabled = true }
  enable_intranode_visibility = true
}

enable_private_endpoint = false is a deliberate compromise. Master authorized networks would be the right control, but GitHub-hosted runners have dynamic, unbounded egress IPs — there is no list to allow. Locking the control plane down properly means self-hosted runners or a bastion. Worth doing for production; worth writing down as a known gap rather than pretending otherwise.

The node pool carries the metadata server and free hardening:

node_config {
  machine_type = var.node_machine_type
  spot         = var.node_spot

  workload_metadata_config { mode = "GKE_METADATA" }

  shielded_instance_config {
    enable_secure_boot          = true
    enable_integrity_monitoring = true
  }
}

management {
  auto_repair  = true
  auto_upgrade = true
}

Note that remove_default_node_pool = true means the cluster resource has no node_config of its own. Some policy checks inspect exactly that and report the metadata server as disabled. It is enabled on the pool that actually runs workloads, which is the one that matters.

Keyless CI: Workload Identity Federation

No service account keys. GitHub issues an OIDC token, GCP trades it for short-lived credentials.

resource "google_iam_workload_identity_pool_provider" "github_oidc" {
  project                            = var.project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.github.workload_identity_pool_id
  workload_identity_pool_provider_id = var.wif_provider_id

  oidc { issuer_uri = "https://token.actions.githubusercontent.com" }

  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.actor"      = "assertion.actor"
    "attribute.aud"        = "assertion.aud"
    "attribute.repository" = "assertion.repository"
    "attribute.ref"        = "assertion.ref"
  }

  attribute_condition = "assertion.repository_owner == '${var.github_owner}' && assertion.repository == '${var.github_owner}/${var.github_repo}'"
}

The attribute condition is scoped to the repository, not to a branch. That is not laziness: GitHub issues a different sub per event type, so a PR plan authenticates as refs/pull/<n>/merge and pinning refs/heads/main breaks it. Authorisation to change anything is enforced by the GitHub Environment gate instead, which a pull_request run cannot pass. The cleaner long-term shape is two identities — a read-only planner and a main-pinned deployer — which is where I would take this next.

There is one non-obvious IAM binding. Creating a node pool requires actAs on the node service account. Granting roles/iam.serviceAccountUser at project level would let CI impersonate every service account in the project, so bind it to exactly one:

resource "google_service_account_iam_member" "ci_acts_as_node_sa" {
  service_account_id = "projects/${var.project_id}/serviceAccounts/${data.google_project.this.number}-compute@developer.gserviceaccount.com"
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.ci.email}"
}

And the deploy roles, chosen by watching applies fail rather than by starting from roles/editor:

for_each = toset([
  "roles/container.admin",
  "roles/cloudsql.admin",
  "roles/compute.networkAdmin",
  # networkAdmin excludes firewall rules, which the master-to-webhook rule needs.
  "roles/compute.securityAdmin",
  # Private Service Access needs servicenetworking.services.addPeering.
  "roles/servicenetworking.networksAdmin",
  "roles/dns.admin",
  "roles/logging.configWriter",
  "roles/monitoring.editor",
  "roles/storage.admin",
  "roles/serviceusage.serviceUsageConsumer"
])

One more thing that surfaces late: the human who runs the seed layer needs roles/iam.workloadIdentityPoolAdmin, and roles/editor does not include it. That shows up as a failure partway through the first apply, not at login.

The pipeline

Four workflows, one implementation.

deploy-reusable.yml holds the single implementation; the two deploy workflows are thin callers differing only by the environment input:

jobs:
  dev:
    uses: ./.github/workflows/deploy-reusable.yml
    with:
      environment: dev

Production has no push trigger. That is the whole point: prod is never reconciled as a side effect of merging. Promotion is an explicit dispatch of the same commit already verified in dev, behind a typed phrase:

on:
  workflow_dispatch:
    inputs:
      confirm:
        description: Type "deploy-prod" to confirm this is intentional.
        required: true
        type: string

jobs:
  guard:
    name: Confirm intent
    runs-on: ubuntu-latest
    steps:
      - run: |
          if [[ "${{ inputs.confirm }}" != "deploy-prod" ]]; then
            echo "Confirmation phrase did not match. Refusing to deploy to production." >&2
            exit 1
          fi

The typed phrase stops fat fingers. It does not stop a determined mistake — that is what required reviewers on the prod GitHub Environment are for, and being honest about which of the two you have actually configured is part of the job.

validate.yml runs on every PR and touches no cloud credentials at all: terraform fmt -check, init -backend=false and validate for both root modules, TFLint, Checkov, and a helm lint against the pinned chart version. That last one needs placeholder values, because the chart aborts on its own database-hostname assertion otherwise:

helm pull codecentric/keycloakx --untar --untardir /tmp --version 7.2.2
LINT_SET=(
  --set database.hostname=10.0.0.1
  --set ingress.rules[0].host=keycloak.lint.example.com
  --set ingress.tls[0].hosts[0]=keycloak.lint.example.com
)
helm lint /tmp/keycloakx -f helm/keycloakx/values.yaml -f helm/keycloakx/values-dev.yaml "${LINT_SET[@]}"

One source of truth for environment inputs

The pipeline renders terraform.tfvars and backend.hcl from GitHub Environment variables rather than reading a file from the repository:

- name: Render backend and tfvars from environment vars
  env:
    ALERT_EMAILS: ${{ vars.MONITORING_ALERT_EMAILS }}
  run: |
    cat > "terraform/environments/${ENVIRONMENT}/backend.hcl" <<EOF
    bucket = "${{ vars.TF_STATE_BUCKET_NAME }}"
    prefix = "terraform/state/${ENVIRONMENT}"
    EOF

    # Comma-separated input becomes a JSON array, which is valid HCL.
    EMAILS_HCL="$(python3 -c 'import sys,json; print(json.dumps([e.strip() for e in sys.argv[1].split(",") if e.strip()]))' "${ALERT_EMAILS}")"

    cat > "terraform/environments/${ENVIRONMENT}/terraform.tfvars" <<EOF
    project_id = "${{ vars.GCP_PROJECT_ID }}"
    region = "${{ vars.GCP_REGION }}"
    zone = "${{ vars.GCP_ZONE }}"
    environment = "${ENVIRONMENT}"
    dns_root_domain = "${{ vars.DNS_ROOT_DOMAIN }}"
    gke_node_spot = ${{ vars.GKE_NODE_SPOT || 'false' }}
    sql_deletion_protection = ${{ vars.SQL_DELETION_PROTECTION || 'true' }}
    monitoring_alert_emails = ${EMAILS_HCL}
    EOF

Two consequences to internalise:

1. The plan workflow must render exactly what the deploy workflow renders. If they diverge, the plan reviewed on the PR is not the plan that gets applied. Same block, copied deliberately, in both files.
2. Anything not in that list falls back to its variables.tf default in CI, regardless of what a developer has in a local tfvars. That is why gke_node_spot and sql_deletion_protection appear explicitly: dev takes preemption risk for cheaper nodes and stays destroyable, prod does neither.

Real values never enter git — only terraform.tfvars.example templates do. A small script generates the whole Environment variable set from the same tfvars and Terraform outputs the local runs use, which is what keeps CI and local in sync. Moving the whole thing to a new project changes inputs only: no module, workflow, script or Helm logic is touched.

The cluster name is deliberately not a variable. It is read from the Terraform output at deploy time, so it cannot drift from what was just applied:

- name: Configure kubectl for GKE
  run: |
    gcloud container clusters get-credentials \
      "$(terraform -chdir=terraform output -raw gke_cluster_name)" \
      --zone "$(terraform -chdir=terraform output -raw gke_cluster_location)" \
      --project "${{ vars.GCP_PROJECT_ID }}"

And the apply is serialised per environment, never cancelled:

concurrency:
  group: terraform-${{ inputs.environment }}
  cancel-in-progress: false

Interrupting an apply is how you get a locked state file and a half-reconciled stack. Queuing is the cheaper failure.

The Helm layer

Base values, shared across environments:

replicas: 3

podDisruptionBudget:
  maxUnavailable: 1

http:
  relativePath: /

command:
  - /opt/keycloak/bin/kc.sh
args:
  - start
  - --import-realm

resources:
  requests: { cpu: "500m", memory: "1Gi" }
  limits:   { cpu: "1500m", memory: "2Gi" }

dbchecker:
  enabled: true

database:
  vendor: postgres
  existingSecret: keycloak-db
  existingSecretKey: password

proxy:
  enabled: true
  mode: xforwarded
  http:
    enabled: true

metrics: { enabled: true }
health:  { enabled: true }

Every one of those lines is there for a reason:

• proxy.mode: xforwarded. ingress-nginx sends X-Forwarded-* headers, not the RFC 7239 Forwarded header. With the default forwarded mode Keycloak ignores them and starts advertising http:// issuer and redirect URLs in its OIDC discovery document, which breaks every client in a way that looks like a client bug.
• http.relativePath: /. The chart otherwise serves at /auth. Serving at the root keeps the URLs Keycloak advertises identical to the ones it answers on. The health probe paths derive from this value, so changing it changes more than the URL.
• The PodDisruptionBudget. This is the subtle one. The chart's default pod anti-affinity is requiredDuringSchedulingIgnoredDuringExecution on kubernetes.io/hostname, and the pool runs exactly as many nodes as there are replicas. A voluntary disruption — a node-pool upgrade, an autoscaler scale-down, a manual kubectl drain — evicts a pod that then has no node left to land on, and sits Pending until its own node returns. Nothing stops the next node draining meanwhile, so a routine rolling upgrade can walk the entire StatefulSet down one node at a time. maxUnavailable: 1 rather than minAvailable: 2 so the budget stays correct if replicas changes. Note that this bounds voluntary disruption only — spot preemption and node failure are involuntary and ignore a PDB entirely. Surviving those is what the replicas and anti-affinity are for.
• --import-realm. Realm and client definitions are configuration and belong in git. Keycloak leaves existing realms alone, so this is a no-op on every restart after the first and safe to leave on permanently. Users are data, and seeding a user means putting a password somewhere — so users are created by a script that generates a password and prints it once, never by a tracked JSON file.

Per-environment values carry only the things that differ:

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: keycloak-issuer
    # Keycloak sets large headers; the nginx default buffer returns 502 without this.
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
  rules:
    - host: INJECTED_BY_DEPLOY_SCRIPT
      paths:
        - path: /
          pathType: Prefix
  tls:
    - hosts: [INJECTED_BY_DEPLOY_SCRIPT]
      secretName: keycloak-tls

extraEnv: |
  - name: KC_HOSTNAME
    value: {{ printf "https://%s" (index .Values.ingress.rules 0).host | quote }}
  - name: KC_HOSTNAME_STRICT
    value: "true"    # "false" in dev

INJECTED_BY_DEPLOY_SCRIPT is not a placeholder I forgot to fill in. The Cloud SQL private IP is allocated from the PSA range and changes whenever the instance is recreated; hardcoding it silently breaks the next deploy. Environment identity comes from Terraform state at deploy time, always.

A Helm deploy that recovers from its own failures

helm upgrade --install --atomic is the right baseline, but --atomic only cleans up if the client survives long enough to run its cleanup. If a CI runner is cancelled or evicted mid-upgrade, Helm leaves the release in a pending-* state, and every subsequent run fails with another operation (install/upgrade/rollback) is in progress — forever, with no self-recovery. That is the single most likely way this pipeline deadlocks, so the deploy script resolves it up front:

RELEASE_STATUS="$(helm -n "${NAMESPACE}" status "${RELEASE}" -o json 2>/dev/null \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["status"])' 2>/dev/null || echo "none")"

case "${RELEASE_STATUS}" in
  pending-install)
    # Nothing was ever successfully deployed, so there is no revision to
    # return to; remove the half-created release.
    helm -n "${NAMESPACE}" uninstall "${RELEASE}" --wait --timeout "${TIMEOUT}" || true
    ;;
  pending-upgrade | pending-rollback)
    LAST_DEPLOYED="$(helm -n "${NAMESPACE}" history "${RELEASE}" 2>/dev/null \
      | awk 'NR>1 && $3=="deployed" {rev=$1} END{print rev}')"
    if [[ -n "${LAST_DEPLOYED}" ]]; then
      helm -n "${NAMESPACE}" rollback "${RELEASE}" "${LAST_DEPLOYED}" --wait --timeout "${TIMEOUT}" || true
    else
      helm -n "${NAMESPACE}" uninstall "${RELEASE}" --wait --timeout "${TIMEOUT}" || true
    fi
    ;;
  uninstalling)
    helm -n "${NAMESPACE}" uninstall "${RELEASE}" --no-hooks || true
    ;;
esac

Then the actual upgrade, with environment identity injected from Terraform outputs:

DB_HOST="$(terraform -chdir=terraform output -raw cloudsql_private_ip)"
KC_HOST="$(terraform -chdir=terraform output -raw keycloak_hostname)"

helm -n "${NAMESPACE}" upgrade --install "${RELEASE}" codecentric/keycloakx \
  --version "${CHART_VERSION}" \
  -f helm/keycloakx/values.yaml \
  -f "helm/keycloakx/values-${ENVIRONMENT}.yaml" \
  --set "database.hostname=${DB_HOST}" \
  --set "ingress.rules[0].host=${KC_HOST}" \
  --set "ingress.tls[0].hosts[0]=${KC_HOST}" \
  --wait --timeout "${TIMEOUT}" \
  --atomic --cleanup-on-fail --history-max 20

On failure it collects diagnostics before doing anything destructive — helm status --show-resources, kubectl get pods,sts,svc, and the last 80 events sorted by timestamp — then rolls back to the last revision whose status is deployed, or uninstalls if there has never been one. A rollback that erases the evidence is only half a safety net.

The chart version is pinned. So is the Keycloak image tag used by the admin-bootstrap helper. An unpinned chart means the deployment you tested on Tuesday is not the deployment that ships on Thursday.

One ordering detail lives inside this script rather than in the workflow: the realm-import ConfigMap is published here, immediately before the upgrade. The StatefulSet mounts it, so if it does not exist yet every pod sits in ContainerCreating until --atomic times out. Keeping it in the deploy script means there is exactly one entry point for deploying Keycloak, used identically by CI and by hand, with no ordering step left for someone to forget.

Ingress, certificates, DNS — and the order they must happen in

The sequence in the deploy job is not arbitrary:

1. terraform apply — cluster and database exist
2. create namespace + DB secret from Terraform output
3. install ingress-nginx
4. install cert-manager and the ClusterIssuer
5. Helm deploy Keycloak
6. read the ingress controller's external IP
7. upsert the DNS A record
8. smoke test

cert-manager has to come after ingress-nginx, because the ACME HTTP-01 solver needs the ingress class to exist. And Let's Encrypt needs the DNS A record to already resolve — which is the one genuine chicken-and-egg in the whole design, since the IP to point at only exists after the ingress controller has been provisioned.

The way out is issuer modes. The script takes selfsigned, letsencrypt-staging, or letsencrypt:

if [[ "${ISSUER_MODE}" == "selfsigned" ]]; then
  ISSUER_SPEC="  selfSigned: {}"
else
  ISSUER_SPEC="$(cat <<EOF
  acme:
    server: ${ACME_SERVER}
    email: ${ACME_EMAIL}
    privateKeySecretRef:
      name: ${ISSUER_NAME}-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: ${INGRESS_CLASS}
EOF
)"
fi

Self-signed for a first bring-up with no DNS yet; staging while iterating so you do not burn production rate limits; production once the record resolves. The ClusterIssuer name must match the cert-manager.io/cluster-issuer annotation in the Helm values — a mismatch leaves the certificate Pending with no obvious error.

Waiting for the webhook matters too:

kubectl -n cert-manager rollout status deployment/cert-manager-webhook --timeout=5m

The cert-manager webhook rejects Issuer resources until its endpoints are actually serving, so applying the issuer immediately after the Helm install fails intermittently — the worst kind of failure.

DNS is upserted idempotently through a Cloud DNS transaction that removes the existing record set before adding the new one, and exits early if the record already points where it should. The IP itself comes from polling the controller service:

for i in $(seq 1 60); do
  IP="$(kubectl -n ingress-nginx get svc ingress-nginx-controller \
        -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
  [ -n "${IP}" ] && { echo "ip=${IP}" >> "$GITHUB_OUTPUT"; exit 0; }
  sleep 10
done
echo "Timed out waiting for ingress controller external IP" >&2
exit 1

Finally the smoke test goes through the management port via a port-forward rather than the public URL, so readiness is verified whether or not a hostname and trusted certificate exist yet:

- name: Smoke test Keycloak readiness
  run: |
    kubectl -n keycloak port-forward svc/keycloak-keycloakx-http 19000:9000 >/tmp/pf.log 2>&1 &
    PF_PID=$!
    trap "kill ${PF_PID}" EXIT
    bash scripts/smoke-test.sh http://127.0.0.1:19000/health/ready 30 10

Monitoring: three alerts, and the traps in each

This is where I spent the most time getting things correct rather than merely present, because a wrong alert is worse than no alert.

Uptime check. An HTTPS check on /realms/master, plus a policy on it. The condition looks harmless and contains the nastiest trap in the stack:

condition_threshold {
  filter = "metric.type=\"monitoring.googleapis.com/uptime_check/check_passed\" AND resource.type=\"uptime_url\" AND metric.label.\"check_id\"=\"${google_monitoring_uptime_check_config.keycloak_https.uptime_check_id}\""

  comparison      = "COMPARISON_GT"
  threshold_value = 1
  duration        = "120s"

  aggregations {
    alignment_period     = "120s"
    per_series_aligner   = "ALIGN_NEXT_OLDER"
    cross_series_reducer = "REDUCE_COUNT_FALSE"
    group_by_fields      = ["resource.label.host"]
  }

  evaluation_missing_data = "EVALUATION_MISSING_DATA_INACTIVE"
}

REDUCE_COUNT_FALSE turns the boolean check_passed series into a count of probe locations currently failing. So the comparison must be GT: "more than one region reports a failure." COMPARISON_LT belongs to the un-reduced form, where the value is the boolean itself and 0 means down. Pair LT with the reducer — which is exactly what you get by copying half of one example and half of another — and the policy inverts: a healthy service reduces to 0, which is less than 1, so it alerts continuously while everything is fine and goes silent during an actual outage.

The reducer also collapses what would otherwise be one incident per probe location, turning six near-identical emails into one. group_by_fields must use a label the resource actually has: uptime_url carries only host and project_id. Grouping by a label it lacks is not an error — the label is silently dropped, every series collapses into one unlabelled group, and the incident can no longer say which host it is about.

EVALUATION_MISSING_DATA_INACTIVE is deliberate: a gap in uptime data means the probe did not report, not that the service is down. Treating absence as failure means alerting on the monitoring system's own hiccups.

And auto_close = "1800s" (the minimum GCP accepts) — without it, an incident whose metric stops arriving stays open for the API default of seven days, suppressing re-notification for the next genuine outage.

Failed logins. A logs-based metric over the Keycloak pods:

resource "google_logging_metric" "keycloak_failed_logins" {
  filter = <<-EOT
resource.type="k8s_container"
resource.labels.cluster_name="${var.cluster_name}"
resource.labels.namespace_name="keycloak"
(
  textPayload=~"LOGIN_ERROR" OR
  textPayload=~"invalid_user_credentials" OR
  jsonPayload.message=~"LOGIN_ERROR" OR
  jsonPayload.message=~"invalid_user_credentials"
)
EOT

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
  }
}

Then the trap: google_logging_metric returns as soon as the metric is created, but the Monitoring API cannot query it for several minutes. Terraform's dependency graph is satisfied at exactly the wrong moment, and the alert policy referencing it fails with:

Error 404: Cannot find metric(s) that match type = "logging.googleapis.com/user/..."
If a metric was created recently, it could take up to 10 minutes to become available.

Nothing is misconfigured; it is pure eventual consistency. This is the wait Terraform does not know to make:

resource "time_sleep" "wait_for_failed_login_metric" {
  depends_on      = [google_logging_metric.keycloak_failed_logins]
  create_duration = var.metric_propagation_delay   # 180s
}

No triggers, so it costs that once per environment and nothing on later runs.

The threshold aggregation sums across pods with REDUCE_SUM, ungrouped: the requirement is "wrong credentials per minute against Keycloak", not per replica, and the ingress spreads attempts across all three. Left ungrouped on purpose, so adding replicas does not quietly raise the effective threshold.

Degraded replicas. The alert I did not have at first, and the one I would add first next time.

The uptime check answers is the service reachable, which is not the same question as is the service still redundant. The ingress happily returns 200 off a single surviving replica. A three-replica cluster can sit at 1/3 indefinitely with every other alert silent — mine did, for over two hours. That is the state where the next preemption is an outage rather than a survivable event.

condition_prometheus_query_language {
  query = <<-EOT
    kube_statefulset_status_replicas_ready{cluster="${var.cluster_name}",namespace="keycloak",statefulset="${var.statefulset_name}"}
      < on(statefulset,namespace)
    kube_statefulset_replicas{cluster="${var.cluster_name}",namespace="keycloak",statefulset="${var.statefulset_name}"}
  EOT

  duration            = var.replicas_degraded_duration
  evaluation_interval = "60s"
}

PromQL rather than a threshold condition, for one specific reason: the desired replica count lives in the Helm values, not in Terraform. Any number written here would be a second copy of it that drifts the first time someone scales the StatefulSet. Comparing the two metrics reads the desired count from the cluster itself, so the policy has nothing to keep in sync. An empty result is the healthy state.

The duration needs to be long enough that an ordinary rolling restart finishes inside it — podManagementPolicy is OrderedReady, so replicas come back one at a time and a legitimate deploy sits below target for minutes — and short enough that a replica which never comes back is noticed the same hour. And it is a WARNING, not a page: degraded redundancy, not an outage.

Both policies get email notification channels, and the alert email list is an input with an empty default rather than a baked-in address. A recipient hardcoded in a module follows the repository into projects it was never meant to page.

Backups around deploys

Keycloak's schema migrations are not undone by rolling back the Helm chart. PITR covers recovery generally, so a pre-deploy backup is opt-in per environment:

- name: Pre-deploy Cloud SQL backup
  if: ${{ vars.PRE_DEPLOY_BACKUP == 'true' }}
  run: |
    INSTANCE="$(terraform -chdir=terraform output -raw cloudsql_instance_name)"
    gcloud sql backups create --instance="${INSTANCE}" \
      --description="pre-deploy ${{ github.sha }}"

What it adds over PITR is a named restore point tied to a commit, one that outlives the PITR window. Worth the storage in prod; not worth it on every dev merge. It runs synchronously on purpose — a backup that may not have finished is not a restore point.

The short list of things that bit me

If you take nothing else from this:

• Private nodes need Cloud NAT or image pulls fail in a way that looks like a registry outage.
• Private GKE opens only 443 and 10250 to nodes; the ingress-nginx admission webhook on 8443 needs its own firewall rule.
• Cloud SQL reserves deleted instance names for a week — suffix them.
• proxy.mode must be xforwarded behind ingress-nginx, or Keycloak advertises http:// URLs.
• Keycloak's large headers need proxy-buffer-size: 128k or nginx returns 502.
• The default anti-affinity plus one node per replica makes rolling node upgrades dangerous without a PDB.
• --atomic does not save you from a cancelled runner; handle pending-* explicitly.
• REDUCE_COUNT_FALSE needs COMPARISON_GT — the wrong pairing inverts your uptime alert silently.
• Logs-based metrics are not immediately queryable; a time_sleep is the honest fix.
• Reachability is not redundancy — alert on ready-versus-desired replicas separately.

None of this is exotic. It is the ordinary distance between a service that starts and a service you would leave running unattended, and almost all of it is only discoverable by breaking things on purpose and reading what actually happened.

Cost

GKE, Cloud SQL, the load balancer and NAT all cost money while running. If you are following along in a scratch project, tear it down between sessions:

terraform -chdir=terraform destroy -var-file=environments/dev/terraform.tfvars

The seed layer stays, because it lives in its own state — which is the whole reason it was split out to begin with.