Add Secrets Securely in Google Kubernetes Engine (GKE) with kubernetes-external-secrets

How to extend GCP Secret Manager to add secret in k8s with kubernetes-external-secrets.

🧾 Requirements

Google Cloud SDK 336.0.0

🚶🏻‍♂️ Steps

Setup workload identity, IAM binding, and permission

Create mapping between K8s service account (KSA) and GCP service account (GSA), enabling K8s pod to authenticate as GSA and use Google Cloud APIs.

Create a new cluster with workload identity enabled

gcloud container clusters create ${CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog --zone ${ZONE}

Enable workload identity on existing cluster

gcloud container clusters update ${CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog --zone ${ZONE}

Enable workload identity config in the node pool where application pod is running

gcloud container node-pools update ${POOL_NAME} --cluster ${CLUSTER_NAME} --workload-metadata-from-node=GKE_METADATA_SERVER --zone ${ZONE}

Create GSA

GSA=secret-manager
gcloud iam service-account create ${GSA}
GSA_EMAIL=${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com

Create KSA

KSA=dev-kubernetes-external-secrets
kubectl apply -f serviceaccount.yaml
# or
kubectl create serviceaccount ${KSA}
kubectl annotate serviceaccount ${KSA} iam.gke.io/gcp-service-account=${GSA_EMAIL}

Binding KSA with GSA

# Namespace where KSA resides
KSA_NAMESPACE=default
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${KSA_NAMESPACE}/${KSA}]" ${GSA_EMAIL}

Grant GSA secretAccessor permission

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member=serviceAccount:${GSA_EMAIL} --role=roles/secretmanager.secretAccessor

Create k8s resources

# Install kubernetes-external-secrets CRDs
kubectl apply -f https://raw.githubusercontent.com/external-secrets/kubernetes-external-secrets/master/charts/kubernetes-external-secrets/crds/kubernetes-client.io_externalsecrets_crd.yaml

Apply kubernetes-external-secrets resources

The following files were generated with Helm

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
RELEASE_NAME=dev
helm install --dry-run ${RELEASE_NAME} external-secrets/kubernetes-external-secrets

kubectl apply -f rbac.yaml
kubectl apply -f service.yaml
kubectl apply -f deployment.yaml

Create a secret in GCP secret manager

SECRET_NAME=
SECRET_VALUE=
echo -n ${SECRET_VALUE} | gcloud secrets create ${SECRET_NAME} --data-file=-

Create external secret in desired namespace

kubectl apply -f my-external-secret.yaml

Check if the secret exists

kubectl get externalsecret -n ${SECRET_NAMESPACE}
kubectl get secret -n ${SECRET_NAMESPACE}
kubectl get secrets/my-secret --template={{.data.K8S_SECRET_KEY_1}} | base64 -D

Use the secret in pod 🙌🏻🙌🏻

🎰 Optimization

GCP secret manager charges by API calls

The external secret controller will poll for changes to the secret, so setting appropriate interval to avoid surprise on your bill.

POLLER_INTERVAL_MILLISECONDS=

Watch specific namespaces

kubernetes-external-secrets by default watches all namespaces. Adding `WATCHED_NAMESPACES` config in kubernetes-external-secrets controller allows scoping its access to predefined namespaces only.

WATCHED_NAMESPACES: “beta,staging”

😧 Troubleshooting

• Make sure kubernetes-external-secrets controller pod is in the node pool where workload identity is enabled.
• Make sure GSA has secretAccessor role.
• Not to be confused with setting a role on the GCP service account (as a resource), which is for setting members who can view / update the GSA. Granting a particular role to GSA (as an identity) should be done with the above-mentioned command in section `Grant GSA secretAccessor permission`.

📚 References

Using Workload Identity | Kubernetes Engine Documentation

external-secrets/kubernetes-external-secrets

Adding roles to service accounts on Google Cloud Platform using REST API

👨🏻‍💻 About me

DevOps engineer / SRE based in Taiwan.

#EngineeringSuccess #csiejoey

csiejoey - Overview

Joey Huang Official Website | A Tech Star Is Born | 91 Till Infinity