Add Secrets Securely in Google Kubernetes Engine (GKE) with kubernetes-external-secrets
How to extend GCP Secret Manager to add secret in k8s with kubernetes-external-secrets.
🧾 Requirements
Google Cloud SDK 336.0.0
🚶🏻♂️ Steps
Setup workload identity, IAM binding, and permission
Create mapping between K8s service account (KSA) and GCP service account (GSA), enabling K8s pod to authenticate as GSA and use Google Cloud APIs.
Create a new cluster with workload identity enabled
gcloud container clusters create ${CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog --zone ${ZONE}
Enable workload identity on existing cluster
gcloud container clusters update ${CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog --zone ${ZONE}
Enable workload identity config in the node pool where application pod is running
gcloud container node-pools update ${POOL_NAME} --cluster ${CLUSTER_NAME} --workload-metadata-from-node=GKE_METADATA_SERVER --zone ${ZONE}
Create GSA
GSA=secret-manager
gcloud iam service-account create ${GSA}
GSA_EMAIL=${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com
Create KSA
KSA=dev-kubernetes-external-secrets
kubectl apply -f serviceaccount.yaml
# or
kubectl create serviceaccount ${KSA}
kubectl annotate serviceaccount ${KSA} iam.gke.io/gcp-service-account=${GSA_EMAIL}
Binding KSA with GSA
# Namespace where KSA resides
KSA_NAMESPACE=default
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${KSA_NAMESPACE}/${KSA}]" ${GSA_EMAIL}
Grant GSA secretAccessor permission
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} --member=serviceAccount:${GSA_EMAIL} --role=roles/secretmanager.secretAccessor
Create k8s resources
# Install kubernetes-external-secrets CRDs
kubectl apply -f https://raw.githubusercontent.com/external-secrets/kubernetes-external-secrets/master/charts/kubernetes-external-secrets/crds/kubernetes-client.io_externalsecrets_crd.yaml
Apply kubernetes-external-secrets resources
The following files were generated with Helm
helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
RELEASE_NAME=dev
helm install --dry-run ${RELEASE_NAME} external-secrets/kubernetes-external-secrets
kubectl apply -f rbac.yaml
kubectl apply -f service.yaml
kubectl apply -f deployment.yaml
Create a secret in GCP secret manager
SECRET_NAME=
SECRET_VALUE=
echo -n ${SECRET_VALUE} | gcloud secrets create ${SECRET_NAME} --data-file=-
Create external secret in desired namespace
kubectl apply -f my-external-secret.yaml
Check if the secret exists
kubectl get externalsecret -n ${SECRET_NAMESPACE}
kubectl get secret -n ${SECRET_NAMESPACE}
kubectl get secrets/my-secret --template={{.data.K8S_SECRET_KEY_1}} | base64 -D
Use the secret in pod 🙌🏻🙌🏻
🎰 Optimization
GCP secret manager charges by API calls
The external secret controller will poll for changes to the secret, so setting appropriate interval to avoid surprise on your bill.
POLLER_INTERVAL_MILLISECONDS=
Watch specific namespaces
kubernetes-external-secrets by default watches all namespaces. Adding `WATCHED_NAMESPACES` config in kubernetes-external-secrets controller allows scoping its access to predefined namespaces only.
WATCHED_NAMESPACES: “beta,staging”
😧 Troubleshooting
- Make sure kubernetes-external-secrets controller pod is in the node pool where workload identity is enabled.
- Make sure GSA has secretAccessor role.
- Not to be confused with setting a role on the GCP service account (as a resource), which is for setting members who can view / update the GSA. Granting a particular role to GSA (as an identity) should be done with the above-mentioned command in section `Grant GSA secretAccessor permission`.
📚 References
👨🏻💻 About me
DevOps engineer / SRE based in Taiwan.
#EngineeringSuccess #csiejoey