ETCD Encryption Configuration for Kubernetes

To enable static encryption at rest for Kubernetes secrets in ETCD, you need to configure the API Server with an EncryptionConfiguration.

1. Generate an Encryption Key

# Generate a 32-byte base64-encoded key
head -c 32 /dev/urandom | base64
# Example output: sxFdbKYquCe3EbRWVV+pFe2lS8K8hbiv3V8ExQZ0fD4=

2. Create EncryptionConfiguration File

Create /etc/kubernetes/encryption-config.yaml on the control plane node:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: sxFdbKYquCe3EbRWVV+pFe2lS8K8hbiv3V8ExQZ0fD4=
      - identity: {}

3. Update kube-apiserver Static Pod Manifest

Edit /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node:

spec:
  containers:
  - name: kube-apiserver
    command:
    - kube-apiserver
    # ... existing flags ...
    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
    volumeMounts:
    - name: encryption-config
      mountPath: /etc/kubernetes
      readOnly: true
  volumes:
  - name: encryption-config
    hostPath:
      path: /etc/kubernetes
      type: DirectoryOrCreate

4. Verify Encryption is Working

# After restarting the API server, create a secret and verify it's encrypted
kubectl create secret generic test-secret --from-literal=key=value -n juwan

# Check if the secret is encrypted in etcd
kubectl get secret test-secret -o yaml

# You can also check raw etcd data (requires etcd access):
# etcdctl --endpoints=https://127.0.0.1:2379 get /kubernetes.io/secrets/juwan/test-secret
# The data should be encrypted (not human-readable)

5. Important Notes

Backup your encryption key in a secure location
Never commit encryption keys to version control
If you lose the key, all encrypted secrets will be unrecoverable
After enabling encryption, existing unencrypted secrets will not be automatically encrypted
- To encrypt existing secrets, you can use: kubectl delete secret <name> && kubectl create secret ...
- Or use: kubectl patch secret <name> -p '{}' --type=merge (triggers re-encryption)

6. RBAC Configuration for JWT Secret

The jwt-secret.yaml includes RBAC rules that:

Create a jwt-secret Secret in the juwan namespace
Create ServiceAccounts for user-rpc and envoy-gateway
Create a Role jwt-secret-reader that allows reading only the jwt-secret Secret
Bind this Role to both ServiceAccounts via RoleBindings

This ensures:

Only user-rpc and envoy-gateway Pods can read the JWT secret
Other services and users cannot access the JWT secret
Least privilege access principle is enforced

7. Update Deployment to Use ServiceAccount

Make sure your Deployment references the ServiceAccount:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-rpc
  namespace: juwan
spec:
  template:
    spec:
      serviceAccountName: user-rpc  # This is important!
      containers:
      - name: user-rpc
        env:
        - name: JWT_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: jwt-secret
              key: secret-key

8. For Minikube Users

If using Minikube, you can enable encryption with:

minikube config set apiserver.encryption-provider-config /path/to/encryption-config.yaml
minikube start

Or manually edit the kube-apiserver manifest after starting Minikube:

minikube ssh
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
# Add the flags and volume mounts as shown above

3.7 KiB Raw Blame History