AWS Secrets Manager for Kubernetes: Tutorial & Best Practices

Effective secrets management, such as securing tokens and passwords, is essential to Kubernetes security. Kubernetes provides a native secret management capability, Kubernetes Secrets, to manage and store sensitive information like passwords, API keys, database credentials, and certificates in Kubernetes. However, Kubernetes Secrets, by default, stores your secrets unencrypted in the API server’s data store (etcd). This can make the secrets vulnerable to anyone with access to etcd, API server, or permission to create a Pod in that namespace.

AWS Secrets Manager is a popular external secret management service supported by Kubernetes that can provide enhanced secrets protection. Secrets Manager provides a robust and enterprise-grade secret storage solution compared to native Kubernetes Secrets.

This article will explore AWS Secrets Manager for Kubernetes and how you can integrate AWS Secrets Manager with Elastic Kubernetes Services (EKS) and Kubernetes on an EC2 instance. We will also review how tools like Doppler can improve Kubernetes secret management with features like integrating multiple secret stores and automatically injecting secrets into your Kubernetes workloads.

Summary of key Kubernetes secret management best practices

The table below summarizes five Kubernetes secret management best practices this article will explore in depth.

Five best practices for Kubernetes and AWS Secrets Manager

The five best practices below enable organizations to reduce the risk of a secret being compromised and reduce the damage a threat actor can do if a secret is exposed.

Use the dedicated Secrets object to store secrets instead of ConfigMaps

Kubernetes Secrets stores sensitive data, such as passwords, OAuth Tokens, SSH keys, and API keys, in base64 encoded format. ConfigMaps store values like environment variables as plain text. Using Kubernetes Secrets allows users to enable secret encryption at rest, restrict access to the secrets, and rotate them periodically.

Enable encryption at rest

Kubernetes Secrets are, by default, stored in etcd as base64 encoded unencrypted strings. Enabling Secrets encryption at rest provides an extra layer of security and helps ensure the secrets remain confidential. The encryption is managed by the Kubernetes control plane and by default, only Pods that need access to a Secret can view the decrypted Secret data

Rotate secrets periodically

Rotating secrets periodically can help limit the vulnerability window if a secret is compromised. Regularly rotating secrets make it harder for attackers to steal and exploit secrets over long periods. Also, specific certificates and API keys have expiration dates, so rotating ensures the secret is still valid.

{{banner-1="/design/banners"}}

Restrict access to secrets

Access control is fundamental to a strong security posture. The four tips below can help teams implement strong access controls to protect their Kubernetes secrets.

• Use Kubernetes Role-Based Access Control (RBAC) to restrict access to Secrets based on roles and role bindings. Administrators can implement least privilege access by creating roles that provide only the minimum access required for business use cases.
• Limit lateral movement by creating namespace-scoped roles with get or list access to specific secrets, and avoid granting full access to secrets in a namespace.
• Bind roles to only the service accounts that need the secrets.
• Define volume mount or environment variable configuration if you have multiple containers in a Pod and only a few of them need access to Secrets.

Use a centralized secrets management store

Using a centralized secrets management store like AWS Secrets Manager or Azure Key Vault can offer many benefits, such as easier management of secrets, enhanced security, auditing, integration support, and automatic secret injection. This can provide a more scalable and secure approach than storing secrets using Kubernetes Secrets.

Eight limitations of Kubernetes native secret management

The native Kubernetes Secrets solution has the benefit of being simple and being part of Kubernetes by default. However, it lacks many security features that production-grade solutions may need. Specifically, here are eight limitations of Kubernetes Secrets:

• Unencrypted by default: All the Secrets are stored as base64 encoded unencrypted strings in the Kubernetes etcd database. This can be a security risk if etcd is compromised.
• Native encryption issues: While native encryption can protect the secrets from an etcd compromise, a skilled attacker can easily extract the encryption keys and get the secrets if the host is compromised.
• No automatic secret rotation: Kubernetes does not have built-in support for automatic rotation. You must update secret resources to rotate secrets manually.
• No audit logging: Kubernetes does not log when applications or users access the secrets.
• No secret versioning: Kubernetes Secrets does not support secret versioning. The old version of your secret is lost when you update it.
• Manual generation and injection: Secrets must be manually generated and injected into Pods. There is no support for the automatic generation of secrets.
• The risk associated with Pods: By default, all the containers in a Pod can access the secrets defined in that namespace. You must define volume mounts or environment variable configurations to restrict access.
• No centralized secrets discovery: Secrets are spread across namespaces and cannot be managed or discovered from a central location.

Managing Kubernetes Secrets using AWS Secrets Manager

AWS Secrets Manager is a popular alternative solution for managing secrets in Kubernetes. It can improve Kubernetes secret management by providing the following advantages over native Kubernetes Secrets:

• Encrypted by default: All Secrets Manager secrets are encrypted at rest and in transit using AWS-managed keys, providing an extra layer of security. Users can also create Customer-managed keys if they want complete control over the keys and encrypt the secrets using these.
• Automatic secret rotation: Secrets Manager provides built-in support for automatic rotation for certain types of secrets.
• Audit logging: Secrets Manager supports integration with AWS CloudTrail to monitor the audit logs for the secrets.
• Secret versioning: By default, secrets have versioning for up to 3 versions (current, previous, and pending version). Additionally, users can attach labels to the secrets so they are not removed automatically.
• Access control: Secrets Manager allows fine-grained IAM policies at the resource level to limit access to your secrets.
• Centralized secrets discovery: All the secrets in Secrets Manager can be managed and discovered from a central console.

{{banner-2="/design/banners"}}

Limitations of managing Kubernetes Secrets using AWS Secrets Manager

Though AWS Secrets Manager can improve secret management in Kubernetes, it faces the following limitations:

• Limited secret rotation support: Secrets Manager supports automatic secret rotation for only certain types of secrets. For others, users will have to create a Lambda function to carry out the rotation. This can be cumbersome and error-prone.
• No automatic update of secrets: Secrets Manager does not have a built-in capability to refresh secrets automatically in Kubernetes Pods. Users will either have to manually restart the Pods or use features like rotation reconciler to refresh secrets.
• Complex audit logging: Auditing functionality is not built into Secrets Manager. AWS CloudTrail is difficult to set up and comes with additional cost.
• No secrets categorization: Even though all the secrets in Secrets Manager are displayed in a central console, it does not support categorizing secrets to manage secrets related to your application easily.

How to integrate AWS Secrets Manager with Kubernetes

Now that we understand how AWS Secrets Manager can address Kubernetes Secrets limitations, let’s see how we can integrate Secrets Manager with EKS and Kubernetes on an EC2 instance.

This tutorial assumes you already have Kubernetes clusters and Secrets Manager stores. We will cover two use cases:

• AWS Secrets Manager with EKS
• AWS Secrets Manager with Kubernetes on an EC2 instance

AWS Secrets Manager with EKS

The following method works only for EKS versions 1.17 and above, having Amazon EC2 node groups (AWS Fargate node groups are not supported). If you want to integrate AWS Secrets stores on EKS Clusters with Fargate node groups, follow this article from AWS.

Create a Kubernetes service account

First, create a Kubernetes service account in the EKS Cluster by running the following command. Make sure to replace the name and namespace with the desired values.

cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
EOF
kubectl apply -f my-service-account.yaml

Create OpenID Connect provider

Next, let’s create the OpenID Connect (OIDC) provider associated with the EKS cluster in your AWS account so that your cluster can access AWS resources.

Grab the OIDC provider URL of your cluster from the EKS Cluster dashboard under the Overview tab.

Navigate to the IAM console, click on Identity providers, and choose Add provider.

Choose OpenID Connect for the Provider type and enter the Provider URL that you copied from the above step. Then, click on Get thumbprint and add Audience as sts.amazonaws.com.

Finish creating the Identity provider by clicking on Add provider.

{{banner-3="/design/banners"}}

Create IAM policy and role

Now, let’s create the IAM policy that grants access to the Secrets Manager store. The IAM role associated with this policy will be used to limit access to secrets within namespaces in the cluster by attaching it to the Kubernetes service account.

Navigate to the IAM console and create a policy that allows “secretsmanager:GetSecretValue” and “secretsmanager:DescribeSecret” access to your secret.

Once you have the IAM policy ready, create the IAM role that your Kubernetes service account can use.

Go to the IAM console, click Create role, and choose Custom trust policy.

Next, add the following trust policy replacing $account_id, $oidc_provider, $namespace, and $service_account with appropriate values.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:$namespace:$service_account"
        }
      }
    }
  ]
}

Here’s an example of how your trust policy should look:

Finish the creation of your IAM role by attaching the policy that you previously created.

Now that you’ve created your IAM role, let’s attach it to the service account in the Kubernetes namespace you specified by running the following command:

kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=$roleARN

Replace $namespace, $service_account, and $roleARN with appropriate values from above steps.

{{banner-4="/design/banners"}}

Install AWS Secrets and Configuration Provider

Next, let’s install the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver. We will be using this to fetch secrets from AWS Secrets Manager and mount them into Pods.

To install Secrets Store CSI Driver and ASCP using Helm, use the following commands:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver


helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Mount the secrets

Once you’ve finished setting up ASCP, create a SecretProviderClass that contains the secrets you want to mount. We will reference this resource to mount the secrets to the containers inside the Pods.

Note that the SecretProviderClass should be created in the same namespace you created the service account.

Run the following command to create the SecretProviderClass:

cat >secret-provider-class.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: default
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "stage/myapp/secret"
          objectType: "secretsmanager"
EOF
kubectl apply -f secret-provider-class.yaml

You can view more details on the other supported secret mounting options provided by the SecretProviderClass here.

Now that the secret store is ready, let’s verify if the secrets can be mounted by creating a deployment using the following command:

cat >sample-deployment.yaml <<EOF
kind: Service
apiVersion: v1
metadata:
  name: secret-app
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-app
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: my-service-account
      volumes:
      - name: secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: aws-secrets
      containers:
      - name: secret-app
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets-store"
          readOnly: true
EOF
kubectl apply -f sample-deployment.yaml

You can log into the container in the Pod to verify if the secrets are stored in it or run the following command to do that:

kubectl exec -it $(kubectl get pods | awk '/secret-app/{print $1}' | head -1) -- cat /mnt/secrets-store/stage_myapp_secret; echo

In the above example, secrets are mounted as volume inside the containers. You can also expose secrets to containers using environment variables.

Update secrets inside Pods after changing the secret value

Since Kubernetes Pods do not automatically pull the Secrets Manager changes, we will have to manually restart the Pods to update the secrets. You can do this using the following command:

kubectl rollout restart deployment secret-app

This will ensure that your Pods restart gradually without downtime and refresh the secrets with the latest values.

Optional periodic remount of secrets using rotation reconciler

Secrets Store CSI Driver’s rotation reconciler is currently an alpha feature that periodically remounts the secrets in the SecretProviderClass so that the secrets are automatically updated in your Pods when they’re rotated.

To enable this optional feature, you can use the following command:

helm upgrade -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set enableSecretRotation=true --set rotationPollInterval=3600s

Note that this runs for a defined interval and can create additional API calls and charges. Also, users might still need to restart their Pods based on their application as it might only re-read secret changes inside the application during the container startup.

Secrets Manager with Kubernetes on an EC2 instance

Accessing Secrets Manager stores within Kubernetes on an EC2 instance can be done with the help of External Secrets Operator. However, this requires an IAM user credential that the External Secrets Operator can use to assume the IAM role attached to your secret, to retrieve the secret.

Follow the tutorial here to learn how to integrate Secrets Manager with Kubernetes on an EC2 instance.

How Doppler improves AWS Secrets Manager Kubernetes integration

Doppler is a centralized secret management platform that unifies handling secrets across AWS, Azure, GCP, GitHub, GitLab, and Kubernetes among others. It can improve the Secrets Manager Kubernetes integration by providing the following capabilities:

• Automatic secret injection: Doppler can automatically inject secrets from Secret Manager into Kubernetes when the secrets are updated, without requiring manual triggers to ensure applications always have the latest version of secrets.
• Simplified configuration: Doppler can automate the configuration of the Secrets Controller and IAM roles, making it easier to integrate Secrets Manager with Kubernetes.
• Centralized policy management: Doppler allows you to define policies centrally and apply them to relevant Secrets Controller roles, simplifying permissions management.
• Unified audit logs: Doppler can log all secret access events from both Secrets Manager and Kubernetes with its built-in auditing capability, providing a single place to view audit logs.
• Automated secret rotation: Doppler can rotate secrets stored in Secrets Manager and propagate the new values to your Pods, without manual intervention.
• Centralized secret discovery: Doppler provides a unified platform to manage all your secrets in a central place, that could be spread across multiple environments and cloud providers.
• Multi-cluster support: Doppler can manage secrets from a single console for multiple Kubernetes clusters connected to the same Secrets Manager.

Doppler can integrate your secrets directly with Kubernetes using the Doppler Secrets Operator, External Secrets Operator, CI/CD Secrets Sync, and Doppler CLI. Learn more in the official Doppler documentation.

{{banner-5="/design/banners"}}

Conclusion

Kubernetes' native Secrets Manager is often insufficient for enterprises and production workloads. Fortunately, organizations can address complex use cases and improve security posture with the right practices and tools. For example, third-party tools like AWS Secrets Manager and Doppler can tightly integrate with Kubernetes and provide robust security controls. As a result, teams can leverage the power of Kubernetes without compromising on security.