How to Mount Secret Manager Secret in EKS Pod: [10 Easy Steps]

Introduction

In this tutorial, we will learn about how to mount secret manager secret in EKS Pod using 10 easy steps. When it comes to enhancing the security and manageability of applications deployed in AWS, Secret Manager plays an important role as it reduces the chance of exposing sensitive credentials. It also provides a secret rotation feature using which credential can be changes after every certain period which protects the data from un authorized access and potential security breaches. We will learn about Secret Manager service in detail along with a demo in the upcoming sections of this tutorial. So let’s get started.

Secret Manager Overview

AWS Secret Manager is a service that helps you protect access to your application, services and IT resources without exposing sensitive information directly in your code or configuration files. It allows you to store, retrieve and rotate (periodically change) secrets such as database passwords, API keys, encryption keys and other sensitive data. Some key features of Secret Manager are given below.

• Secret Storage
• Automatic Rotation
• Integration with AWS Services
• Fine-grained Access Control
• Auditing and Logging
• Tagging and Organization

How to Mount Secret Manager Secret in EKS Pod: [10 Easy Steps]

Also read: How to Create Key Pairs Using AWS Management Console

There are multiple steps involved in order to create the secret using secret manager and mount the secret inside a Pod. Please make sure all the given prerequisite are met before you start with below steps for this tutorial.

Prerequisite

• EC2 instance created
• helm3 package installed on EC2 instance
• eksctl utility installed on EC2 instance
• EKS cluster created

If you need help with any of the prerequisites, please check below refernces.

How to Create EC2 Instance Using 10 Easy Steps
How to install Helm3
How to Provision EKS Cluster in AWS Using 10 Easy Steps

Step-1:  Create Secret using Secret Manager  

In this step, we will set two shell variables which will be used throughout this tutorial.  REGION will store the region where EKS cluster is created. CLUSTERNAME will store the name of cluster which is created in set REGION. Set the variables using below commands.

[ec2-user@linuxnasa]$ export REGION=<region-name>
[ec2-user@linuxnasa]$ export CLUSTERNAME=<cluster-name>

Next,  we will create the secret manager secret usi CLI command. We can also create the secret manager secret using the secret manager console. We will create the secret manager secret and store extract the ARN from the command output and store it in the ‘SECRET_ARN‘ shell variable. Below is the command syntax.

Syntax:
aws –region “$REGION” secretsmanager create-secret –name <secret-name> –secret-string ‘{“key1″:”value1”, “key2″:”value2”}’ –query ‘ARN’ –output text

I have created the secret ‘dev/secret/Usercredential’ which will store two data. username with value developer and password with value pass123 as shown below.

[ec2-user@linuxnasa]$ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret --name dev/secret/Usercredential --secret-string '{"username":"developer", "password":"pass123"}' --query 'ARN' --output text)

Next, if above command is executed successfully, you can also check the value set to SECRET_ARN variable using below command.

[ec2-user@linuxnasa]$ echo $SECRET_ARN
arn:aws:secretsmanager:ap-south-1:451867090622:secret:dev/secret/Usercredential-DjVw2x

Step-2:  Create IAM Policy

In this step, we will create a policy for the pod that limits its access to the secret we created in step-1. We are also storing the ARN of the policy from the command output in a shell variable ‘POLICY_ARN‘. This variable will be required in upcoming steps.

[ec2-user@linuxnasa]$ POLICY_ARN=$(aws --region "$REGION" --output text iam create-policy --query Policy.Arn --policy-name secretmanager-iam-policy --policy-document '{
"Version": "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
"Resource": ["$SECRET_ARN"]
} ]
}')

In the above command, we give ‘GetSecretValue‘ and ‘DescribeSecret‘ permission to the pod via the policy. Pod will not have any other permission on the secrets other than the given permission.  I have given the <policy-name> ‘secretmanager-iam-policy’ in the above command.

Step-3: Create an IAM OIDC Provider

In this step, we will create the IAM OIDC for the cluster if it does not already exist using below command.

[ec2-user@linuxnasa]$ eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve

Step-4:  Create Service Account for Pod

In this step, we will create a service account and attach the policy to the service account using below command

[ec2-user@linuxnasa]$ eksctl create iamserviceaccount --name pod-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

If above command executes successfully, you will see a service account created using below command.

[ec2-user@linuxnasa]$ kubectl get sa pod-deployment-sa
NAME                   SECRETS   AGE
pod-deployment-sa      0         58s

Step-5: Install Secret Store CSI Driver

In this step, we will install Secret Store CSI Driver which integrates secrets stores with Kubernetes via CSI volume. First add the helm repo as shown below.

[ec2-user@linuxnasa]$ helm3 repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
"secrets-store-csi-driver" has been added to your repositories

Next, install the secret store csi driver as shown below

[ec2-user@linuxnasa]$ helm3 install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system
NAME: csi-secrets-store
LAST DEPLOYED: Tue Sep 19 12:26:34 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Secrets Store CSI Driver is getting deployed to your cluster.

To verify that Secrets Store CSI Driver has started, run:

kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"

Now you can follow these steps https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html
to create a SecretProviderClass resource, and a deployment using the SecretProviderClass.

Make sure pods are running. Use below command to verify.

[ec2-user@linuxnasa]$ kubectl get po -A | grep secret
kube-system    csi-secrets-store-secrets-store-csi-driver-7bjzg   3/3   Running   0  8m47s
kube-system    csi-secrets-store-secrets-store-csi-driver-pxb54   3/3   Running   0  8m47s

Step-6: Install AWS Secrets and Config Provider

In this step, we will install ASCP (AWS Secrets and Configuration Provider). This is required to mount the Secret Manager secret in the Pod as file.

[ec2-user@linuxnasa]$ helm3 repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
"aws-secrets-manager" has been added to your repositories

[ec2-user@linuxnasa]$ helm3 install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws
NAME: secrets-provider-aws
LAST DEPLOYED: Tue Sep 19 12:35:02 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Make sure pods are running. Use below command to verify.

[ec2-user@linuxnasa]$ kubectl get po -A | grep secrets-provider
kube-system    secrets-provider-aws-secrets-store-csi-driver-provider-awsnk47l        1/1      Running    0    50s
kube-system    secrets-provider-aws-secrets-store-csi-driver-provider-awsvrh7d        1/1      Running    0    50s

Step-7:  Create ServiceProviderClass

In this step, we will write a configuration file to create SecretProviderClass. We will specify the secret which will be mount in the pod in SecretProviderClass resource. Create a new file and add below content to it.

[ec2-user@linuxnasa]$ cat secret-storage-class.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: secret-manager-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "dev/secret/Usercredential"
        objectType: "secretsmanager"
        jmesPath:
            - path: username
              objectAlias: user
            - path: password
              objectAlias: pass

Save the above file and apply the configuration file using below command.

[ec2-user@linuxnasa]$ kubectl apply -f secret-storage-class.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/secret-manager-secrets created

You can check if the SecretProviderClass is created successfully using below command.

[ec2-user@linuxnasa]$ kubectl get secretproviderclass
NAME                         AGE
secret-manager-secrets        87s

Step-8: Create and Deploy Pod

In this step, we will create a new pod using Deployment and mount the secret created in previous step. Create a new file and add below content to it.

[ec2-user@linuxnasa]$ cat pod-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: pod-deployment-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "secret-manager-secrets"
      containers:
      - name: pod-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true

Save the above file and apply the configuration file using below command.

[ec2-user@linuxnasa]$ kubectl apply -f pod-deployment.yaml
deployment.apps/pod-deployment created

Check the pod status using below command. If there are no issues, pod will be in running state.

[ec2-user@linuxnasa]$ kubectl get po
NAME                                        READY        STATUS     RESTARTS   AGE
pod-deployment-5fc95558f8-zcffq             1/1          Running     0         19s

Step-9:  Verify Mount Secret in Pod

In this step, login to the pod using below command.

[ec2-user@linuxnasa]$ kubectl exec -it <pod-name> bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@pod-deployment-5fc95558f8-zcffq:/#

Next, switch to the path inside the pod where secrets are mounted i.e “/mnt/secret-store” using below command.

root@pod-deployment-5fc95558f8-zcffq:/# cd /mnt/secrets-store/

Next, list down all the files in the path. You will see there are three files residing in this path as shown below.

root@pod-deployment-5fc95558f8-zcffq:/mnt/secrets-store# ls -lhtr
total 12K
-rw-r--r-- 1 root root 9 Sep 19 14:34 user
-rw-r--r-- 1 root root 7 Sep 19 14:34 pass
-rw-r--r-- 1 root root 46 Sep 19 14:34 dev_secret_Usercredential

Next, cat each of the file and see the content as shown below. You will find the secret data mapped in these files.

root@pod-deployment-5fc95558f8-zcffq:/mnt/secrets-store# cat user
developer

root@pod-deployment-5fc95558f8-zcffq:/mnt/secrets-store# cat pass
pass123

root@pod-deployment-5fc95558f8-zcffq:/mnt/secrets-store# cat dev_secret_Usercredential
{"username":"developer", "password":"pass123"}

Step-10: Delete the Pod

In this step, delete all the resource post completion of he POC. Since we have created the Pod using Deployment, we will delete the deployment resource which will automatically terminates the Pods. Use below command.

[ec2-user@linuxnasa]$ kubectl get deployment

[ec2-user@linuxnasa]$ kubectl delete deployment <deployment-name>

Possible Errors

If you see above error, that means there is issue creating a role using the Policy file we have created in step-2. I will suggest you to create the IAM role using the IAM console and attach the created Policy file to it. Then execute below command in order in step-4.

[ec2-user@linuxnasa]$ export ROLE_ARN=arn:aws:iam::451867090622:role/<role-name>

[ec2-user@linuxnasa]$ eksctl create iamserviceaccount --name <service-account-name> --region="$REGION" --cluster "$CLUSTERNAME" --attach-role-arn=$ROLE_ARN --approve --override-existing-serviceaccounts

If you get above error, that means you have not specified the objectType in SecretProviderClass configuration file in step-7. Make sure below parameter is added in the configuration file.

Summary

We have successfully mounted the Secret Manager secret inside a POD. To know more about AWS Secret Manager service, refer to aws official documentation.