While building a kubernetes cluster to deploy your containers in there you will find that most of the times you need to pass information to those containers for them to work properly, this can be a connectio nstring to a certain resource (databases, redis cache, storage accounts, etc).
Usually you will define an environment variable for the container and will set the value when running the deployment to kubernetes, this works just fine but the downside is that those values are visible to anyone who does a "describe" of the pod containing the containers.
This is a problem when you consider to version your yaml files, this will means that the values of those environment variables will be placed in the repository and more importantly if you deploy to multiple environments where you have different values for those secrets you can not have them hardcoded in your yaml file, instead, you need them to be dynamically adjusted to the environment where you are deploying them.
In this article you will learn how to configure an AKS cluster (Microsoft Azure offer for kubernetes) to consume secrets, keys and certificates from an Azure KeyVault (secure store offer from Microsoft Azure).
The first thing you will need of course is the AKS cluster itself and a keyvault to store your sensitive data.
az aks create --resource-group "aks-demo" --name "aks-cluster" --network-plugin azure --enable-managed-identity
With this command you will create an AKS cluster called "aks-cluster" in the resource group "aks-demo". An important thing to note is the "--enabled-managed-identity" flag, this will create a managed identity that the cluster will use to manage it's interaction with Azure, this is needed for this whole article to work.
Then we will create a keyvault
az keyvault create --location westus2 --name "aks-keyvault" --resource-group "aks-demo"
Here we are creating the keyvault called "aks-keyvault" in the same resource group as our AKS cluster (please note that in that resource group you will only see the AKS resource but all the needed infrastructure for that cluster will be built in another resource group which name starts with "MC_" this can be changed when creating the aks cluster if needed). We are choosing a location to deploy our keyvault as "westus2" but this is not mandatory to be the location.
Now we need to configure our access to the AKS cluster and we will do this using the Azure Cli as well!!
az aks install-cli
This will install kubectl which is a tool to manage our aks/k8s cluster from the command line.
Next we need to setup our kubectl to access our AKS cluster
az aks get-credentials --resource-group "aks-demo" --name "aks-cluster"
At this point we can test if the connection from kubectl to the AKS cluster is correct by simply running a "kubectl get nodes"
This whole solution works by adding a couple of demonsets and pods that will manage the access to the keyvault to retrieve the secreta/keys/certs and place them inside the pods you choose, thankfully we can do this by installing some helm charts in our AKS.
First we need to install the repos and the charts
helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name
This will build a demonset and you will see it with this output (this was the name in my machine but will be different in your case as the name is autogenerated because of the --generate-name)
NAME: csi-secrets-store-provider-azure-1594729889 LAST DEPLOYED: Tue Jul 14 14:31:29 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None
Our next step is to build a set of resources in our cluster that will manage the integration between the cluster, azure and the kevault it self, this set of resources will notice when a change is done in the pods (a redeploy) and will take care of going to the keyvault and extract the values to populate your pods configurations, they will also manage the selection of which secret/key or cert to extract from the keyvault.
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
the output should look something similar to this
serviceaccount/aad-pod-id-nmi-service-account created customresourcedefinition.apiextensions.k8s.io/azureassignedidentities.aadpodidentity.k8s.io configured customresourcedefinition.apiextensions.k8s.io/azureidentitybindings.aadpodidentity.k8s.io configured customresourcedefinition.apiextensions.k8s.io/azureidentities.aadpodidentity.k8s.io configured customresourcedefinition.apiextensions.k8s.io/azurepodidentityexceptions.aadpodidentity.k8s.io configured clusterrole.rbac.authorization.k8s.io/aad-pod-id-nmi-role created clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-nmi-binding created daemonset.apps/nmi created serviceaccount/aad-pod-id-mic-service-account created clusterrole.rbac.authorization.k8s.io/aad-pod-id-mic-role created clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-mic-binding created deployment.apps/mic created
Now we will need a managed identity (User assigned) that will be used to read the values from our keyvault, we can do this using the azure cli as follows
az identity create -g "aks-demo" -n aksownidentity
The output will show you several lines of information about the configuration of this identity but the ones you will use in the next steps are the one called "principalId" and the "clientID"
With this identity created we will continue to assign the "Reader" role at the keyvault level
az role assignment create --role Reader --assignee "PRINCIPALID OF IDENTITY" --scope /subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourcegroups/aksdemo/providers/Microsoft.KeyVault/vaults/aks-keyvault
Next we need to provide access to the identity to read secrets, keys and certificates, you don't need to include the tree, if you only need to read secrets, only add the "get" permission to "secrets"
# set policy to access keys in your keyvault az keyvault set-policy -n aks-keyvault --key-permissions get --spn CLIENT_ID # set policy to access secrets in your keyvault az keyvault set-policy -n aks-keyvault --secret-permissions get --spn CLIENT_ID # set policy to access certs in your keyvault az keyvault set-policy -n aks-keyvault --certificate-permissions get --spn CLIENT_ID
After this step all the infrastructure permissions and deployments are done, you only need to provision some items inside your kubernetes cluster and you are all set, let's see those yaml files
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentity metadata: name: aksuseridentity spec: type: 0 resourceID: /subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourcegroups/aksdemo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aksownidentity clientID: CLIENT ID OF YOUR IDENTITY
Store this yaml as "aadpodidentity.yaml" and execute it in your aks with
kubectl apply -f aadpodidentity.yaml
Please note the name of this resource, you will need it in the next yaml file
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentityBinding metadata: name: aksidentitybinding spec: azureIdentity: aksuseridentity selector: aksidentitybindingselector
Note in the azureIdentity we are using the same as the name of our previous resource "aksuseridentity" in this example.
The selector is the tag with which you will identify this resource when accessing the keyvault from a pod.
Store this file as "aadpodidentitybinding.yaml" and do
kubectl apply -f aadpodidentitybinding.yaml
Now our last step is to deploy a pod and consume a secret from our keyvaul, first go to your keyvault in Azure and create a secret, a key and a certificate (depending on which permissions you granted previously you might want only to create the one you provided access for)
I will be using a simple linux container running alpine to present the skeleton of how should look our deployment definition to consume the keyvault but please feel free to use the one that fits the best for you
apiVersion: apps/v1 kind: Deployment metadata: name: alpine labels: app: alpine spec: replicas: 1 template: metadata: name: alpine labels: app: alpine aadpodidbinding: "aksidentitybindingselector" spec: nodeSelector: "beta.kubernetes.io/os": linux containers: - image: nginx name: nginx volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: providerName: "azure" usePodIdentity: "true" # [OPTIONAL] if not provided, will default to "false" keyvaultName: "aks-keyvault" # the name of the KeyVault cloudName: "" # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud objects: | array: - | objectName: secret1 objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty resourceGroup: "aks-demo" subscriptionId: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # [REQUIRED for version < 0.0.4] the subscription ID of the KeyVault tenantId: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # the tenant ID of the KeyVault selector: matchLabels: app: alpine
Again store this yaml as "mypod.yaml" and execute
kubectl apply -f mypod.yaml
Some details of this last yaml
The "aadpodbinding" should be the same as the "selector" you put in the aadpodidentitybinding.yaml file
The mountPath is the place where the secrets, keys and certificates will be stored, they will be represented as files in that directory
For each item (secret, key or certificate) that you want to access from the keyvault you will need an entry in "objects"
The tenantId is the directory id where you have the subscription that contains the keyvault.
After you complete this steps you can go inside your container and check that directory (/mnt/secrets-store) to verify if your secret is in there.
kubectl exec -it PODNAME /bin/bash ls -l /mnt/secrets-store
you should see one file per secret/key/certificate that you defined in the objects in the deployment yaml that you executed.
An important note is that if you added a new item in the keyvault, it will be only visible if you add it in the objects in the deployment and redeploy it also if you change the value of the secret in the keyvault, you need to force a redeploy in kubernetes to read the new value, this can be done adding a label in the deployment that you can change and apply the deployment again to aks, this will force a reconfiguration in kubernetes that will update the value of the file inside the pods.
I decided to create this article as I found there is a lot of information on this topic but it took me some time to understand how to configure everything properly to make it to work, it is not complicated but sometimes the documentation links other documentation and following that chain is complex so I decided to put together the minimum needed steps to make this work (also tested all of them myself)
I will place here the links to the documentation of each part that makes this happen as the people working in this deserves the kudos and love, please go and share your love with them
If at any step of this guide you can't continue or find something confusing, please don't hesitate to let me know and reach back to me I will try to help you the best as I can, we are all in the same boat constantly learning ;)
If you liked this article, please consider letting me know by liking this article, leaving a comment or sharing this article with others, you can also follow me in my networks.