Configuring Azure Blob containers as staging area

The Hyperscale Compliance Orchestrator requires a staging area to read from the source files and write to the target files. In Kubernetes deployment of Hyperscale Compliance, you can now use Azure Blob Containers as a staging area to access the Hyperscale Compliance Orchestrator and the Continuous Compliance engine’s source and target files.

This feature is supported on MicroK8s and Microsoft’s Azure Kubernetes Service (AKS) distributions and it is only available for Continuous Compliance engine version 28.0.0.0 onwards.

Prerequisites:

For Hyperscale Compliance containers to access Blob objects through a file system interface, a CSI driver (Container Storage Interface) must be configured on the Kubernetes cluster.

To configure the mount point through the CSI driver, a Kubernetes cluster administrator or an Azure account administrator will need to:

  1. Create an Azure Storage account.

  2. Open the Azure storage account and create a Blob Container.

  3. Create and assign Azure’s built-in roles such as Owner or Reader roles to users. For instructions, see Azure RBAC documentation.

  4. Create a managed identity by following the instructions provided in Managed identities for Azure resources.

  5. Install the mount point for Blob CSI driver version v1.25.1 or later. For installation instructions, see Azure Kubernetes Service and MicroK8s.

 

Setting Up Azure Blob Containers with AKS with Managed Identity

The kubelet identity is a user-assigned managed identity that is primarily used by the AKS nodes, unless explicitly overridden, to interact with Azure resources, such as Azure Container Registry (ACR) and Azure Storage.

When creating an AKS cluster, you need to assign the kubelet identity the 'Storage Blob Data Contributor' role on the Azure Blob Container resource. This role is required for the kubelet to interact with Azure Storage for persistent volume mounting.

To enable Managed Identity on AKS cluster, Azure CLI must be installed on the environment before following the below steps. For more information, see Setup managed identity on AKS

 

Retrieve the Kubelet Identity

To use the Kubelet Identity for authentication, retrieve its details using the following:

Copy 
az aks show --resource-group <your-resource-group> --name <your-cluster-name> --query "identityProfile.kubeletidentity"

Example output:
{
  "clientId": "<kubelet-identity-client-id>",
  "objectId": "<kubelet-identity-object-id>",
  "resourceId": "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<kubelet-identity-name>"
}
 

 

  • clientId: The client ID of the Kubelet Identity

  • objectId: The principal ID of the Kubelet Identity

  • resourceId: The resource ID of the Kubelet Identity

 

Assign Roles to the Kubelet Identity

To allow the Kubelet Identity to access the Azure Blob Container, assign the Storage Blob Data Contributor role to the Kubelet Identity.

To create the respective role for the kubelet-identity, run the following:

Copy 
az role assignment create \
  --assignee <kubelet-identity-object-id> \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>

Configure the values.yaml

Provide the azureStorageIdentityClientID obtained above in values.yaml

Copy 
azureStorageIdentityClientID=<kubelet-identity-client-id>

 

Troubleshooting

As MicroK8s uses a different path to other standard distributions, when you are installing the Blob CSI driver on MicroK8s using:/var/snap/microk8s/common/var/lib/kubelet you may receive this error:

Could not check if "/var/snap/microk8s/common/var/lib/kubelet/pods/4a5d6dfb-646a-462d-a223-d510c059f8b7/volumes/kubernetes.io~csi/blob-pv/mount" is a mount point, Failed to read /host/proc/mounts: open /host/proc/mounts: invalid argument

If you encounter the above error, you will need to inform the /var/snap/microk8s/common/var/lib/kubelet path to driver while installing it using helm.

This is the helm chart installation command:

Copy 

helm install blob-csi-driver blob-csi-driver/blob-csi-driver --set node.enableBlobfuseProxy=true -–set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --namespace kube-system --version v1.25.1

You may also need to make the mount location shared:

Copy 
/bin/mount --make-shared /var/snap/microk8s/common/var/lib/kubelet

 

Setup:

When you have completed the steps in the prerequisites section, you will need to create a PV (persistent volume). A PV can be created via cluster administration or by using the Azure Blob related properties defined in values.yaml. Both options are discussed below:

  • Cluster administration creates a PV using the template below and shares the PV name which is configured against the parameter stagePvName in values.yaml.

  • Hyperscale Orchestrator creates the PV automatically during deployment using the Azure Blob related properties defined in values.yaml.

    Copy 
    apiVersion: v1   
    kind: PersistentVolume   
    metadata:   
      annotations:   
        pv.kubernetes.io/provisioned-by: blob.csi.azure.com   
      name: blob-pv   
    spec:   
      capacity:   
        storage: 10Gi   
      accessModes:   
        - ReadWriteMany   
      mountOptions:   
        - -o allow_other   
        - -o direct_io   
        - --file-cache-timeout-in-seconds=0   
        - --use-attr-cache=false   
        - --invalidate-on-sync=true  
        - --attr-timeout=0 
      csi:   
        driver: blob.csi.azure.com   
        volumeHandle: azure-csi-driver-volume-handle # Should be unique for a container   
        volumeAttributes:   
          storageAccount: <STORAGE_ACCOUNT> 
          containerName: <CONTAINER_NAME> 
          AzureStorageAuthType: MSI # key, MSI   
          protocol: fuse2   



The following properties shall be specified in values.yaml (even if PV is created by cluster admin).

  • stagingStorageType

  • blobAccountName

  • blobContainerName

  • blobAuthType

  • blobContainerPrefix

  • blobContainerDelimiter

  • blobSecretKey

  • stagingAzureBlobSecretName

For more information on these parameters, see Commonly Used Properties

 

Additional information

  • Set the stagingStorageType as AZURE_BLOB_STORAGE

  • To connect to the Blob Container from the Continuous Compliance engine, you can use either of the following authentication mechanisms:

    • Instance profile-based authentication (AZURE_MANAGED_IDENTITY)

    • Access and secret key based authentication (AZURE_SECRET)

  • If authMechanism is set as AZURE_MANAGED_IDENTITY, a Managed Identity that grants access to the Blob Container must be attached to the Azure instances of either the Hyperscale Compliance engine, Continuous Compliance engine, or the AKS cluster. Refer to the Azure Kubernetes Services (AKS) knowledge guide for further information.

  • If authMechanism is set as AZURE_SECRET, you will need to provide either of the following:

    • stagingAzureBlobSecretName: Name of the Kubernetes secret that holds blobSecretKey information. The stagingAzureBlobSecretName can be created by a cluster administrator using this command line:

      Copy 

      kubectl create secret generic <secret-name> --from-literal=azurestorageaccountname=<storage-account> --from-literal azurestorageaccountkey=<blob-secret--key> --type=Opaque

    • blobSecretKey: Hyperscale Orchestrator will use these values and automatically create a generic secret.