Apply GitOps configurations on AKS as an Azure Arc Connected Cluster using Azure Policy for Kubernetes

The following README will guide you on how to enable Azure Policy for Kubernetes on an Azure Kubernetes Service (AKS) cluster that is projected as an Azure Arc connected cluster as well as how to create GitOps policy to apply on the cluster.

Note: This guide assumes you already deployed an AKS cluster and connected it to Azure Arc. If you haven’t, this repository offers you a way to do so in an automated fashion using either ARM Template or Terraform.

Prerequisites

  • Clone the Azure Arc Jumpstart repository

    git clone https://github.com/microsoft/azure_arc.git
    
  • Fork the “Hello Arc” demo application repository.

  • Install or update Azure CLI to version 2.15.0 and above. Use the below command to check your current installed version.

    az --version
    
  • As mentioned, this guide starts at the point where you already have a connected AKS cluster to Azure Arc.

    Existing AKS Azure Arc connected cluster

    Existing AKS Azure Arc connected cluster

  • Before installing the Azure Policy Add-on or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource provider and create a role assignment for the cluster service principal. To do that, open Azure Cloud Shell and run either the Azure CLI or Azure PowerShell command.

    Open Azure Cloud Shell

    Azure CLI:

    az provider register --namespace 'Microsoft.PolicyInsights'
    

    Azure PowerShell:

    Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
    

    To verify successful registration, run either the below Azure CLI or Azure PowerShell command.

    Azure CLI:

    az provider show --namespace 'Microsoft.PolicyInsights'
    

    Azure PowerShell:

    Get-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
    

    AzResourceProvider Bash

    AzResourceProvider PowerShell

  • Create Azure service principal (SP)

    Note: This guide assumes you will be working with a service principal assigned with the ‘Contributor’ role as described below. If you want to further limit the RBAC scope of your service Principal, you can assign it with the ‘Policy Insights Data Writer (Preview)’ role the Azure Arc enabled Kubernetes cluster as described here.

    To be able to complete the scenario and its related automation, Azure service principal assigned with the “Contributor” role is required. To create it, login to your Azure account run the below command (this can also be done in Azure Cloud Shell).

    az login
    az ad sp create-for-rbac -n "<Unique SP Name>" --role contributor
    

    For example:

    az ad sp create-for-rbac -n "http://AzureArcK8s" --role contributor
    

    Output should look like this:

    {
    "appId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "displayName": "AzureArcK8s",
    "name": "http://AzureArcK8s",
    "password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "tenant": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    }
    

    Note: The Jumpstart scenarios are designed with as much ease of use in-mind and adhering to security-related best practices whenever possible. It is optional but highly recommended to scope the service principal to a specific Azure subscription and resource group as well considering using a less privileged service principal account

Azure Policy for Azure Arc Connected Cluster Integration

  • In order to keep your local environment clean and untouched, we will use Azure Cloud Shell (located in the top-right corner in the Azure portal) to run the aks_policy_onboarding script against the AKS connected cluster. For your convenient, shell script is provided to you.

  • Edit the environment variables in the script to match your environment parameters, upload it to the Cloud Shell environment and run it using the . ./aks_policy_onboarding.sh command. If you decided to use the ‘Policy Insights Data Writer (Preview)’ role assignment as described in the perquisites section, make sure to use it’s respective appId, password and tenantId

    Note: The extra dot is due to the shell script has an export function and needs to have the vars exported in the same shell session as the rest of the commands.

    Upload onboarding script

    Upload onboarding script

    Upload onboarding script

The script will:

  • Login to your Azure subscription using the SPN credentials
  • Install NGINX Ingress Controller
  • Retrieve the Azure Arc Connected Cluster Azure Resource ID
  • Install the ‘azure-policy-addon’ helm chart & Gatekeeper

After few seconds, by running the the kubectl get pods -A command, you will notice all pods have been deployed.

Showing pods deployment

Deploy GitOps to Azure Arc Kubernetes cluster using Azure Policy

Although you can deploy GitOps configuration individually on each of your Azure Arc connected clusters, Azure Policy for Kubernetes allows to do the same on a broader scope (i.e subscription or resource group). That way, you can guarantee existing and newly added Azure Arc connected clusters to all have the same GitOps configuration and as a result, the same cluster baseline and/or application version deployed.

  • Before assigning the policy, in the Azure portal, click the Configuration setting in your AKS connected cluster. Notice how no GitOps configurations are deployed.

    Empty GitOps configurations

  • In the Search bar, look for Policy and click on Definitions which will show you all of the available Azure policies.

    Searching for Azure Policy definitions

    Searching for Azure Policy definitions

  • The “[Preview]: Deploy GitOps to Kubernetes cluster” policy is part of the Kubernetes policies. Once you filter to find these, click on the policy and the ‘Assign’ button.

    GitOps to Kubernetes cluster Azure policy

    GitOps to Kubernetes cluster Azure policy

  • In the below example, the scope of the policy represent the resource group where the AKS connected cluster Azure Arc resource is located. Alternatively, the scope could have been the entire Azure subscription or a resource group with many Azure Arc connected clusters. Also, make sure Policy enforcement is set to Enabled.

    For this GitOps configuration policy, we will be using the “Hello Arc” application repository which includes the Kubernetes Service for external access, Deployment as well as the ingress rule to be used by the NGINX ingress controller.

    Assigning Azure policy

    Assigning Azure policy

    Assigning Azure policy

    Assigning Azure policy

    Assigning Azure policy

  • Once the policy configuration deployed, after ~15-25min, the policy remediation task will start the evaluation against the Kubernetes cluster, recognize it as “Non-compliant” (since it’s still does note have the GitOps configuration deployed) and lastly, after the configuration has been deployed the policy will move to a “Compliant” state. To check this, go back to the main Policy page in the Azure portal.

    Note: The process of evaluation all the way to the point that the GitOps configuration is deployed against the cluster can take ~15-30min.

    Azure policy evaluation

    Azure policy evaluation

    Azure policy evaluation

    Azure policy evaluation

    Azure policy evaluation

Verify GitOps Configuration & App Deployment

  • Now that the policy is in compliant state, let’s first verify the GitOps configurations. In the Azure portal click the AKS connected Azure Arc cluster and open the Configurations settings.

    Successful GitOps config deployment

    Successful GitOps config deployment

    Successful GitOps config deployment

  • In order to verify the “Hello Arc” application and it’s component has been deployed, In the Azure Cloud Shell, run the below commands.

    kubectl get pods -n hello-arc
    kubectl get ing -n hello-arc
    kubectl get svc -n hello-arc
    

    You can see how the Flux GitOps operator, Memcached, the “Hello Arc” application and the ingress rule now deployed on the cluster as well the Service with an external IP.

    “Hello Arc” Kubernetes pods

  • Copy the Service external IP and paste in your browser.

    “Hello Arc” application deployed

Clean up environment

Complete the following steps to clean up your environment.