Apply GitOps configurations on GKE as an Azure Arc Connected Cluster using Azure Policy for Kubernetes

The following README will guide you on how to enable Azure Policy for Kubernetes on a Google Kubernetes Engine (GKE) cluster that is projected as an Azure Arc connected cluster as well as how to create GitOps policy to apply on the cluster.

Note: This guide assumes you already deployed a GKE cluster and connected it to Azure Arc. If you haven’t, this repository offers you a way to do so in an automated fashion using Terraform.

Prerequisites

  • Clone the Azure Arc Jumpstart repository

    git clone https://github.com/microsoft/azure_arc.git
    
  • Install or update Azure CLI to version 2.15.0 and above. Use the below command to check your current installed version.

    az --version
    
  • As mentioned, this guide starts at the point where you already have a connected GKE cluster to Azure Arc.

    Existing GKE Azure Arc-enabled Kubernetes cluster

    Existing GKE Azure Arc-enabled Kubernetes cluster

  • Before installing the Azure Policy addon or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource provider and create a role assignment for the cluster service principal. To do that, open Azure Cloud Shell and run either the Azure CLI or Azure PowerShell command.

    Azure Cloud Shell

    Azure CLI:

    az provider register --namespace 'Microsoft.PolicyInsights'
    

    PowerShell:

    Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
    
  • To verify successful registration, run either the below Azure CLI or Azure PowerShell command.

    Azure CLI:

    az provider show --namespace 'Microsoft.PolicyInsights'
    

    PowerShell:

    Get-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
    

    Installing the Azure Policy addon resource provider

    Installing the Azure Policy addon resource provider

  • Create Azure service principal (SP)

    Note: This guide assumes you will be working with a service principal assigned with the ‘Contributor’ role as described below. If you want to further limit the RBAC scope of your service Principal, you can assign it with the ‘Policy Insights Data Writer (Preview)’ role the Azure Arc-enabled Kubernetes cluster as described here.

  • To be able to complete the scenario and its related automation, Azure service principal assigned with the “Contributor” role is required. To create it, login to your Azure account run the below command (this can also be done in Azure Cloud Shell).

    az login
    az ad sp create-for-rbac -n "<Unique SP Name>" --role contributor
    

    For example:

    az ad sp create-for-rbac -n "http://AzureArcK8s" --role contributor
    

    Output should look like this:

    {
    "appId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "displayName": "AzureArcK8s",
    "name": "http://AzureArcK8s",
    "password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "tenant": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    }
    

    Note: It is optional but highly recommended to scope the SP to a specific Azure subscription and resource group

Azure Policy for Azure Arc Connected Cluster Integration

  • In order to keep your local environment clean and untouched, we will use Google Cloud Shell to run the gke_policy_onboarding shell script against the GKE connected cluster.

  • Edit the environment variables in the script to match your environment parameters, upload it to the Cloud Shell environment and run it using the . ./gke_policy_onboarding.sh command. If you decided to use the ‘Policy Insights Data Writer (Preview)’ role assignment as described in the perquisites section, make sure to use it’s respective appId, password and tenantId.

    Note: The extra dot is due to the shell script has an export function and needs to have the vars exported in the same shell session as the rest of the commands.

    Run GCP Cloud Shell

    Run GCP Cloud Shell

    Upload a file to GCP Cloud Shell

    Upload a file to GCP Cloud Shell

    Upload a file to GCP Cloud Shell

    The script will:

    • Install Helm 3 & Azure CLI

    • Install NGINX Ingress Controller

    • Login to your Azure subscription using the SPN credentials

    • Retrieve cluster Azure Resource ID

    • Install the ‘azure-policy-addon’ helm chart & Gatekeeper

      After few seconds, by running the the kubectl get pods -A command, you will notice all pods have been deployed.

      kubectl get pods

Deploy GitOps to Azure Arc Kubernetes cluster using Azure Policy

Although you can deploy GitOps configuration individually on each of your Azure Arc connected clusters, Azure Policy for Kubernetes allows to do the same on a broader scope (i.e Subscription or resource group). That way, you can guarantee existing and newly added Azure Arc connected clusters to all have the same GitOps configuration and as a result, the same cluster baseline and/or application version deployed.

  • Before assigning the policy, in the Azure portal, click the Configuration setting in your GKE connected cluster. Notice how no GitOps configurations are deployed.

    No GitOps configurations

  • In the Search bar, look for Policy and click on Definitions which will show you all of the available Azure policies.

    Search for Azure Policy Definitions in the Azure portal

    Search for Azure Policy Definitions in the Azure portal

  • The “[Preview]: Deploy GitOps to Kubernetes cluster” policy is part of the Kubernetes policies. Once you filter to find these, click on the policy and the ‘Assign’ button.

    Azure Policy in the Azure portal

    Azure Policy definitions in the Azure portal

  • In the below example, the scope of the policy represent the resource group where the GKE connected cluster Azure Arc resource is located. Alternatively, the scope could have been the entire Azure subscription or a resource group with many Azure Arc connected clusters. Also, make sure Policy enforcement is set to Enabled.

    For this GitOps configuration policy, we will be using the “Hello Arc” application repository which includes the Kubernetes Service for external access, Deployment as well as the ingress rule to be used by the NGINX ingress controller.

    Assign the “[Preview]: Deploy GitOps to Kubernetes cluster” Azure Policy

    Assign the “[Preview]: Deploy GitOps to Kubernetes cluster” Azure Policy

    Assign the “[Preview]: Deploy GitOps to Kubernetes cluster” Azure Policy

    Assign the “[Preview]: Deploy GitOps to Kubernetes cluster” Azure Policy

  • Once the policy configuration deployed, after ~10-20min, the policy remediation task will start the evaluation against the Kubernetes cluster, recognize it as “Non-compliant” (since it’s still does note have the GitOps configuration deployed) and lastly, after the configuration has been deployed the policy will move to a “Compliant” state. To check this, go back to the main Policy page in the Azure portal.

    Note: The process of evaluation all the way to the point that the GitOps configuration is deployed against the cluster can take ~15-30min.

    Verifying Azure Policy status and compliance state

    Verifying Azure Policy status and compliance state

    Verifying Azure Policy status and compliance state

    Verifying Azure Policy status and compliance state

    Verifying Azure Policy status and compliance state

Verify GitOps Configuration & App Deployment

  • Now that the policy is in compliant state, let’s first verify the GitOps configurations. In the Azure portal click the GKE connected Azure Arc cluster and open the Configurations settings.

    Newly created GitOps configurations

    Newly created GitOps configurations

    Newly created GitOps configurations

  • In order to verify the “Hello Arc” application and it’s component has been deployed, In the Google Cloud Shell, run the below commands.

    kubectl get pods -n hello-arc
    kubectl get ing -n hello-arc
    kubectl get svc -n hello-arc
    

    You can see how the Flux GitOps operator, Memcached, the “Hello Arc” application and the ingress rule now deployed on the cluster as well the Kubernetes service with an external IP.

    Running kubectl commands to verifying successful deployment

  • Copy the Service external IP and paste in your browser to see the deployed “Hello Arc” application.

    Deployed “Hello Arc” application

Clean up environment

Complete the following steps to clean up your environment.

  • Delete the GKE cluster as described in the teardown instructions.

  • From the Policy page in the Azure portal, remove the “[Preview]: Deploy GitOps to Kubernetes cluster” policy assignment from the cluster.

    Delete Azure Policy assignment

    Delete Azure Policy assignment