Use Azure Policy on an Azure Arc-enabled Kubernetes cluster to audit pods’ labels for use with Calico network policy

Disclaimer: This scenario was contributed by the Tigera Project Calico team as part of the “Jumpstart Friends” program

jumpstart-friends-logo

The following Jumpstart scenario will guide you how to use Azure Policy Azure Policy for Kubernetes on an Azure Arc-enabled Kubernetes cluster to audit/enforce labels for pods. Network policy is applied to pods using label selectors.

NOTE: This guide assumes you already deployed an Amazon Elastic Kubernetes Service (EKS) or Google Kubernetes Engine (GKE) cluster and connected it to Azure Arc. If you haven’t, this repository offers you a way to do so in an automated fashion using these Jumpstart scenarios:

Calico Network Policy uses labels to select pods in Kubernetes for applying ingress/egress rules.

In this scenario, we will be using Azure Policy on an Azure-Arc enabled Kubernetes cluster to check whether the “fw-zone” label is present on pods in the “storefront” namespace, and how these labels are used in network policy enforcement.

The Azure Policy will be set to “Audit” mode to check the configuration of existing clusters (it can also be set to “Deny” mode to avoid any missing labels)

Prerequisites

  • Clone the Azure Arc Jumpstart repository

    git clone https://github.com/microsoft/azure_arc.git

  • Install or update Azure CLI to version 2.49.0 and above. Use the following command to check your current installed version.

    az --version

  • As mentioned, this scenario starts at the point where you already have a connected EKS/GKE cluster to Azure Arc.

    Existing EKS Azure Arc connected cluster

    Existing GKE Azure Arc connected cluster

  • Before installing the Azure Policy Add-on or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource provider and create a role assignment for the cluster service principal. To do that, open Azure Cloud Shell and run either the Azure CLI or PowerShell command.

    Open Azure Cloud Shell

    Azure CLI:

    az provider register --namespace 'Microsoft.PolicyInsights'

    PowerShell:

    Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'

    To verify successful registration, run either the below Azure CLI or PowerShell command.

    Azure CLI:

    az provider show --namespace 'Microsoft.PolicyInsights'

    PowerShell:

    Get-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'

    AzResourceProvider Bash

  • Check whether Azure Policy is installed in an Azure Arc-enabled Kubernetes cluster

    by running the kubectl get pods -n gatekeeper-system command.

  • You should see the following pods running.

    Showing pods deployment

  • Deploy a demo application and Calico Network policy in your Azure Arc-enabled Kubernetes cluster.

    cd azure_arc/azure_arc_k8s_jumpstart/multi_distributions/calico_policy

    kubectl create -f storefront.yaml

  • Deploy the storefront demo application in your cluster by running the kubectl create -f storefront.yaml command.

    Showing storefront pods

    kubectl create -f calico-firewall-policy.yaml

  • Deploy a sample Calico network policy by running the kubectl create -f calico-firewall-policy.yaml command.

    Showing storefront network policy

Deploy an Azure policy to verify if “fw-zone” label exists for pods in the storefront namespace

  • In the Azure portal search bar, look for Policy and click on Definitions which will show you all of the available Azure policies.

    Searching for Azure Policy definitions

    Searching for Azure Policy definitions

  • Click on Category to search for built-in policies.

    Choose policy category

  • In the below example, make sure Category is set to Kubernetes only, and type label in Search, you will find Kubernetes cluster pods should use specified labels in BuiltIn Type.

    Find the pod label policy for Kubernetes cluster

    BuiltIn label policy for Kubernetes cluster

  • Click this policy, and assign it to your Azure resource group which includes the Azure Arc-enabled Kubernetes clusters. Alternatively, you can assign the policy to the entire Azure subscription.

    Assign Azure policy

  • Name the assignment, for example storefront namespace pods should use fw-zone labels for calico _networkpolicy_ enforcement.

    Assignment name

  • In the below example, we only audit one namespace which is storefront with fw-zone as the required label.

    Set Parameters

  • Once assigned, the policy task will start the evaluation against the Arc connected cluster in your resource group. You can check the status of this assignment under Compliance.

    Compliance list

    NOTE: The process of evaluation can take up 30min, in the meantime will see 0 out of 0 in Resource compliance.

  • After the evaluation, the policy will inform you of the result and provide details if there are any resources that are non-compliant.

    We will deploy a centos and nginx pod to the storefront namespace without the fw-zone label by running the commands below. Creation of these pods will be allowed as the policy effect is set to audit instead of deny.

    kubectl run centos --image=centos -n storefront --restart=Never -- /bin/sh -c "sleep 24h"

    kubectl run nginx --image=nginx -n storefront
kubectl -n storefront expose pod nginx --port=80

    Compliance result

    Audit details

  • Test Calico network policy enforcement by adding the fw-zone labels to the centos and nginx pods.

    The network policy is set to allow only trusted zone connections to the restricted zone, and will deny traffic from dmz zone. We will label the nginx pod with fw-zone=restricted, and test connectivity when changing the centos pod label from fw-zone=trusted to fw-zone=dmz.

  • Test Calico network policy enforcement without labels.

    test result

  • Test Calico network policy enforcement with proper labels.

    test result

  • Test Calico network policy enforcement with wrong labels.

    test result

Clean up environment

Complete the following steps to clean up your environment.