External Data Sources

The Variables section discusses how variables can help create smarter and reusable policy definitions and introduced the concept of a rule context that stores all variables.

This section provides details on using ConfigMaps and API Calls to reference external data as variables in policies.

Variables from ConfigMaps

A ConfigMap resource in Kubernetes is commonly used as a source of configuration details which can be consumed by applications. This data can be written in multiple formats, stored in a Namespace, and accessed easily. Kyverno supports using a ConfigMap as a data source for variables. When a policy referencing a ConfigMap resource is evaluated, the ConfigMap data is checked at that time ensuring that references to the ConfigMap are always dynamic. Should the ConfigMap be updated, subsequent policy lookups will pick up the latest data at that point.

In order to consume data from a ConfigMap in a rule, a context is required. For each rule you wish to consume data from a ConfigMap, you must define a context. The context data can then be referenced in the policy rule using JMESPath notation.

Looking up ConfigMap values

A ConfigMap that is defined in a rule’s context can be referred to using its unique name within the context. ConfigMap values can be referenced using a JMESPath style expression.

{{ <context-name>.data.<key-name> }}

Consider a simple ConfigMap definition like so.

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: some-config-map
5  namespace: some-namespace
6data:
7  env: production

To refer to values from a ConfigMap inside a rule, define a context inside the rule with one or more ConfigMap declarations. Using the sample ConfigMap snippet referenced above, the below rule defines a context which references this specific ConfigMap by name.

 1rules:
 2  - name: example-lookup
 3    # Define a context for the rule
 4    context:
 5    # A unique name for the ConfigMap
 6    - name: dictionary
 7      configMap:
 8        # Name of the ConfigMap which will be looked up
 9        name: some-config-map
10        # Namespace in which this ConfigMap is stored
11        namespace: some-namespace 

Based on the example above, we can now refer to a ConfigMap value using {{dictionary.data.env}}. The variable will be substituted with the value production during policy execution.

Put into context of a full ClusterPolicy, referencing a ConfigMap as a variable looks like the following.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: cm-variable-example
 5  annotations:
 6    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,StatefulSet
 7spec:
 8    rules:
 9    - name: example-configmap-lookup
10      context:
11      - name: dictionary
12        configMap:
13          name: some-config-map
14          namespace: some-namespace
15      match:
16        resources:
17          kinds:
18          - Pod
19      mutate:
20        patchStrategicMerge:
21          metadata:
22            labels:
23              my-environment-name: "{{dictionary.data.env}}"

In the above ClusterPolicy, a mutate rule matches all incoming Pod resources and adds a label to them with the name of my-environment-name. Because we have defined a context which points to our earlier ConfigMap named mycmap, we can reference the value with the expression {{dictionary.data.env}}. A new Pod will then receive the label my-environment-name=production.

{{ "<name>".data."<key>" }}

Handling ConfigMap Array Values

In addition to simple string values, Kyverno has the ability to consume array values from a ConfigMap. You have the choice of storing those array values in either YAML format within a block scalar or JSON-encoded content.

For example, let’s say you wanted to define a list of allowed roles in a ConfigMap. A Kyverno policy can refer to this list to deny a request where the role, defined as an annotation, does not match one of the values in the list.

Consider a ConfigMap with the following content written as a YAML multi-line value.

 1apiVersion: v1
 2kind: ConfigMap
 3metadata:
 4  name: roles-dictionary
 5  namespace: default
 6data:
 7  allowed-roles: |-
 8    cluster-admin
 9    cluster-operator
10    tenant-admin    

Or this as a JSON-encoded value.

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: roles-dictionary
5  namespace: default
6data:
7  allowed-roles: "[\"cluster-admin\", \"cluster-operator\", \"tenant-admin\"]"

Now that the array data is saved in the allowed-roles key, here is a sample ClusterPolicy containing a single rule that blocks a Deployment if the value of the annotation named role is not in the allowed list:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: cm-array-example
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  rules:
 9  - name: validate-role-annotation
10    context:
11      - name: roles-dictionary
12        configMap:
13          name: roles-dictionary
14          namespace: default
15    match:
16      resources:
17        kinds:
18        - Deployment
19    validate:
20      message: "The role {{ request.object.metadata.annotations.role }} is not in the allowed list of roles: {{ \"roles-dictionary\".data.\"allowed-roles\" }}."
21      deny:
22        conditions:
23        - key: "{{ request.object.metadata.annotations.role }}"
24          operator: NotIn
25          value:  "{{ \"roles-dictionary\".data.\"allowed-roles\" }}"

This rule denies the request for a new Deployment if the annotation role is not found in the array we defined in the earlier ConfigMap named roles-dictionary.

Once creating this sample ClusterPolicy, attempt to create a new Deployment where the annotation role=super-user and test the result.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: busybox
 5  annotations:
 6    role: super-user
 7  labels:
 8    app: busybox
 9spec:
10  replicas: 1
11  selector:
12    matchLabels:
13      app: busybox
14  template:
15    metadata:
16      labels:
17        app: busybox
18    spec:
19      containers:
20      - image: busybox:1.28
21        name: busybox
22        command: ["sleep", "9999"]

Submit the manifest and see how Kyverno reacts.

1kubectl create -f deploy.yaml

Error from server: error when creating "deploy.yaml": admission webhook "validate.kyverno.svc" denied the request:

resource Deployment/default/busybox was blocked due to the following policies

cm-array-example:
  validate-role-annotation: 'The role super-user is not in the allowed list of roles: ["cluster-admin", "cluster-operator", "tenant-admin"].'

Changing the role annotation to one of the values present in the ConfigMap, for example tenant-admin, allows the Deployment resource to be created.

Variables from Kubernetes API Server Calls

Kubernetes is powered by a declarative API that allows querying and manipulating resources. Kyverno policies can use the Kubernetes API to fetch a resource, or even collections of resource types, for use in a policy. Additionally, Kyverno allows applying JMESPath (JSON Match Expression) to the resource data to extract and transform values into a format that is easy to use within a policy.

A Kyverno Kubernetes API call works just as with kubectl and other API clients, and can be tested using existing tools.

For example, here is a command line that uses kubectl to fetch the list of Pods in a Namespace and then pipes the output to jp which counts the number of pods:

1kubectl get --raw /api/v1/namespaces/kyverno/pods | jp "items | length(@)"

The corresponding API call in Kyverno is defined as below. It uses a variable {{request.namespace}} to use the Namespace of the object being operated on, and then applies the same JMESPath to store the count of Pods in the Namespace in the context as the variable podCount. This new variable can then be used in the policy rule.

1rules:
2- name: example-api-call
3  context:
4  - name: podCount
5    apiCall:
6      urlPath: "/api/v1/namespaces/{{request.namespace}}/pods"
7      jmesPath: "items | length(@)"   

URL Paths

The Kubernetes API organizes resources under groups and versions. For example, the resource type Deployment is available in the API Group apps with a version v1.

The HTTP URL paths of the API calls are based on the group, version, and resource type as follows:

• /apis/{GROUP}/{VERSION}/{RESOURCETYPE}: get a collection of resources
• /apis/{GROUP}/{VERSION}/{RESOURCETYPE}/{NAME}: get a resource

For namespaced resources, to get a specific resource by name or to get all resources in a Namespace, the Namespace name must also be provided as follows:

• /apis/{GROUP}/{VERSION}/namespaces/{NAMESPACE}/{RESOURCETYPE}: get a collection of resources in the namespace
• /apis/{GROUP}/{VERSION}/namespaces/{NAMESPACE}/{RESOURCETYPE}/{NAME}: get a resource in a namespace

For historic resources, the Kubernetes Core API is available under /api/v1. For example, to query all Namespace resources the path /api/v1/namespaces is used.

The Kubernetes API groups are defined in the API reference documentation and can also be retrieved via the kubectl api-resources command shown below:

 1$ kubectl api-resources
 2NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
 3bindings                                                                      true         Binding
 4componentstatuses                 cs                                          false        ComponentStatus
 5configmaps                        cm                                          true         ConfigMap
 6endpoints                         ep                                          true         Endpoints
 7events                            ev                                          true         Event
 8limitranges                       limits                                      true         LimitRange
 9namespaces                        ns                                          false        Namespace
10nodes                             no                                          false        Node
11persistentvolumeclaims            pvc                                         true         PersistentVolumeClaim
12
13...

The kubectl api-versions command prints out the available versions for each API group. Here is a sample:

 1$ kubectl api-versions
 2admissionregistration.k8s.io/v1
 3admissionregistration.k8s.io/v1beta1
 4apiextensions.k8s.io/v1
 5apiextensions.k8s.io/v1beta1
 6apiregistration.k8s.io/v1
 7apiregistration.k8s.io/v1beta1
 8apps/v1
 9authentication.k8s.io/v1
10authentication.k8s.io/v1beta1
11authorization.k8s.io/v1
12authorization.k8s.io/v1beta1
13autoscaling/v1
14autoscaling/v2beta1
15autoscaling/v2beta2
16batch/v1
17...

You can use these commands together to find the URL path for resources, as shown below:

1kubectl api-resources | grep deploy

1kubectl api-versions | grep apps

Handling collections

The API server response for a HTTP GET on a URL path that requests collections of resources will be an object with a list of items (resources).

Here is an example that fetches all Namespace resources:

1kubectl get --raw /api/v1/namespaces | jq

This will return a NamespaceList object with a property items that contains the list of Namespaces:

 1{
 2    "kind": "NamespaceList",
 3    "apiVersion": "v1",
 4    "metadata": {
 5      "selfLink": "/api/v1/namespaces",
 6      "resourceVersion": "2009258"
 7    },
 8    "items": [
 9      {
10        "metadata": {
11          "name": "default",
12          "selfLink": "/api/v1/namespaces/default",
13          "uid": "5011b5d5-abb7-4fef-93f9-8b5fa4b2eba9",
14          "resourceVersion": "155",
15          "creationTimestamp": "2021-01-19T20:20:37Z",
16          "managedFields": [
17            {
18              "manager": "kube-apiserver",
19              "operation": "Update",
20              "apiVersion": "v1",
21              "time": "2021-01-19T20:20:37Z",
22              "fieldsType": "FieldsV1",
23              "fieldsV1": {
24                "f:status": {
25                  "f:phase": {}
26                }
27              }
28            }
29          ]
30        },
31        "spec": {
32          "finalizers": [
33            "kubernetes"
34          ]
35        },
36        "status": {
37          "phase": "Active"
38        }
39      },
40
41      ...

To process this data in JMESPath, reference the items. Here is an example which extracts a few metadata fields across all Namespace resources:

1kubectl get --raw /api/v1/namespaces | jp "items[*].{name: metadata.name, creationTime: metadata.creationTimestamp}"

This produces a new JSON list of objects with properties name and creationTime.

 1[
 2  {
 3    "creationTimestamp": "2021-01-19T20:20:37Z",
 4    "name": "default"
 5  },
 6  {
 7    "creationTimestamp": "2021-01-19T20:20:36Z",
 8    "name": "kube-node-lease"
 9  },
10
11  ...

To find an item in the list you can use JMESPath filters. For example, this command will match a Namespace by its name:

1 kubectl get --raw /api/v1/namespaces | jp "items[?metadata.name == 'default'].{uid: metadata.uid, creationTimestamp: metadata.creationTimestamp}"

In addition to wildcards and filters, JMESPath has many additional powerful features including several useful functions. Be sure to go through the JMESPath tutorial and try the interactive examples.

Sample Policy: Limit Services of type LoadBalancer in a Namespace

Here is a complete sample policy that limits each namespace to a single service of type LoadBalancer.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: limits
 5spec:
 6  validationFailureAction: enforce
 7  rules:
 8  - name: limit-lb-svc
 9    match:
10      resources:
11        kinds:
12        - Service
13    context:
14    - name: serviceCount
15      apiCall:
16        urlPath: "/api/v1/namespaces/{{ request.object.metadata.namespace }}/services"
17        jmesPath: "items[?spec.type == 'LoadBalancer'] | length(@)"    
18    preconditions:
19    - key: "{{ request.operation }}"
20      operator: Equals
21      value: "CREATE"
22    validate:
23      message: "Only one LoadBalancer service is allowed per namespace"
24      deny:
25        conditions:
26        - key: "{{ serviceCount }}"
27          operator: GreaterThanOrEquals
28          value: 1

This sample policy retrieves the list of Services in the Namespace and stores the count of type LoadBalancer in a variable called serviceCount. A deny rule is used to ensure that the count cannot exceed one.