Open Policy Agent with Kubernetes – Tutorial (Pt. 1)

Introduction

As Kubernetes becomes the de-facto platform for organizing containerized workloads, more and more users are looking for ways to control and secure Kubernetes clusters.

We’ve already extensively explored the Kubernetes threat model, as well as NSA/CISA hardening guidelines, and delved deeper into a series of tutorials that I’ll put below in case you missed it:

Stiffness is a sure thing, but what about it enforce policies within a block? This is a completely different task and requires a different set of tools.

As you may have already guessed, the proper way to do this is select Policies as a symbolAnd a great tool for that is the Open Policy Agent, or OPA. If you don’t know what I’m talking about, please take the time to read this introduction first:

What is politics as a symbol? Introduction to open proxy policy

Learn about the benefits of policy as code and start testing your policies for cloud native environments.

Why not just use RBAC?

To better understand why we need to use a new policy tool, let’s take a concrete example: imagine you’re a group administrator, and you want to restrict what can run in your group.

At first, it looks like a perfectly valid use case for Role-Based Access Control (RBAC, a permission system for creating and managing Kubernetes objects at the resource level): with RBAC you can easily delegate states like “User X can do Y in namespace Z”.

You start by selecting a role in default Namespace to grant read access to pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Now, it’s more complicated: How would you do if you wanted to restrict, not access to resources like pods, but How is it configured?

This is where RBAC’s powers end: access controls cannot override and limit the configurations, settings, and contents of Kubernetes objects. However, as a cluster maintainer, you might just like it!

Why don’t you write my admission console?

For the sake of the exercise, we’ll imagine that you need to order all resource objects in your collection It has a specific designation.

As you can see, this is not related to a role or group, in fact, it is related to a very specific domain for all the resources in your group.

Let’s say you need to control pod fields (or any fields on other resource types for that matter). In this case, you have one option: create your own admission console. The approval console is a piece of code that intercepts requests to the Kubernetes API server before the object is persisted.

In detail, it would work as follows: a request to create a new pod is made to the cluster API service; This will trigger a custom ValidatingAdmissionWebhook that matches that request; The console will call a webhook to check; If the controller rejects the request, the API service will also reject it.

The problem with this is that no scale: You will have to write as many custom entry controllers as rules or policies you want to enforce!

This is where OPA comes in, and together we’ll see how to set it up.

Open the Kubernetes policy agent

To solve the above challenge, what we really need here is a system that supports multiple configurations that cover different types and fields of resources and allows for reuse. The Open Policy Agent (OPA) provides exactly that.

In short, the OPA policy engine evaluates requests to determine if they comply with the configured policies.

OPA can easily integrate with Kubernetes: it expects JSON injection, is easy to place in containers, and supports dynamic configuration, which makes it well suited for delivering policy evaluation to the Kubernetes API service.

So let’s dive in and show how to deploy and integrate OPA with Kubernetes.

Tutorial: How to use OPA with Kubernetes

In this example, we will demonstrate how OPA integrates with Kubernetes by deploying a policy that ensures that the login hostname must be in the allowlist in the namespace containing Ingress.

This means that we want to reject all creations of Ingress objects whose hostname does not match allowlist.

Before you begin, download the OPA if you haven’t already.

Prepare a Kubernetes cluster

Let’s Begin minikubeMaybe minikube ingress addon , create a namespace (for OPA deployment), and configure the Kubernetes context:

minikube start
minikube addons enable ingress
kubectl create namespace opa
kubectl config set-context opa-tutorial --user minikube --cluster minikube --namespace opa
kubectl config use-context opa-tutorial

Communication between Kubernetes and OPA It must be secured using TLS. Let’s use openssl To do that:

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -sha256 -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"

# generate the TLS key and certificate for OPA:
cat >server.conf <<EOF
[ req ]
prompt = no
req_extensions = v3_ext
distinguished_name = dn

[ dn ]
CN = opa.opa.svc

[ v3_ext ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = DNS:opa.opa.svc,DNS:opa.opa.svc.cluster,DNS:opa.opa.svc.cluster.local
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -sha256 -out server.csr -extensions v3_ext -config server.conf
openssl x509 -req -in server.csr -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_ext -extfile server.conf

# create a secret to store the TLS credentials for OPA
kubectl create secret tls opa-server --cert=server.crt --key=server.key --namespace opa

Define policy with Rego

Now that our Kubernetes cluster is ready, let’s write some policies. First, create a new folder to store them:

mkdir policies && cd policies

We want a policy that restricts which hostnames the login can use. Only hostnames that match the given regular expressions will be allowed. Create a file ingress-allowlist.rego With the following content:

package kubernetes.admission

import data.kubernetes.namespaces

operations := {"CREATE", "UPDATE"}

deny[msg] {
  input.request.kind.kind == "Ingress"
  operations[input.request.operation]
  host := input.request.object.spec.rules[_].host
  not fqdn_matches_any(host, valid_ingress_hosts)
  msg := sprintf("invalid ingress host %q", [host])
}

valid_ingress_hosts := {host |
  allowlist := namespaces[input.request.namespace].metadata.annotations["ingress-allowlist"]
  hosts := split(allowlist, ",")
  host := hosts[_]
}

fqdn_matches_any(str, patterns) {
  fqdn_matches(str, patterns[_])
}

fqdn_matches(str, pattern) {
  pattern_parts := split(pattern, ".")
  pattern_parts[0] == "*"
  suffix := trim(pattern, "*.")
  endswith(str, suffix)
}

fqdn_matches(str, pattern) {
    not contains(pattern, "*")
    str == pattern
}

If you don’t know much about Rego’s policy language yet, read the official documentation here.

Basically, this piece of code does the following:

• the valid_ingress_hosts A function that gets the ‘ingress-allowlist’ annotation in the metadata section of the namespace.
• Then it tries to match the hostname to it using the regular expression.
• If it does not match, the request will be rejected with a message.

Next, we will define the master policy that will import the above hostname restriction policy and respond to the global policy decision.

Note that in this example, since we are only using one policy, this master policy is redundant. However, in real-life situations where you want to enforce multiple policies, the master policy is necessary to make an overall decision in the end.

Create a file main.rego With the following content:

package system

import data.kubernetes.admission

main := {
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": response,
}

default uid := ""

uid := input.request.uid

response := {
    "allowed": false,
    "uid": uid,
    "status": {
        "message": reason,
    },
} {
    reason = concat(", ", admission.deny)
    reason != ""
}

else := {"allowed": true, "uid": uid}

Build and deploy an OPA package

With your Rego policy code ready, now it’s time to build it. Run the following commands in the policies folder for creation and deployment:

cat > .manifest <<EOF 
{ 
"roots": ["kubernetes/admission", "system"] 
}
EOF 
opa build -b . 

# serve the OPA bundle using Nginx:
docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest

Here we will build a “bundle” from Rego’s code, and then serve the bundle in the Nginx server in a docker container locally, which will be merged with OPA in the next step. Read on.

Install OPA as an access permission controller

First, let’s post the OPA:

kubectl apply -f https://gist.githubusercontent.com/IronCore864/035f7feca2c89ffd2809ec604fb3b873/raw/3e85364f72970b82f418544fd009fd478bc655ae/admission-controller.yaml

Let’s take a look at part of this admission-controller.yaml a file:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
# ...
        - name: opa
          image: openpolicyagent/opa:0.47.3-rootless
# ...
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=networking.k8s.io/v1/ingresses"

There is a container named kube-mgmt which acts as a side vehicle for the OPA container. kube-mgmt Manages OPA instance policies/data in Kubernetes by loading the Kubernetes Namespace and entry objects (see ‘args’ in the YAML file) into OPA on OPA startup. The side tool creates clocks on the Kubernetes API server so that OPA can have access to and inject Kubernetes namespaces (which is what our policy is concerned with in this tutorial).

Next, let’s sort kube-system and the opa So that OPA has no control over the resources in those namespaces:

kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore

Finally, we register the OPA as an access console by creating a ValidatingWebhookConfiguration:

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d 'n')
      service:
        namespace: opa
        name: opa
    admissionReviewVersions: ["v1"]
    sideEffects: None
EOF

kubectl apply -f webhook-configuration.yaml

Test our policy

Now that everything is posted, let’s see if our policy works. Therefore, we need an experimental namespace.

Create a file: qa-namespace.yaml With the following content:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ingress-allowlist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
  name: qa

We can see that metadata.annotations.ingress-allowlist The label corresponds to the code we wrote earlier. Basically, here we only allow Ingress objects to be created if their hostnames match the pattern “.qa.acmecorp.com” or “.internal.acmecorp.com.”“.

Let’s create this namespace:

kubectl create -f qa-namespace.yaml

Next, we prepare a good and bad test case for it. create a file ingress-ok.yaml With the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-ok
spec:
  rules:
  - host: signin.qa.acmecorp.com
    http:
      paths:
      - pathType: ImplementationSpecific
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80

The hostname matches the regular expression from the namespace annotation.

Create the file ingress-bad.yaml With the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-bad
spec:
  rules:
  - host: acmecorp.com
    http:
      paths:
      - pathType: ImplementationSpecific
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80

This hostname does not match. Now if we run the first test:

kubectl create -f ingress-ok.yaml -n qa

We can see that the conference has been created successfully. If we run the second:

kubectl create -f ingress-bad.yaml -n qa 

We will get the following error:

Error from server: error when creating "ingress-bad.yaml": admission webhook "validating-webhook.openpolicyagent.org" denied the request: invalid ingress host "acmecorp.com"

congratulations! This means that our policy is working correctly now 😃

sweep up

Run the following commands to destroy everything:

minikube delete
docker stop bundle-server

summary

Well, in this tutorial, we have shown how to create OPA policies, how to create and publish them as a package served by Nginx, and register them in OPA. It should be noted that we have installed OPA with kube-mgmt as a sidecar.

In the second part of this tutorial, we’ll look at a more practical example policy using an OPA gatekeeper, which can limit the pollution tolerances that pods can use. We will also show the importance of policy tests and try to answer the question: How is policy measured as code?

Thank you for making it this far, and see you on the next tutorial!

*** This is a blog aggregator for the Security Blogger Network from the GitGuardian Blog – Auto-Disclosure of Secrets written by a guest expert. Read the original post at: https://blog.gitguardian.com/open-policy-agent-with-kubernetes-tutorial-pt-1/