Enforcing Kubernetes Best Practices Using Kyverno: A Beginner’s Guide

Kubernetes has been helping cloud-native eco-system since a long time by allowing users to deploy their distributed workload. It is a best container orchestration platform with a lot of features, however, it comes up with its own complexities such as managing it with multiple teams is not a easy task specifically ensuring that people do right thing and do not do such thing which leads to an security and vulnerability threat.

Additionally, Did you know that 70% of Kubernetes users report security misconfiguration as major concern when creating resources? In this article, we’ll explore how Kyverno, a powerful Kubernetes policy engine, can help you to automate security and compliance check effortlessly.

Kyverno is the right choice for these problems. It is open-source policy engine designed specifically for Kubernetes from Nirmata, which was later donated to Cloud Native Computing Foundation (CNCF). Kyverno is used to validate, mutate, generate and cleanup any Kubernetes resources by applying its policies as custom resource. Now a days organizations adopting it to meet their compliance and regulatory standards.

How Kyverno works?

When a request is sent to the Kubernetes API Server by Kubectl or any API client to create a resource in cluster, Kyverno uses its admission webhook controller to validate the request against the defined policies. Kyverno intercepts the request, checks if it matches the configured policies, and then decides whether to allow or reject the resource creation. If the request meets all policy requirements, Kyverno approves and creates the resource; otherwise, it denies the request, ensuring that only compliant resources are deployed.

Reference: Kyverno Documentation

Kyverno policies can match resources using various criteria, including resource kind, name, label selectors, annotations, and namespaces. This flexibility allows users to define granular policies that target specific resources or groups of resources, ensuring precise enforcement of governance and compliance requirements across the Kubernetes cluster.

Why we use Kyverno?

It provides features such as:

• Restricting users from deploying pods with root user access.
• Ensuring that all pods have CPU and memory requests and limits set.
• Enforcing image registry policies to only allow images from approved repositories.
• Automatically mutating resource configurations, such as adding default labels or annotations.
• Validating that certain resources (e.g., ConfigMaps or Secrets) meet security standards, like requiring encryption.
• Applying network policies to control traffic between pods and namespaces.
• Managing ingress and egress rules to maintain security and compliance so on.

Example: Enforcing Pod Security Standards

I’m going to explain one example from the list above, where I’ll guide you through setting up Kyverno and writing a policy to prevent users from deploying pods with root user access. I also recommend trying out the other use cases mentioned above

Prerequisites

• Kubernetes Cluster: A running Kubernetes cluster where Kyverno can be installed.
• Kyverno Installation: You need to install Kyverno on your cluster. You can do this using Helm or a direct YAML file from Kyverno’s GitHub repository.

Kyverno Installation

I am installing using simple YAML menifest files. If you want to install using Helm — Use following link: https://kyverno.io/docs/installation/methods/#install-kyverno-using-helm

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.11.1/install.yaml

Kyverno Policy: Restrict Root Privileges

This policy ensures that no Pod is allowed to run with root privileges:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-root-user
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: restrict-root-user
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Running as root user is not allowed."
      pattern:
        spec:
          containers:
          - securityContext:
              runAsNonRoot: true

How the Policy Works

1. kind: ClusterPolicy: This defines a policy that applies cluster-wide.
2. validationFailureAction: enforce: This ensures that if the policy is violated, the resource is rejected.
3. match: This specifies that the policy applies to Pods.
4. validate: This section defines the rule Kyverno should enforce. In this case, it checks if the Pod’s containers have a securityContext set with runAsNonRoot: true.

Deploying Policy:

kubectl apply -f restrict-root-user.yaml

Testing the Policy: Let’s deploying a Pod that violates this policy:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: nginx
    image: nginx

Execute the following command to deploying Pod in your kubernetes cluster.

kubectl apply -f test-pod.yaml

As soon you deploy it. You should see an error message because the Pod does not comply with the runAsNonRoot policy. Y

Error from server: error when creating "test-pod.yaml": admission webhook "validate.kyverno.svc-fn.k8s.io" denied the request: Running as root user is not allowed.

Bonus: Pod Manifest Passing the Kyverno Policy

Below is a YAML configuration for a Pod that adheres to the Kyverno policy requiring containers to run as non-root:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000 # A non-root user ID

Explanation

1. securityContext: This is the key section that enforces security settings for the container.

• runAsNonRoot: true: This ensures that the container must run as a non-root user.
• runAsUser: 1000: This specifies a user ID (UID) that is not 0 (the root user). You can set it to any UID that your container supports, as long as it is not 0.

Steps to Deploy a Pod with the Correct Configuration

1. Save the above YAML configuration to a file, for example, secure-pod.yaml.
2. Apply the configuration using kubectl:

kubectl apply -f secure-pod.yaml

3. Verify that the Pod is running:

kubectl get pods

If the policy and configuration are correct, the Pod should be created successfully without any errors related to the Kyverno policy. This setup ensures that your Pod runs securely without root privileges, in line with the enforced policy.

Conclusion

As shown in the article how policy is stopping users to create pod with root user access. By integrating Kyverno, organizations can automate policy enforcement, reduce human error, and ensure that best practices are consistently applied across their Kubernetes environments.

I encourage you to explore other use cases from the list above (in the ‘How Kyverno Works’ section) and implement Kyverno policies tailored to meet your organization’s standards. Experiment with these policies in your workspace and share your experiences by leaving a comment below — I’d love to hear your feedback! If you run into any challenges along the way, I’m here to help. Let’s connect on social media and build a stronger community together! Thanks for reading this article.

1. Topmate: https://topmate.io/umeshtyagi829
2. Linkedin: https://www.linkedin.com/in/umeshtyagi829/