4. Custom Resource Definitions (CRDs) and Operators

Tip

This tutorial requires access to a public container registry, which your trainer will provide.

Trainer Instructions

Note: We’ll use a simple container registry hosted in Azure. Trainer tip: Ensure anonymous (public) pull access is enabled using:

az acr update --name <your-acr-name> --anonymous-pull-enabled true

1. Custom Resource Definitions (CRDs)

Custom resources extend the Kubernetes API to support application-specific objects. These resources aren’t available by default and must be defined explicitly.

Follow this example from the Kubernetes docs to create a CRD for participants.

Your custom resource should define the following schema:

fullName: string
email: string
registered: boolean

It should be namespaced in scope, with these naming attributes:

api-group: stable.workshop.io
scope: Namespaced
names:
  plural: participants
  singular: participant
  kind: Participant
  shortNames:
    - part

Tip

You can choose any api group you like. The solutions are based on stable.workshop.io

Once defined, apply your CRD to the cluster.

Solution

The CRD should look like this one:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: participants.stable.workshop.io
spec:
  group: stable.workshop.io
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                fullName:
                  type: string
                email:
                  type: string
                registered:
                  type: boolean
  scope: Namespaced
  names:
    plural: participants
    singular: participant
    kind: Participant
    shortNames:
      - part

A solution is provided in the exercise folder:

kubectl apply -f ~/exercise/kubernetes/CRD-Operator/participant-crd.yaml

Check that your CRD has been successfully created:

kubectl get crd

Let’s now create an actual Participant object named Alice in the classroom namespace.

apiVersion: stable.workshop.io/v1
kind: Participant
metadata:
  name: alice-smith
  namespace: classroom
spec:
  fullName: Alice Smith
  email: alice@example.com
  registered: true

Solution

kubectl apply -f ~/exercise/kubernetes/CRD-Operator/participant.yaml

Verify that your custom resource was created by using kubectl:

Solution

kubectl get participants -n classroom
#OR
kubectl describe participant alice-smith -n classroom

2. Operators

Operators are powerful Kubernetes extensions that automate the management of complex workloads. They act as controllers for custom resources and perform operational tasks in response to changes.

In this section, we’ll write a basic operator that logs when a new participant is created, deleted or modified. It’s intentionally simple—just enough to show how operators work without getting lost in complexity.

We’ll use the Shell-Operator, a tool that lets you write operators using shell scripts. This is ideal for educational purposes and rapid prototyping.

Create a Simple Operator

First, create the folder structure for the operator:

mkdir -p operator/hooks
cd operator

Create a Dockerfile in the operator directory:

FROM ghcr.io/flant/shell-operator:latest
ADD ./hooks /hooks
RUN chmod +x /hooks/hook.sh

Now, inside the hooks folder, create a script file (e.g., hook.sh). Based on the documentation, try to fill in the blanks and questionmarks! Note, that our operator should just work on the namespace classroom, ignoring all other namespaces.

Example

#!/usr/bin/env bash
if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: <enter correct version>
  kind: <enter correct kind>
  executeHookOnEvent: ["<enter correct hooks>"]
  namespace: ???
EOF
else
  notifierName=$(jq -r .[0].??? $BINDING_CONTEXT_PATH)
  watchEvent=???
  echo "Particpant ${notifierName} was ${watchEvent,,}."
fi

This script outputs the hook configuration when called with --config (which is used by the shell operator), and otherwise logs a message using the resource name and the event.

Solution

#!/usr/bin/env bash
if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: stable.workshop.io/v1
  kind: Participant
  executeHookOnEvent: [ "Added", "Modified", "Deleted" ]
  namespace:
    nameSelector:
      matchNames: ["classroom"]
EOF
else
  notifierName=$(jq -r .[0].object.metadata.name $BINDING_CONTEXT_PATH)
  watchEvent=$(jq -r .[0].watchEvent $BINDING_CONTEXT_PATH)
  echo "Particpant ${notifierName} was ${watchEvent,,}."
fi

Build and Push Your Operator Image

Login to your Azure Container Registry:

docker login <your-acr-name>.azurecr.io -u <your-username> -p <your-password>

Solution

From within /kubernetes/CRD-Operator/operator, build and push the image:

docker buildx build -t <your-acr-name>.azurecr.io/participant-operator:<your-name> .
docker push <your-acr-name>.azurecr.io/participant-operator:<your-name>

Deploy the Operator Pod

Now it’s time to run your operator inside a Kubernetes pod. Create a yaml file for a pod that is using our previously built image. Add our service-account participant-notifier to monitor resources and log events.

name: participant-notifier-operator
namespace: classroom
serviceAccountName: participant-notifier

Solution

apiVersion: v1
kind: Pod
metadata:
  name: participant-notifier-operator
  namespace: classroom
spec:
  containers:
  - name: participant-notifier-operator
    image: philipptest.azurecr.io/participant-operator:latest
    #image: <your-acr-name>.azurecr.io/participant-operator:<your-name>
    imagePullPolicy: Always
  serviceAccountName: participant-notifier

A solution is provided in the exercise folder:

kubectl apply -f ~/exercise/kubernetes/CRD-Operator/participant-operator-pod.yaml

Testing

Watch the pod logs:

kubectl logs participant-notifier-operator -n classroom -f

Question

You will notice that the operator cannot watch for new participant-ressources. What could be the reason?

Solution and Answer

We are missing permissions for our service account. We need to give our service account permissions to watch (get, list) for participants instead for pods. Modify the created role to include permissions for participants:

k create clusterrole participant-notifier --verb=* --resource=participants.stable.workshop.io --dry-run=client -o yaml

or modify the exiting yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: participant-notifier-role
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - stable.workshop.io
  resources:
  - participants
  verbs:
  - '*'

Copy and past the new rules section to the existing yaml. Run kubectl apply -f <path to role file> to adjust the role. A solution is provided in ~/exercise/kubernetes/CRD-Operator/participant-notifier-role-update.yaml.

Now create a new Participant in the classroom namespace. You should see log output from your operator confirming that it reacted to the new resource.

Solution

k delete -f participant.yaml
k apply -f participant.yaml
k logs -n classroom pods/participant-notifier-operator | grep alice

Feel free to create more participants.

End of Lab