6. Monitoring

This guide walks you through setting up a full monitoring stack using Prometheus and Grafana on Kubernetes. By the end, you’ll monitor a guestbook application, visualize metrics in Grafana, and configure dashboards via ConfigMaps.

1. Install Prometheus & Grafana

We’ll use the official kube-prometheus-stack Helm chart.

Create a namespace called monitoring and install the chart using the values from ~/exercise/kubernetes/Monitoring/values-kube-prometheus.yaml into it:

Solution

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring -f ~/exercise/kubernetes/Monitoring/values-kube-prometheus.yaml

All resources in this tutorial will be placed in this namespace.

2. Track the Guestbook Application

We’ve preconfigured the guestbook app to expose metrics in Prometheus format. Use the following image in your deployment:

ghcr.io/thinkportrepo/k8s-admin-lab:prometheus

Update the container section in your Deployment YAML and deploy it in the monitoring namespace.

Solution

containers:
- image: ghcr.io/thinkportrepo/k8s-admin-lab:prometheus
  name: k8s-example-app

Now, expose the app via a load-balancer service (or ingress). Then access http://<example-app-service external-IP>/metrics and confirm the metric guestbook_new_entries_total appears. Try adding a new guestbook entry and watch how the metric updates!

Solution

Deployment yaml (~/exercise/kubernetes/Monitoring/guestbook-prometheus-deploy.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: guestbook
  name: guestbook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: guestbook
  template:
    metadata:
      labels:
        app: guestbook
    spec:
      containers:
        - image: ghcr.io/thinkportrepo/k8s-admin-lab:prometheus
          name: k8s-guestbook
          ports:
            - containerPort: 8080

Service yaml (~/kubernetes/Monitoring/guestbook-svc-lb.yaml):

apiVersion: v1
kind: Service
metadata:
  labels:
    app: guestbook
  name: guestbook-svc-lb
spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
      name: http-web # important for mapping the Service Monitor
  selector:
    app: guestbook

3. Investigate in Grafana and Prometheus

We’ll access the Prometheus and Grafana UIs via LoadBalancer services due to the adjusted value.yaml.

• Access Prometheus UI via http://<prometheus-stack-kube-prom-prometheus external-IP>:9090
• Access Grafana UI via http://<prometheus-stack-grafana external IP>

Retrieve the Grafana login credentials:

Solution

kubectl -n monitoring get secret prometheus-stack-grafana -o jsonpath="{.data.admin-user}" | base64 -d && echo
kubectl -n monitoring get secret prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d && echo

Tip

Take some time to look at the default Dashboards provided by the prom-grafana stack.

Set up a ServiceMonitor

Prometheus discovers targets using ServiceMonitor and PodMonitor resources. To scrape metrics from the guestbook app, create a ServiceMonitor that:

• Is in the monitoring namespace
• Has the release: prometheus-stack label (matching your Helm release)
• Targets a named port in the guestbook service

Refer to:

• ServiceMonitor Docs
• kube-prometheus-stack scrape config guide

Hint

Your guestbook service should have a named port like:

ports:
- protocol: TCP
  port: 80
  targetPort: 8080
  name: http-web  # Required for the ServiceMonitor to reference

And your `ServiceMonitor` might look like:

metadata:
  name: guestbook-sm
  labels:
    release: prometheus-stack

Solution

A working example can be found at ~/exercise/kubernetes/Monitoring/guestbook-service-monitor.yaml:

#https://github.com/prometheus-operator/prometheus-operator/blob/5b60eb7f1037736eb540ceded294ea005c4b97ec/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml#L4
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: guestbook-sm
  labels:
    release: prometheus-stack  # Must match Helm release name of prometheus (or adjust prometheus config)
spec:
  endpoints:
  - port: http-web
    interval: 15s
    path: /metrics
  jobLabel: guestbook-scrape
  selector:
    matchLabels:
      app: guestbook
  namespaceSelector:
    matchNames:
      - monitoring

Verify in Prometheus

Go to:

• http://<prometheus-stack-kube-prom-prometheus external-IP>:9090/targets — confirm your ServiceMonitor is listed and healthy
• http://<prometheus-stack-kube-prom-prometheus external-IP>:9090/query — query guestbook_new_entries_total to see metric values

Create a Grafana Dashboard

Visit Grafana at http://<external-IP-of-grafana>.

1. Go to Dashboards → New Dashboard
2. Click Add Visualization
3. Choose Prometheus as the data source
4. Use the query:

guestbook_new_entries_total

1. Save the dashboard

4. Save a Grafana Dashboard as a ConfigMap

Once you’ve built a dashboard, you can export it as JSON and persist it via a Kubernetes ConfigMap.

Grafana (via the kube-prometheus-stack) uses a sidecar to load dashboards from ConfigMaps. This is configured in values.yaml.

To see an example dashboard ConfigMap, try:

kubectl get cm prometheus-stack-kube-prom-alertmanager-overview -n monitoring -o yaml

Extract the .json from the Grafana UI (go into the settings of the Dashboard), copy its content and create a ConfigMap! Delete the Dashboard from the UI after you have copied the .json.

Solution

Use the example configuration at ~/exercise/kubernetes/Monitoring/grafana-dashboard-guestbook.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-myapp
  labels:
    grafana_dashboard: "1"   # Important for sidecar auto-discovery
data:
  myapp-dashboard.json: |
    {
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": {
              "type": "grafana",
              "uid": "-- Grafana --"
            },
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 0,
      "id": 29,
      "links": [],
      "panels": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "fieldConfig": {
            "defaults": {
              "color": {
                "mode": "palette-classic"
              },
              "custom": {
                "axisBorderShow": false,
                "axisCenteredZero": false,
                "axisColorMode": "text",
                "axisLabel": "",
                "axisPlacement": "auto",
                "barAlignment": 0,
                "barWidthFactor": 0.6,
                "drawStyle": "line",
                "fillOpacity": 0,
                "gradientMode": "none",
                "hideFrom": {
                  "legend": false,
                  "tooltip": false,
                  "viz": false
                },
                "insertNulls": false,
                "lineInterpolation": "linear",
                "lineWidth": 1,
                "pointSize": 5,
                "scaleDistribution": {
                  "type": "linear"
                },
                "showPoints": "auto",
                "spanNulls": false,
                "stacking": {
                  "group": "A",
                  "mode": "none"
                },
                "thresholdsStyle": {
                  "mode": "off"
                }
              },
              "mappings": [],
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  {
                    "color": "green"
                  },
                  {
                    "color": "red",
                    "value": 80
                  }
                ]
              }
            },
            "overrides": []
          },
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "id": 1,
          "options": {
            "legend": {
              "calcs": [],
              "displayMode": "list",
              "placement": "bottom",
              "showLegend": true
            },
            "tooltip": {
              "hideZeros": false,
              "mode": "single",
              "sort": "none"
            }
          },
          "pluginVersion": "12.0.2",
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "prometheus"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "guestbook_new_entries_total",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "Guestbook new entries",
          "type": "timeseries"
        }
      ],
      "preload": false,
      "schemaVersion": 41,
      "tags": [],
      "templating": {
        "list": []
      },
      "time": {
        "from": "now-3h",
        "to": "now"
      },
      "timepicker": {},
      "timezone": "browser",
      "title": "Guestbook new entries",
      "uid": "97dee7b9-ce49-4131-ad90-d7383bbf541d",
      "version": 1
    }