Creating a Recording Rule

The Scenario

One of our Prometheus expressions in Grafana is running a bit slow. After a careful diagnosis, we’ve decided that the expression would be more effective if it was turned into a recording rule. We will need to configure our Prometheus environment to be able to use recording rules.

Logging In and Setting up the Environment

Use the IP address and credentials provided on the hands-on lab overview page, and log in with SSH to the server. Once we're in, become root right off and navigate into /root/prometheus. In that directory, there is a bootstrap.sh script that we need to execute. Once we've done that, we can use kubectl to show running pods. There should be two, and they both should be in a running state. Here are all of those commands:

sudo su -
cd /root/prometheus
./bootstrap.sh
kubectl get pods -n monitoring

Create the Recording Rule

Add the job:prcoess_cpu_seconds:rate5m recording rule to prometheus-rules-config-map.yml. The file should look like this when we're done editing it:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: prometheus-rules-conf
  namespace: monitoring
data:
  process_rules.yml: |
    groups:
    - name: process_rules
      rules:
      - record: job:process_cpu_seconds:rate5m
        expr: (rate(process_cpu_seconds_total[5m]))

Apply the changes with this command:

kubectl apply -f prometheus-rules-config-map.yml

Update the Prometheus Configuration Edit prometheus-config-map.yml

vi prometheus-config-map.yml

Add the rule files to prometheus-config-map.yml:

rule_files:
 - /var/prometheus/rules/*_rules.yml

It should look like this when we're done:

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
 name: prometheus-server-conf
namespace: monitoring
data:
prometheus.yml: |-
 global:
   scrape_interval: 5s
   evaluation_interval: 5s

 rule_files:
 - /var/prometheus/rules/*_rules.yml

 scrape_configs:
   - job_name: 'kubernetes-apiservers'
...

Apply the changes with this command:

kubectl apply -f prometheus-config-map.yml

Add prometheus-rules-conf to the Prometheus Deployment Edit prometheus-deployment.yml and add the prometheus-rules-volume volume to it. We've got to add a name and a mountPath down in the volumeMounts section. Then we've got to add the name again, and where the name of the confgMap, down in the volumes section. When we're finished, it should look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.2.1
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--web.enable-lifecycle"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
            - name: prometheus-rules-volume
              mountPath: /var/prometheus/rules
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf

        - name: prometheus-rules-volume
          configMap:
            name: prometheus-rules-conf

        - name: prometheus-storage-volume
          emptyDir: {}
...

Apply the changes with this command:

kubectl apply -f prometheus-deployment.yml

View It in a Browser Back on the hands-on lab overview page, grab our server's public IP. In a new browser tab, visit that IP on port 8080:

http://<IP>:8080

In the web interface we land on, click the Status dropdown, and select Rules.#