Permanent or Volatile?

Before installing Prometheus, you should consider whether you want to install the software inside or outside the Kubernetes environment. An installation outside can open up many options for permanent data storage. Monitoring also works independent of the monitored system.

However, you can set up integration in Kubernetes far more easily; this applies to both the network and authentication. Thanks to persistent volumes [7] or stateful sets [8], Kubernetes has the option to keep data permanently. If you operate further external monitoring, you will likely combine Prometheus with Kubernetes.

Tested

To illustrate the information outlined above, I will demonstrate how you can run your own small Kubernetes cluster with a Prometheus extension based on Minikube [9]. Minikube offers the easiest way to test Kubernetes on your own computer, whether Linux, OS X, or Windows (Table 1). If you want to follow the steps, you will find an installation manual online [10]. The minikube start command generates a new Kubernetes; depending on the base system, Minikube still requires VirtualBox or kubectl to be in place.

Complete listings of the extracts shown in the article are available online [11], in particular, the YAML files with the Kubernetes definitions (*.yml): Unpack them in a working directory to send them later to Kubernetes using kubectl. Kubernetes internally stores the content generated from the YAML files and creates corresponding objects as namespaces, deployments, or services.

Because the following steps affect Minikube, I omit advanced topics such as persistent storage and role-based access (RBAC) [12], introduced in Kubernetes 1.6, that can be used with Prometheus.

Name Tag

Kubernetes uses namespaces to isolate the resources of individual users or a group of users from one another on a physical cluster. For the sample project, generate the monitoring namespace:

kubectl create -f 01-monitoring-namespace.yml

If you simply want to understand what is happening in the small Kubernetes cluster, launch the administration interface with

minikube dashboard

then select the monitoring namespace as shown in Figure 2.

Figure 2: The Minikube dashboard displays information about the cluster on the basis of its namespace.

The next step is then carried out by Prometheus. The software is available as an official Docker image [13], but without a configuration. To avoid having to build a new image for each change, pack the Kubernetes configuration as a prometheus.yml data object in a ConfigMap [14] with the name prometheus-configmap. You can then independently modify, delete, or create a new ConfigMap:

kubectl create -f 02-prometheus-configmap.yml

Deployments provide declarations [15] for updating pods and replica sets. The Kubernetes deployment for Prometheus (Listing 4) integrates the recently created ConfigMap as a new volume with the name prometheus volume-config by means of a volume mount in the /etc/prometheus/prometheus.yml path. This establishes a connection between Prometheus and its configuration:

kubectl create -f 03-prometheus-deployment.yml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
    labels:
        app: prometheus
    name: prometheus
    namespace: monitoring
spec:
    replicas: 1
    template:
        metadata:
            labels:
                app: prometheus
        spec:
            containers:
            - image: prom/prometheus:v1.7.1
                name: prometheus
                args:
                    - -config.file=/etc/prometheus/prometheus.yml
                    - -storage.local.path=/prometheus
                ports:
                - containerPort: 9090
                volumeMounts:
                - mountPath: /etc/prometheus
                    name: prometheus-volume-config
                - mountPath: /prometheus
                    name: prometheus-volume-data
            volumes:
            - name: prometheus-volume-config
                configMap:
                    name: prometheus-configmap
            - emptyDir: {}
                name: prometheus-volume-data

You can configure the directory that stores the Prometheus database with volumes, and more specifically as emptyDir. It discards the data when you relaunch the Prometheus pod; you will want to use persistent volumes here for a production setup.

You are still missing an appropriate service for Prometheus to access the current Prometheus instance:

kubectl create -f 04-prometheus-service.yml

The service can then be called via kubectl (Listing 5). At this point, note that Minikube sometimes displays services as pending . Do not worry, they are still working.

# kubectl get service --namespace=monitoring
NAME        CLUSTER-IP  EXTERNAL-IP  PORT(S)         AGE
prometheus  10.0.0.221  <pending>    9090:31244/TCP   1m