Prometheus custom scrape

Scrape job – is another way to configure Prometheus metrics collecting.

Here is example with the Drone.io (Drone-CI) app monitoring by Prometheus and Grafana as a Helm deployment in the Kubernetes environment.

Drone

• We will use Drone Helm charts from here: https://github.com/helm/charts/tree/master/stable/drone . It has built-in metrics exporter as part of drone application.

• Turn on metrics in the Drone’s values.yaml file.

metrics:
  prometheus:
    enabled: true

• Now Drone can export metrics on /metrics url. But it requires a bearer token. I changed done’s deployment-server.yaml file to be able to create a machine user:

          - name: DRONE_USER_CREATE
            value: username:system,machine:true,admin:true,token:{{ .Values.server.systemToken }}

• I also added a new option in the Drone’s values.yaml file:

server:
  systemToken: GeN6RatemE

Prometheus

• We will use helm charts from here: https://github.com/helm/charts/tree/master/stable/prometheus-operator

• Now we need to create a new additional scrape configuration in the prometheus values.yaml file:

prometheus:
  prometheusSpec:
    # scrape configuration
    additionalScrapeConfigs:
    - job_name: "monitoring/drone"
      bearer_token: {{ .Values.drone.token }}
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: drone
        - source_labels: [__meta_kubernetes_pod_container_port_number]
          action: keep
          regex: 80

Here you can see a kubernetes pod configuration for this job. And label-based selection with kubernetes meta data: app and port_number. You can find it using kubectl describe pod command or in the Prometheus /service-discovery url.

Grafana

To install custom dashboards automatically I’m using existing script: prometheus-operator/hack/sync_grafana_dashboards.py but with small changes. I replaced existing charts with my own. For example for drone:

...
-        'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/grafana.json',
-        'destination': '../templates/grafana/dashboards',
+        'source': 'https://raw.githubusercontent.com/carldanley/radium-grafana-dashboards/master/drone-ci.json',
+        'destination': '../templates/grafana/dashboards/custom',
         'type': 'json',
         'min_kubernetes': '1.10.0-0',
         'max_kubernetes': '1.16.0-0'
     },
...

You need to run this script to generate new Helm ConfigMap yaml-file from Grafana json. Now you can deploy prometheus-operator with grafana and drone-ci support.