Prometheus custom scrape
Scrape job – is another way to configure Prometheus metrics collecting.
Here is example with the Drone.io
(Drone-CI) app monitoring by Prometheus and Grafana as a Helm deployment in the Kubernetes environment.
Drone
-
We will use Drone Helm charts from here: https://github.com/helm/charts/tree/master/stable/drone . It has built-in metrics exporter as part of drone application.
-
Turn on metrics in the Drone’s
values.yaml
file.
metrics:
prometheus:
enabled: true
- Now Drone can export metrics on /metrics url. But it requires a bearer token. I changed done’s deployment-server.yaml file to be able to create a machine user:
- name: DRONE_USER_CREATE
value: username:system,machine:true,admin:true,token:{{ .Values.server.systemToken }}
- I also added a new option in the Drone’s values.yaml file:
server:
systemToken: GeN6RatemE
Prometheus
-
We will use helm charts from here: https://github.com/helm/charts/tree/master/stable/prometheus-operator
-
Now we need to create a new additional scrape configuration in the prometheus values.yaml file:
prometheus:
prometheusSpec:
# scrape configuration
additionalScrapeConfigs:
- job_name: "monitoring/drone"
bearer_token: {{ .Values.drone.token }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: drone
- source_labels: [__meta_kubernetes_pod_container_port_number]
action: keep
regex: 80
Here you can see a kubernetes pod
configuration for this job. And label-based selection with kubernetes meta data: app
and port_number
. You can find it using kubectl describe pod
command or in the Prometheus /service-discovery url.
Grafana
To install custom dashboards automatically I’m using existing script: prometheus-operator/hack/sync_grafana_dashboards.py but with small changes. I replaced existing charts with my own. For example for drone:
...
- 'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/grafana.json',
- 'destination': '../templates/grafana/dashboards',
+ 'source': 'https://raw.githubusercontent.com/carldanley/radium-grafana-dashboards/master/drone-ci.json',
+ 'destination': '../templates/grafana/dashboards/custom',
'type': 'json',
'min_kubernetes': '1.10.0-0',
'max_kubernetes': '1.16.0-0'
},
...
You need to run this script to generate new Helm ConfigMap yaml-file from Grafana json. Now you can deploy prometheus-operator with grafana and drone-ci support.