Prometheus custom scrape job

Prometheus custom scrape

Scrape job – is another way to configure Prometheus metrics collecting.

Here is example with the (Drone-CI) app monitoring by Prometheus and Grafana as a Helm deployment in the Kubernetes environment.


  • We will use Drone Helm charts from here: . It has built-in metrics exporter as part of drone application.

  • Turn on metrics in the Drone’s values.yaml file.

    enabled: true
  • Now Drone can export metrics on /metrics url. But it requires a bearer token. I changed done’s deployment-server.yaml file to be able to create a machine user:
          - name: DRONE_USER_CREATE
            value: username:system,machine:true,admin:true,token:{{ .Values.server.systemToken }}
  • I also added a new option in the Drone’s values.yaml file:
  systemToken: GeN6RatemE


  • We will use helm charts from here:

  • Now we need to create a new additional scrape configuration in the prometheus values.yaml file:

    # scrape configuration
    - job_name: "monitoring/drone"
      bearer_token: {{ .Values.drone.token }}
        - role: pod
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: drone
        - source_labels: [__meta_kubernetes_pod_container_port_number]
          action: keep
          regex: 80

Here you can see a kubernetes pod configuration for this job. And label-based selection with kubernetes meta data: app and port_number. You can find it using kubectl describe pod command or in the Prometheus /service-discovery url.


To install custom dashboards automatically I’m using existing script: prometheus-operator/hack/ but with small changes. I replaced existing charts with my own. For example for drone:

-        'source': '',
-        'destination': '../templates/grafana/dashboards',
+        'source': '',
+        'destination': '../templates/grafana/dashboards/custom',
         'type': 'json',
         'min_kubernetes': '1.10.0-0',
         'max_kubernetes': '1.16.0-0'

You need to run this script to generate new Helm ConfigMap yaml-file from Grafana json. Now you can deploy prometheus-operator with grafana and drone-ci support.

Tagged with:

Leave a Reply

Your email address will not be published.