How to add a new custom serviceMonitor in Prometheus?
Here is example with PostgreSQL monitoring by Prometheus and Grafana as a Helm deployment in Kubernetes environment.
Posgresql
-
We will use this helm chart: https://github.com/helm/charts/tree/master/stable/postgresql It has built-in prometheus metric exporter (as additional container in pod).
-
Turn on metrics settings in
values.yaml
:
metrics:
enabled: true
serviceMonitor:
enabled: true
Or if you are using helmfile
like me, add these values:
- name: metrics.enabled
value: true
- name: metrics.serviceMonitor.enabled
value: true
- Deploy postgresql:
helm install -f postgres-values.yaml stable/postgresql
- Ensure that postgresql metrics service is available. In my example it’s a gitea-db-postgresql release:
kubectl get services | grep metrics
gitea-db-postgresql-metrics ClusterIP 10.0.2.193 <none> 9187/TCP
Prometheus
-
We will use a helm chart from: https://github.com/helm/charts/tree/master/stable/prometheus-operator
-
By default prometheus will discover only the same namespace which it has been deployed. So if you want to run prometheus in different namespace, you need to mark available namespaces with labels. For example a new label prometheus=enabled.
kubectl label --overwrite namespace --all "prometheus=enabled"
Then turn on serviceMonitorNamespaceSelector
in values.yaml:
prometheus:
prometheusSpec:
serviceMonitorNamespaceSelector:
matchLabels:
prometheus: enabled
- Checkout postgresql metrics service:
kubectl describe service gitea-db-postgresql-metrics
Name: gitea-db-postgresql-metrics
Namespace: default
Labels: app=postgresql
chart=postgresql-6.2.0
heritage=Tiller
release=gitea-db
Annotations: prometheus.io/port: 9187
prometheus.io/scrape: true
Selector: app=postgresql,release=gitea-db,role=master
Type: ClusterIP
IP: 10.0.2.193
Port: metrics 9187/TCP
TargetPort: metrics/TCP
Endpoints: 10.52.3.156:9187
Session Affinity: None
Events: <none>
- Add postgresql serviceMonitor support in prometheus values.yaml file based on description above:
prometheus:
prometheusSpec:
additionalServiceMonitors:
- name: "postgresql"
namespaceSelector:
any: true
selector:
matchLabels:
app: postgresql
endpoints:
- port: "metrics"
- targetPort: "metrics/TCP"
scheme: tcp
- Deploy prometheus:
helm install -f prometheus-values.yaml stable/prometheus-operator
And check /targets url. You can find something like this:
monitoring/postgresql/0 (1/1 up)
Endpoint State Labels Last Scrape Scrape Duration Error
http://10.52.3.156:9187/metrics
up endpoint="metrics" instance="10.52.3.156:9187" job="gitea-db-postgresql-metrics" namespace="default" pod="gitea-db-postgresql-0" service="gitea-db-postgresql-metrics"
Grafana
To install a custom dashboards automatically I’m using existing script: prometheus-operator/hack/sync_grafana_dashboards.py but with small changes. I replaced existing charts with my own. For example for postgresql:
...
- 'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/grafana.json',
- 'destination': '../templates/grafana/dashboards',
+ 'source': 'https://raw.githubusercontent.com/sinsengumi/grafana-dashboards/master/postgresql_statistics.json',
+ 'destination': '../templates/grafana/dashboards/custom',
'type': 'json',
'min_kubernetes': '1.10.0-0',
'max_kubernetes': '1.14.0-0'
},
...
I also fixed some naming issues:
@@ -77,7 +57,7 @@ header = '''# Generated from '%(name)s'
apiVersion: v1
kind: ConfigMap
metadata:
- name: {{ printf "%%s-%%s" (include "prometheus-operator.fullname" $) "%(name)s" | trunc 63 | trimSuffix "-" }}
+ name: {{ printf "%%s-%%s" (include "prometheus-operator.fullname" $) "%(name)s" | lower | replace "_" "-" | trunc 63 | trimSuffi
x "-" }}
labels:
{{- if $.Values.grafana.sidecar.dashboards.label }}
{{ $.Values.grafana.sidecar.dashboards.label }}: "1"
Now you need to run this script to download json files and convert it into Helm ConfigMap yaml file.
I also changed old ${DS_PROMETHEUS} placeholder to Prometheus
for files in ../templates/grafana/dashboards/custom/*.yaml
do
echo "fixing $files..."
sed 's#${DS_PROMETHEUS}#Prometheus#g' -i $files
done
Now you can apply changes (using helmfile
or helm upgrade
). Your custom dashboard will be uploaded to Grafana.