Adding Prometheus Monitoring for a New Service¶

Prometheus (kube-prometheus-stack, monitoring namespace) does not automatically scrape anything just because it exposes /metrics. It only scrapes services that have a ServiceMonitor or PodMonitor CRD telling the Prometheus Operator about them.

This is exactly what bit us with ArgoCD, Loki, Tempo, Alloy, and Traefik — all had working /metrics endpoints but were invisible to Prometheus until these were added.

Step 0 — Check you actually need to do this¶

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].labels.job' | sort -u

(Port-forward Prometheus first: kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090)

If your service's job is already in that list, you're done — someone already covered it.

Step 1 — Confirm the service actually exposes `/metrics`¶

From any pod in the cluster:

kubectl exec -it <any-pod> -- curl -s http://<service>.<namespace>.svc.cluster.local:<port>/metrics | head -5

You want to see real Prometheus exposition format (# HELP ... / # TYPE ... lines), not a 404 or HTML.

Step 2 — Find the right selector labels¶

Check the Service's own labels (not the pod's, unless you're doing Step 2b):

kubectl -n <namespace> get svc <service> -o jsonpath='{.metadata.labels}'

And its named ports (a ServiceMonitor references the port by name, not number):

kubectl -n <namespace> get svc <service> -o jsonpath='{.spec.ports}' | jq

Step 2b — When there's no Service exposing the metrics port¶

Some apps (Traefik is the example we hit) only expose a couple of ports via their Service, even though the pod itself listens on more (a metrics port with no Service in front of it). Check the pod's actual containerPorts:

kubectl -n <namespace> get pods -l <some-label> -o jsonpath='{.items[0].spec.containers[0].ports}' | jq

If the port you need isn't on the Service, use a PodMonitor instead (Step 4b) — it targets the pod's containerPort directly, skipping the Service lookup entirely.

Step 3 — Write the ServiceMonitor¶

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: <service>
  namespace: <namespace>          # can live in the same namespace as the service
  labels:
    release: kube-prometheus-stack # required — must match the Prometheus CR's selector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: <service>  # must match the Service's own labels
  endpoints:
    - port: <named-port>           # e.g. "http-metrics", not "9100"
      path: /metrics
      interval: 30s

The release: kube-prometheus-stack label is not optional

Without it, the Operator will never pick this up, and it'll just silently sit there doing nothing — no error, no event, nothing in the Operator's logs. Confirm the exact required value first:

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'

Step 4b — PodMonitor instead (no Service exposes the port)¶

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: <service>
  namespace: <namespace>
  labels:
    release: kube-prometheus-stack
spec:
  jobLabel: app.kubernetes.io/name   # optional, but avoids an ugly default job name
                                     # like "<namespace>/<podmonitor-name>"
  selector:
    matchLabels:
      app.kubernetes.io/name: <service>   # matches the POD's labels, not a Service
  podMetricsEndpoints:
    - port: <containerPort-name>
      path: /metrics
      interval: 30s

Step 5 — Apply it (via GitOps, not `kubectl apply` directly)¶

Add the manifest under ~/ArgoCD/manifests/<app>/ (or a shared manifests/monitoring-config/ if it doesn't belong to a specific app's own directory), commit, push. Auto-sync picks it up — or force it:

kubectl -n argocd annotate application <app> argocd.argoproj.io/refresh=hard --overwrite

Step 6 — Verify, and expect a reload lag¶

The Operator has to notice the new CRD, regenerate Prometheus's scrape config into a Secret, and a config-reloader sidecar has to notice that and call Prometheus's /-/reload. This chain reliably takes 30–90 seconds — don't panic if it's not showing up after 10 seconds.

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 &
curl -s "http://localhost:9090/api/v1/targets" | jq '.data.activeTargets[] | select(.labels.job=="<service>") | {job: .labels.job, health, lastError}'

If it's still missing after 2 minutes, check the raw generated config to see if the Operator even picked up your ServiceMonitor at all (this tells you whether the problem is the Operator not seeing your CRD, vs. Prometheus not having reloaded yet):

kubectl -n monitoring get secret prometheus-kube-prometheus-stack-prometheus \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip | grep -A5 "job_name.*<service>"

Job present in this config, but not in /targets → reload lag, just wait, or kubectl -n kube-system rollout restart deploy/sealed-secrets-style restart the Prometheus pod if it's really stuck.
Job missing from this config entirely → your selector labels don't match, or the release label is wrong/missing. Recheck Step 3/4b.

Adding it to a Grafana dashboard¶

Once up{job="<service>"} returns 1, you can build panels off it immediately — up is the built-in "is this target healthy" gauge, no extra work needed. See the homelab-status and portfolio-app-metrics dashboards (both version-controlled as ConfigMaps with the grafana_dashboard: "1" label, monitoring Grafana sidecar auto-discovers them across all namespaces) for the pattern.