How do I set up a Prometheus rule to routinely scrape an HTTP endpoint?

HOW TO -️ October 18, 2021

I'm getting started with Prometheus (Prometheus Operator), and I'm looking at adding it to an existing cluster, where each container has a /healthcheck endpoint, that is going to either return true, or an error.

I'm currently playing around with the kube-prometheus instructions in a sandbox cluster, and that basic setup is working, whereby I can monitor Memory/CPU/NetworkIO on each of my pods.

As a starting point, I want to:

  • Add a generic rule that adds /healthcheck as a Prometheus target.
  • Create a dashboard in Grafana that displays a grid of red/green panels for each pod.

How do I go about setting up the /healthcheck endpoint as a target? I've looked at the the out of the box prometheus-rules.yaml but I can't really make sense of it.

In terms of what I'm looking for in an answer here, is a series of high level step by step instructions, with links to documentation or concepts that are relevant.

Answer

Did you ever figure it out? @avi - nope. It was hitting the end point, but not parsing the result properly. Did you try to manually create the results, or were you using a Prometheus client to generate them? I've watched a few more tutorials and I noticed they all use the /metrics endpoint, instead of the /healthy endpoint. Could you reply to the comment?

Initializing...