Get Prometheus Metrics from a Express.js app

Expose the metrics in Express.js app

I use Prometheus all the time for metrics and alert monitoring in Kubernetes. I decided to see how to setup monitoring in a Node/Express.js app. A quick search of npmjs and I found these two package prom-client a really detailed Prometheus client and express-prom-bundle which uses prom-client under the hood, I chose express-prom-bundle as it was a quick win and was producing metrics with a few lines of code, my repo is here. I installed the following packages in my express app

npm install prom-client express-prom-bundle — save

Then added the Prometheus middleware to all routes

Running the app

npm start
> express-prometheus@1.0.0 start /home/austincunningham/repo/express-prometheus
> node index.js
Listening at http://localhost:8080
# curl the hello world endpoint
curl localhost:8080/hello
{“hello”:”world”}%
# curl the metrics endpoint
curl localhost:8080/metrics
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.120868
# I cut the metrics output short here as its a lot of text but you get the idea

Setup the Express app on Openshift

I am using crc which is local Kubernetes development environment based on Red Hat Openshift. I create a container for the app based on the following DockerFile.

I then build, test the image locally and push the image

I can then deploy this on crc/openshift with the following two files
deployment.yaml

service.yaml

Apply the files to the default project

oc project default
oc apply -f deployment.yaml
oc apply -f service.yaml 
service/example-app created
# create a route to the service so you can access from the browser
oc expose service example-app

You can test the route by hitting the /metrics path in the browser you should see

Setup Prometheus Operator on Openshift

I am following the prometheus operator getting started guide. Applied the bundle from the setup on the default namespace

oc project default
oc apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

NOTE: Hit a issue where the prometheus-operator pod was in a crash loop backoff :(

Openshift has an operator hub so I did the following to fix the crashing operator pod. First I deleted the existing prometheus-operator deployment

oc delete deployment prometheus-operator

Logged in to crc/Openshift console as kubeadmin, in the administrator view go to OperatorHub and search for prometheus

Select the `Prometheus Operator` tile and `continue` then select `install` button

Select the default namespace from the drop down and install button again

Phew! that took longer to explain that to do.

Steps to get Prometheus to see the Express.js apps metrics

First we add the Prometheus CR(custom resource) to the default namespace to start the Prometheus instance
prometheus.yaml

And add the service
prometheus-service.yaml

Apply the files and create a route

oc apply -f prometheus.yaml
oc apply -f prometheus-service.yaml
# create a route to the service so you can access from the browser
oc expose service prometheus-operated

The way Prometheus scrapes metrics is that it uses a service monitor to check a service for a particular label. We have already created the service when we deployed the example-app with the label app: example-app in metadata.labels.

Next we create a serviceMonitor in the default namespace and with a selector for the app: example-app label. So we create the following file.
service-monitor.yaml

NOTE: metadata.labels `team: frontend` .

We upload the service-monitor.yaml file to the default namespace to create the serviceMonitor

oc apply -f service-monitor.yaml

In the prometheus.yaml CR we have already selected the service monitor this is done via serviceMonitorSelector label with the label team: frontend

Finally we need some RBAC rules which is Kubernetes version of permissions to allow Prometheus to see everything

Setup a service account, clusterRole and clusterRoleBinding. Create the following files
service-account.yaml

clusterRole.yaml

clusterRoleBinding.yaml

Apply the files to the default namespace

oc apply -f service-account.yaml 
oc apply -f clusterRole.yaml 
oc apply -f clusterRoleBinding.yaml

You should be able to access the route the default namespace

You can open the Prometheus UI by adding a http:// to the Prometheus HOST/PORT returned from the oc get routes command

So how do you know if its working

It takes a little while for the Prometheus operator to reconcile and to show up the new resources. In the Prometheus ui first check the Status\Service Discovery you should see example-app show up

NOTE: be patient it can take a while to show up

Then check the Status\Targets should see the following targets up

You also should be able to see metrics from example-app in the graph tab

That it I may do a follow up on setting up Grafana to use these metrics