Azure Kubernetes Service (AKS) with Different Monitoring Solutions

This article would be introducing what monitoring solutions administrators could use to visually observe AKS metrics. Specifically, we would be talking about Prometheus and Azure Monitor (Log Analytics Workspace) as metrics-monitoring services and Grafana as the dashboard service.

Prometheus

# check HELM version
- helm version
# check whether the namespace for Prometheus is created
- kubectl get ns
# install Prometheus through HELMv3

We would need to note down Prometheus server service IP address as this is needed as the source of Grafana dashboard.

# check all services in monitoring namespace
- kubectl get svc -n monitoring
# save Prometheus service IP address into variable "SVC_IP"
- SVC_IP=$(kubectl get svc --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].spec.clusterIP}")
# check the variable "SVC_IP"
- echo $SVC_IP

Grafana

We could either port forward Grafana to local working machine’s port or we could actually check whether Grafana service is exposed as Load Balancer, so all external clients could leverage the public IP address to access the service.

# get Grafana's Pod name
- export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana " -o jsonpath="{.items[0].metadata.name}")
# Option 1: port forward Grafana's service to local machine's port
- kubectl --namespace monitoring port-forward $POD_NAME 3000
# Option 2: check whether Grafana's service could be changed into Load Balancer, so external clients could use public IP address to access the service
- kubectl get svc -n monitoring | grep grafana
# get the secret in monitoring namespace. The naming convention should be "grafana"
- kubectl get secret -n monitoring
# decode base64 to get the actual password for logging in Grafana dahshboard
- kubectl get secret -n monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Set up the Prometheus server as data source.

Click on “Add data source” → “Prometheus”.

Input Prometheus server service IP we located in the previous step.

There are hundreds of template to choose from here.

Once have the right template is located, copy the dashboard ID.

Import the dashboard with dashboard ID to Grafana.

Load the template and choose the data source “Prometheus”.

If everything goes as expected, we should be seeing screen captures as below.

Another good dashboard template for monitoring AKS cluster metrics.

Azure Monitor

Click on “Add data source” → “Azure Monitor”.

Once input all the required information click on “Load Subscriptions”. Then, select the correct subscription. Lastly, “Save & Test”. If everything goes right, it should show green response.

You could search for other Grafana dashboard templates, but this one is pretty popular and should be providing enough information at the moment.

Import the dashboard template like you did for Prometheus server. This time, instead of using Prometheus as the data source, please select “Azure Monitor”.

If everything goes as expected, we should be seeing similar metrics as below.

That is the end of this article! If you are new to AKS monitoring, this would be a great entry point for setting up something to observe metrics; if you are already an AKS veteran, you might find something useful while setting up trough Prometheus or Azure Monitor for Containers. Happy learning!

Learning new things about Kubernetes every day. Hopefully, the learning notes could help people on the same journey!