Nov 4, 2021
Azure Kubernetes Service (AKS) with Prometheus and Grafana
Whoever that has played around Kubernetes would know Prometheus and Grafana. For those who are not yet in this area, Prometheus is an open source project in Cloud Native Computing Foundation (CNCF) designed for scraping metrics from Kubernetes cluster and storing them as time series data. On the other hand, Grafana is a dashboard service for showing metrics in graphical front end user interface. It could use Prometheus as the data source and also other data sources like Azure Monitor.
Prometheus and Grafana Installation
When deploying Prometheus in K8s clusters, HELM charts come in handy. Prometheus-operator is now deprecated, so further development would be maintained in Prometheus-community. It is the same steps for installing Prometheus as for any HELM-chart installation.
# add Prometheus-community as the local repository list
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts# update local repositories
helm repo update# search for the local repositories
helm repo list# search for the target chart within local repositories
helm search repo prometheus-community# create a namespace to save all Prometheus-related resources
kubectl create ns prometheus# install Prometheus through HELM
helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus# check all resources related with Prometheus
kubectl get all -n prometheus
If you encounter anything while installing Prometheus and Grafana through prometheus-community HELM chart, please check out issues raised by the community. Most of the time, someone in the community would have the answer to your question!
Prometheus and Grafana Test
# get the pod hosting Prometheus service
kubectl get all -n prometheus | grep operator# port forward the Prometheus service to localhost:port
kubectl port-forward -n prometheus prometheus-prometheus-kube-prometheus-prometheus-0 9090
The chart also includes installation of Grafana. So, we would just need to get Grafana’s user credentials for accessing.
# get Grafana's user credentials
kubectl get secrets prometheus-grafana -n prometheus -o yaml# search for admin-user and admin-password
# convert admin-user and admin-password from base64 encoding
echo "YWRtaW4=" | base64 -d
echo "cHJvbS1vcGVyYXRvcg==" | base64 -d
# get the pod hosting Grafana service
kubectl get all -n prometheus | grep grafana# port forward the Grafana service to localhost:port
kubectl port-forward -n prometheus prometheus-grafana-7b6d665f8f-z8xqm 3000
To view the metrics in Grafana dashboard, we would need to ensure 2 components, the data source and the dashboard template. Since we are trying to collect metrics from AKS, we could also use Azure Monitor as data source but we would need to ensure there is an Azure Active Directory identity that has sufficient permissions to read the monitored data.
If you choose the way to use Azure Monitor as data source, follow this article to create a service principal and this article for assigning the “Monitoring Reader Role” in the scope of the resource group of log analytics workspace.
This is to check whether the Azure Active Directory identity has been successfully created.
This is to ensure the role with sufficient permissions is assigned to the AAD identity.
Back to Grafana portal, we could now use all the information we have to add Azure Monitor as data source.
Make sure when clicking on “Save & test”, there is no error prompted.
Next, we would need to find a Grafana dashboard template to use. This can be easily found with giving the keyword of “Azure Kubernetes Service + Grafana Dashboard” in any search engine. However, we would use the most commonly used dashboard for AKS as an example. Notice I have highlighted the dashboard number in red.
Now, we head back to Grafana dashboard and try import this dashboard template.
Put in the dashboard number and click “Load”.
Select “Dashboard” and then “Manage”, you should be seeing something similar to “Azure Monitor for Containers…”. Click on it!
The dashboard should be something like below.
That is it! That is how you set up Prometheus and Grafana on your K8s clusters with just 1 HELM chart deployment! If you are not familiar with HELM, please refer to another article written by me. Happy learning!
