Configurations in Installing Heapster on Kubernetes and Certificate Errors

How to Install Heapster with InfluxDB on Kubernetes

The easiest way to install Heapster with influxDB on a Kubernetes cluster is by using yaml files in Heapster git repository. [1] Choose the latest release version from the git repository, clone the repository and get the yaml files in deploy/kube-config/influxdb folder.

These configurations have to be changed in these files before installing Heapster using these files. 

This is written based on an instance where Kubernetes cluster version v1.1.3 and Heapster version v0.19.1 are used.
  

1) heapster-controller.yaml

•  namespace: create a namespace and give that namespace here. 
• image: the docker image. You can build a new docker image for heapster and use it here. 
•  source: From where heapster get data. In this "--source" flag, options can be specified as URL query parameters.

useServiceAccount=true : use the service account token if one is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. 



Not setting this properly caused the certificate error. Heapster should have both the certificate and token set properly when accessing the api-server. This is further explained below.

•  sink: The backend that Heapster stores data into.
•  ports: Include the container port of Heapster to configure it through a load balancer. Default port of Heapster :8082

2) heapster-service.yaml

• Configure the load balancer in meta data, ports and selector

3) influxdb-grafana-controller.yaml

• Use the same INFLUXDB_SERVICE_URL used in heapster-controller.yaml file. 

After completing the configurations run the yaml files and create the pods and services. You can check the logs of Heapster pod to check whether the installation is properly done. 

 The certificate error 

If you are getting these errors in the logs of Heapster pod

• Failed to list *api.Node: Get <Kubernetes Mater IP>/api/v1/nodes: x509: failed to load system roots and no roots provided
• Failed to load nodes: Get <Kubernetes Mater IP>/api/v1/nodes: x509: certificate signed by unknown authority

It is because the certificates or token is not set when accessing the api server of the kubernetes cluster or the correct token and certificate is not mounted when installing heapster.

This is how I configured these certificate issues.

1) First go to the heapster pod and check whether a 'ca.crt' and a 'token' is mounted at /var/run/secrets/kubernetes.io/serviceaccount/.

2) If 'ca.crt' and 'token' is mounted at this folder, check whether it is the correct token and certificate. For that curl to the api server with the token and certificate. 

ex: 

• cat /var/run/secrets/kubernetes.io/serviceaccount/token 
• get the content of the token 
• export the token first to $newKey ( export newKey=<token content>) 
• curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $newKey" https://StagingAppCLoudLB-876478204.us-east-1.elb.amazonaws.com 443
  

3) If you are not getting the expected response when curled as above, the correct token and ca.crt is not mounted to heapster pod. Each namespace has a token specific to that namespace. You can find that token from another pod in the same namespace, copy that to heapster pod and try curling with that token which is copied. If you get the expected response you should try to mount heapster with the correct token and ca.crt

4) If you get the expected reponse when curled with default ca.crt and token of heapster pod, you get this certificate error as the token is not mounted to the path when accessing api-server. You can configure this by adding "useServiceAccount=true" to the source flag as mentioned above. 

References

[1] https://github.com/kubernetes/heapster/tree/v0.19.1/deploy/kube-config/influxdb  https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md

https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md