How to Monitor Your VKE Cluster with TOBS

Introduction

TOBS - The Observability Stack for Kubernetes is a Kubernetes monitoring stack that stores metrics in Prometheus, visualize them with Grafana, and store metrics in a TimescaleDB as long term storage. In addition, TOBS provides a CLI to make the deployments and operations easier. Some key components of TOBS are:

Prometheus to collect metrics
Grafana to visualize metrics from Prometheus
Promscale to store metrics in long-term storage and allow analysis with both PromQL and SQL
TimescaleDB as the long term database for Promscale
Other useful components such as AlertManager, Node-Exporter, Kube-State-Metrics, and so on.

This tutorial explains how to:

Install TOBS in your Vultr Kubernetes Engine (VKE) cluster
Use Prometheus to store your metrics in a long term database
Use Vultr Object Storage as the backup for your metrics database
Use Grafana to visualize your metrics
Access the database and perform SQL queries for complex analysis on metrics

Prerequisites

Create Vultr Kubernetes Cluster

Create a new VKE cluster with at least two 4GB nodes. 2GB nodes are not recommended for the TOBS stack.
Download the configuration and configure kubectl on your local machine

Create Vultr Object Storage

Create a new Vultr Object Storage to use as an S3-compatible backup for TimescaleDB.
Create a new bucket inside your Vultr Object Storage. The bucket name is demo-tobs in this tutorial.

Install TOBS

Check your Kubernetes version.

$ kubectl get nodes

The result is like:

NAME                STATUS   ROLES    AGE     VERSION
node-02e926515d59   Ready    <none>   2m20s   v1.22.6
node-85a58c1750d1   Ready    <none>   2m17s   v1.22.6

Check the compatible matrix in this link to see if TOBS supports your Kubernetes version.

Install TOBS on your local machine. If you are using Windows, you should create a Linux environment with Windows Subsystem for Linux.

$ curl --proto '=https' --tlsv1.2 -sSLf https://tsdb.co/install-tobs-sh |sh

The result is like:

tobs 0.9.0 was successfully installed 🎉

Binary is available at /home/ubuntu/.local/bin/tobs.

After starting your Kubernetes cluster, run
```
tobs install        
```
To deploy TOBS on your Vultr Kubernetes cluster, you need to customize the configuration of the TOBS stack. Run the following command to create a configuration file named values.yaml
```
$ tobs helm show-values > values.yaml
```
Edit the values.yaml with your text editor and replace storage: 8Gi with storage: 10Gi. This change is essential because Vultr Block Storage requires at least 10GB in volume size.
Under timescaledb-single, you can change the storage volume from size: 150Gi to a smaller number to optimize the resource cost.

Install the TOBS stack with S3 backup with the following command. You should enter the information of your Vultr Object Storage (bucket name, hostname, key, and secret key)

$ tobs install -b -f values.yaml

Your output should look like this. When prompted for the S3 credentials, use the values from your Vultr Object Storage bucket.

WARNING: Using a generated self-signed certificate for TLS access to TimescaleDB.
         This should only be used for development and demonstration purposes.
         To use a signed certificate, use the "--tls-timescaledb-cert" and "--tls-timescaledb-key"
         flags when issuing the tobs install command.

Creating TimescaleDB tobs-certificate secret
Creating TimescaleDB tobs-credentials secret

We'll be asking a few questions about S3 buckets, keys, secrets and endpoints.

For background information, visit these pages:

Amazon Web Services:
- https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
- https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

Digital Ocean:
- https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility

Google Cloud:
- https://cloud.google.com/storage/docs/migrating#migration-simple

What is the name of the S3 bucket?
demo-tobs

What is the name of the S3 endpoint? (leave blank for default)
ewr1.vultrobjects.com

What is the region of the S3 endpoint? (leave blank for default)

What is the S3 Key to use?
NBWOK-redacted-1IMQSU

What is the S3 Secret to use?
tdCV0vN-redacted-Kx3OIVy

Creating TimescaleDB tobs-pgbackrest secret
Installing The Observability Stack

The installation takes a few minutes to complete. You can check the progress with the kubectl get pods command. Keep in mind that you can see many CrashLoopBackOff statuses during the deployments.

The following output of kubectl get pods indicates a successful installation. If you have any errors, check the Troubleshooting section at the end of this tutorial.

NAME                                               READY   STATUS      RESTARTS       AGE
alertmanager-tobs-kube-prometheus-alertmanager-0   2/2     Running     0              2m42s
prometheus-tobs-kube-prometheus-prometheus-0       2/2     Running     0              2m42s
tobs-grafana-95df94dc9-qpnsm                       3/3     Running     4 (95s ago)    2m54s
tobs-grafana-db--1-g7pz7                           0/1     Completed   4              2m53s
tobs-kube-prometheus-operator-d65c55845-xjz6h      1/1     Running     0              2m54s
tobs-kube-state-metrics-56c568fdcc-6f8hp           1/1     Running     0              2m54s
tobs-prometheus-node-exporter-2v74v                1/1     Running     0              2m54s
tobs-prometheus-node-exporter-fj7x9                1/1     Running     0              2m54s
tobs-promlens-7f778cc958-wb9dj                     1/1     Running     0              2m54s
tobs-promscale-57497c97cf-htv2b                    1/1     Running     4 (119s ago)   2m54s
tobs-timescaledb-0                                 2/2     Running     3 (40s ago)    2m53s

Usage

Access Prometheus Dashboard

Run the following command to access the Prometheus through localhost with port 9090.
```
$ tobs prometheus port-forward
```
Visit Prometheus dashboard at http://localhost:9090

Access Grafana Dashboard

Get the Grafana admin password
```
$ tobs grafana get-password
```
Run the following command to access the Grafana dashboard through localhost with port 8080.
```
$ tobs grafana port-forward
```
Visit Grafana dashboard at http://localhost:8080/dashboards

Access the TimescaleDB

Execute a psql session on the main database pod
```
$ tobs timescaledb connect postgres -m
```

Here are some useful commands:

Expanded display:
```
\x on
```

Query all metrics information:

select * from prom_info.metric limit 3;

Query one metric. For example, go_gc_duration_seconds:

select * from go_gc_duration_seconds limit 3;

select time, value, jsonb(labels) as labels from go_gc_duration_seconds limit 4;

Query all views in the database:

SELECT table_schema, table_name 
FROM information_schema.views 
WHERE table_schema NOT IN ('information_schema', 'pg_catalog') 
ORDER BY table_schema, table_name;

Troubleshooting

Get all events:

$ kubectl get events --sort-by='.metadata.creationTimestamp'

Describe any specific pod to find the problem:
```
$ kubectl describe pod <pod name>
```
View logs of any pod:
```
$ kubectl logs <pod name>
```

Promscale pod can have a CrashLoopBackOff error due to failed password authentication. Here are the steps to resolve the problem:

View the log of the Promscale pod. Replace the pod name with your pod name
```
$ kubectl logs tobs-promscale-f984f8bf7-t4jts
```

You may see the following error:

password authentication failed for user \"postgres\" (SQLSTATE 28P01))

Get the password to access the database from tobs-credentials secret.

$ kubectl get secrets tobs-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}"

Edit password stored inside tobs-promscale secret where the key is PROMSCALE_DB_PASSWORD.
```
$ kubectl edit secrets tobs-promscale
```
Delete the Promscale pod. Replace the pod name with your pod name.
```
$ kubectl delete pod tobs-promscale-f984f8bf7-jlxw8
```

More Information

Tobs GitHub Repository

Want to contribute?

You could earn up to $600 by adding new articles.

Submit your article Suggest an update Request an article

Article