Author: Quan Hua
Last Updated: Tue, Mar 29, 2022TOBS - The Observability Stack for Kubernetes is a Kubernetes monitoring stack that stores metrics in Prometheus, visualize them with Grafana, and store metrics in a TimescaleDB as long term storage. In addition, TOBS provides a CLI to make the deployments and operations easier. Some key components of TOBS are:
This tutorial explains how to:
kubectl
on your local machinedemo-tobs
in this tutorial.Check your Kubernetes version.
$ kubectl get nodes
The result is like:
NAME STATUS ROLES AGE VERSION
node-02e926515d59 Ready <none> 2m20s v1.22.6
node-85a58c1750d1 Ready <none> 2m17s v1.22.6
Check the compatible matrix in this link to see if TOBS supports your Kubernetes version.
Install TOBS on your local machine. If you are using Windows, you should create a Linux environment with Windows Subsystem for Linux.
$ curl --proto '=https' --tlsv1.2 -sSLf https://tsdb.co/install-tobs-sh |sh
The result is like:
tobs 0.9.0 was successfully installed 🎉
Binary is available at /home/ubuntu/.local/bin/tobs.
After starting your Kubernetes cluster, run
tobs install
To deploy TOBS on your Vultr Kubernetes cluster, you need to customize the configuration of the TOBS stack. Run the following command to create a configuration file named values.yaml
$ tobs helm show-values > values.yaml
Edit the values.yaml
with your text editor and replace storage: 8Gi
with storage: 10Gi
. This change is essential because Vultr Block Storage requires at least 10GB in volume size.
timescaledb-single
, you can change the storage volume from size: 150Gi
to a smaller number to optimize the resource cost.Install the TOBS stack with S3 backup with the following command. You should enter the information of your Vultr Object Storage (bucket name, hostname, key, and secret key)
$ tobs install -b -f values.yaml
Your output should look like this. When prompted for the S3 credentials, use the values from your Vultr Object Storage bucket.
WARNING: Using a generated self-signed certificate for TLS access to TimescaleDB.
This should only be used for development and demonstration purposes.
To use a signed certificate, use the "--tls-timescaledb-cert" and "--tls-timescaledb-key"
flags when issuing the tobs install command.
Creating TimescaleDB tobs-certificate secret
Creating TimescaleDB tobs-credentials secret
We'll be asking a few questions about S3 buckets, keys, secrets and endpoints.
For background information, visit these pages:
Amazon Web Services:
- https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
- https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
Digital Ocean:
- https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
Google Cloud:
- https://cloud.google.com/storage/docs/migrating#migration-simple
What is the name of the S3 bucket?
demo-tobs
What is the name of the S3 endpoint? (leave blank for default)
ewr1.vultrobjects.com
What is the region of the S3 endpoint? (leave blank for default)
What is the S3 Key to use?
NBWOK-redacted-1IMQSU
What is the S3 Secret to use?
tdCV0vN-redacted-Kx3OIVy
Creating TimescaleDB tobs-pgbackrest secret
Installing The Observability Stack
The installation takes a few minutes to complete. You can check the progress with the kubectl get pods
command. Keep in mind that you can see many CrashLoopBackOff
statuses during the deployments.
The following output of kubectl get pods
indicates a successful installation. If you have any errors, check the Troubleshooting section at the end of this tutorial.
NAME READY STATUS RESTARTS AGE
alertmanager-tobs-kube-prometheus-alertmanager-0 2/2 Running 0 2m42s
prometheus-tobs-kube-prometheus-prometheus-0 2/2 Running 0 2m42s
tobs-grafana-95df94dc9-qpnsm 3/3 Running 4 (95s ago) 2m54s
tobs-grafana-db--1-g7pz7 0/1 Completed 4 2m53s
tobs-kube-prometheus-operator-d65c55845-xjz6h 1/1 Running 0 2m54s
tobs-kube-state-metrics-56c568fdcc-6f8hp 1/1 Running 0 2m54s
tobs-prometheus-node-exporter-2v74v 1/1 Running 0 2m54s
tobs-prometheus-node-exporter-fj7x9 1/1 Running 0 2m54s
tobs-promlens-7f778cc958-wb9dj 1/1 Running 0 2m54s
tobs-promscale-57497c97cf-htv2b 1/1 Running 4 (119s ago) 2m54s
tobs-timescaledb-0 2/2 Running 3 (40s ago) 2m53s
Run the following command to access the Prometheus through localhost with port 9090.
$ tobs prometheus port-forward
Visit Prometheus dashboard at http://localhost:9090
Get the Grafana admin password
$ tobs grafana get-password
Run the following command to access the Grafana dashboard through localhost with port 8080.
$ tobs grafana port-forward
Visit Grafana dashboard at http://localhost:8080/dashboards
Execute a psql session on the main database pod
$ tobs timescaledb connect postgres -m
Here are some useful commands:
Expanded display:
\x on
Query all metrics information:
select * from prom_info.metric limit 3;
Query one metric. For example, go_gc_duration_seconds
:
select * from go_gc_duration_seconds limit 3;
select time, value, jsonb(labels) as labels from go_gc_duration_seconds limit 4;
Query all views in the database:
SELECT table_schema, table_name
FROM information_schema.views
WHERE table_schema NOT IN ('information_schema', 'pg_catalog')
ORDER BY table_schema, table_name;
Get all events:
$ kubectl get events --sort-by='.metadata.creationTimestamp'
Describe any specific pod to find the problem:
$ kubectl describe pod <pod name>
View logs of any pod:
$ kubectl logs <pod name>
Promscale pod can have a CrashLoopBackOff
error due to failed password authentication. Here are the steps to resolve the problem:
View the log of the Promscale pod. Replace the pod name with your pod name
$ kubectl logs tobs-promscale-f984f8bf7-t4jts
You may see the following error:
password authentication failed for user \"postgres\" (SQLSTATE 28P01))
Get the password to access the database from tobs-credentials
secret.
$ kubectl get secrets tobs-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}"
Edit password stored inside tobs-promscale
secret where the key is PROMSCALE_DB_PASSWORD
.
$ kubectl edit secrets tobs-promscale
Delete the Promscale pod. Replace the pod name with your pod name.
$ kubectl delete pod tobs-promscale-f984f8bf7-jlxw8