Article

Table of Contents
Theme:
Was this article helpful?
Try Vultr Today with

$50 Free on Us!

Want to contribute?

You could earn up to $600 by adding new articles.

How to Monitor Your VKE Cluster with TOBS

Author: Quan Hua

Last Updated: Tue, Mar 29, 2022
Kubernetes System Admin Vultr Marketplace

Introduction

TOBS - The Observability Stack for Kubernetes is a Kubernetes monitoring stack that stores metrics in Prometheus, visualize them with Grafana, and store metrics in a TimescaleDB as long term storage. In addition, TOBS provides a CLI to make the deployments and operations easier. Some key components of TOBS are:

  • Prometheus to collect metrics
  • Grafana to visualize metrics from Prometheus
  • Promscale to store metrics in long-term storage and allow analysis with both PromQL and SQL
  • TimescaleDB as the long term database for Promscale
  • Other useful components such as AlertManager, Node-Exporter, Kube-State-Metrics, and so on.

This tutorial explains how to:

  • Install TOBS in your Vultr Kubernetes Engine (VKE) cluster
  • Use Prometheus to store your metrics in a long term database
  • Use Vultr Object Storage as the backup for your metrics database
  • Use Grafana to visualize your metrics
  • Access the database and perform SQL queries for complex analysis on metrics

Prerequisites

Create Vultr Kubernetes Cluster

  • Create a new VKE cluster with at least two 4GB nodes. 2GB nodes are not recommended for the TOBS stack.
  • Download the configuration and configure kubectl on your local machine

Create Vultr Object Storage

  • Create a new Vultr Object Storage to use as an S3-compatible backup for TimescaleDB.
  • Create a new bucket inside your Vultr Object Storage. The bucket name is demo-tobs in this tutorial.

Install TOBS

  1. Check your Kubernetes version.

    $ kubectl get nodes
    

    The result is like:

    NAME                STATUS   ROLES    AGE     VERSION
    node-02e926515d59   Ready    <none>   2m20s   v1.22.6
    node-85a58c1750d1   Ready    <none>   2m17s   v1.22.6
    
  2. Check the compatible matrix in this link to see if TOBS supports your Kubernetes version.

  3. Install TOBS on your local machine. If you are using Windows, you should create a Linux environment with Windows Subsystem for Linux.

    $ curl --proto '=https' --tlsv1.2 -sSLf https://tsdb.co/install-tobs-sh |sh
    

    The result is like:

    tobs 0.9.0 was successfully installed 🎉
    
    Binary is available at /home/ubuntu/.local/bin/tobs.
    
  4. After starting your Kubernetes cluster, run

    tobs install        
    
  5. To deploy TOBS on your Vultr Kubernetes cluster, you need to customize the configuration of the TOBS stack. Run the following command to create a configuration file named values.yaml

    $ tobs helm show-values > values.yaml
    
  6. Edit the values.yaml with your text editor and replace storage: 8Gi with storage: 10Gi. This change is essential because Vultr Block Storage requires at least 10GB in volume size.

  7. Under timescaledb-single, you can change the storage volume from size: 150Gi to a smaller number to optimize the resource cost.
  8. Install the TOBS stack with S3 backup with the following command. You should enter the information of your Vultr Object Storage (bucket name, hostname, key, and secret key)

    $ tobs install -b -f values.yaml
    

    Your output should look like this. When prompted for the S3 credentials, use the values from your Vultr Object Storage bucket.

    WARNING: Using a generated self-signed certificate for TLS access to TimescaleDB.
             This should only be used for development and demonstration purposes.
             To use a signed certificate, use the "--tls-timescaledb-cert" and "--tls-timescaledb-key"
             flags when issuing the tobs install command.
    
    Creating TimescaleDB tobs-certificate secret
    Creating TimescaleDB tobs-credentials secret
    
    We'll be asking a few questions about S3 buckets, keys, secrets and endpoints.
    
    For background information, visit these pages:
    
    Amazon Web Services:
    - https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
    - https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
    
    Digital Ocean:
    - https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
    
    Google Cloud:
    - https://cloud.google.com/storage/docs/migrating#migration-simple
    
    What is the name of the S3 bucket?
    demo-tobs
    
    What is the name of the S3 endpoint? (leave blank for default)
    ewr1.vultrobjects.com
    
    What is the region of the S3 endpoint? (leave blank for default)
    
    What is the S3 Key to use?
    NBWOK-redacted-1IMQSU
    
    What is the S3 Secret to use?
    tdCV0vN-redacted-Kx3OIVy
    
    Creating TimescaleDB tobs-pgbackrest secret
    Installing The Observability Stack
    
  9. The installation takes a few minutes to complete. You can check the progress with the kubectl get pods command. Keep in mind that you can see many CrashLoopBackOff statuses during the deployments.

  10. The following output of kubectl get pods indicates a successful installation. If you have any errors, check the Troubleshooting section at the end of this tutorial.

    NAME                                               READY   STATUS      RESTARTS       AGE
    alertmanager-tobs-kube-prometheus-alertmanager-0   2/2     Running     0              2m42s
    prometheus-tobs-kube-prometheus-prometheus-0       2/2     Running     0              2m42s
    tobs-grafana-95df94dc9-qpnsm                       3/3     Running     4 (95s ago)    2m54s
    tobs-grafana-db--1-g7pz7                           0/1     Completed   4              2m53s
    tobs-kube-prometheus-operator-d65c55845-xjz6h      1/1     Running     0              2m54s
    tobs-kube-state-metrics-56c568fdcc-6f8hp           1/1     Running     0              2m54s
    tobs-prometheus-node-exporter-2v74v                1/1     Running     0              2m54s
    tobs-prometheus-node-exporter-fj7x9                1/1     Running     0              2m54s
    tobs-promlens-7f778cc958-wb9dj                     1/1     Running     0              2m54s
    tobs-promscale-57497c97cf-htv2b                    1/1     Running     4 (119s ago)   2m54s
    tobs-timescaledb-0                                 2/2     Running     3 (40s ago)    2m53s 
    

Usage

Access Prometheus Dashboard

  • Run the following command to access the Prometheus through localhost with port 9090.

    $ tobs prometheus port-forward
    
  • Visit Prometheus dashboard at http://localhost:9090

Access Grafana Dashboard

  • Get the Grafana admin password

    $ tobs grafana get-password
    
  • Run the following command to access the Grafana dashboard through localhost with port 8080.

    $ tobs grafana port-forward
    
  • Visit Grafana dashboard at http://localhost:8080/dashboards

Access the TimescaleDB

  • Execute a psql session on the main database pod

    $ tobs timescaledb connect postgres -m
    

Here are some useful commands:

  • Expanded display:

    \x on
    
  • Query all metrics information:

    select * from prom_info.metric limit 3; 
    
  • Query one metric. For example, go_gc_duration_seconds:

    select * from go_gc_duration_seconds limit 3;
    
    select time, value, jsonb(labels) as labels from go_gc_duration_seconds limit 4;
    
  • Query all views in the database:

    SELECT table_schema, table_name 
    FROM information_schema.views 
    WHERE table_schema NOT IN ('information_schema', 'pg_catalog') 
    ORDER BY table_schema, table_name;
    

Troubleshooting

  • Get all events:

    $ kubectl get events --sort-by='.metadata.creationTimestamp'
    
  • Describe any specific pod to find the problem:

    $ kubectl describe pod <pod name>
    
  • View logs of any pod:

    $ kubectl logs <pod name>
    

Promscale pod can have a CrashLoopBackOff error due to failed password authentication. Here are the steps to resolve the problem:

  1. View the log of the Promscale pod. Replace the pod name with your pod name

    $ kubectl logs tobs-promscale-f984f8bf7-t4jts
    
  2. You may see the following error:

    password authentication failed for user \"postgres\" (SQLSTATE 28P01))
    
  3. Get the password to access the database from tobs-credentials secret.

    $ kubectl get secrets tobs-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}"
    
  4. Edit password stored inside tobs-promscale secret where the key is PROMSCALE_DB_PASSWORD.

    $ kubectl edit secrets tobs-promscale
    
  5. Delete the Promscale pod. Replace the pod name with your pod name.

    $ kubectl delete pod tobs-promscale-f984f8bf7-jlxw8
    

More Information

Want to contribute?

You could earn up to $600 by adding new articles.