Use Karmada to Orchestrate Vultr Kubernetes Engine in Multiple Locations

Updated on December 2, 2022
Use Karmada to Orchestrate Vultr Kubernetes Engine in Multiple Locations header image

Introduction

Karmada, which stands for "Kubernetes Armada," is an open-source, multi-cloud, and multi-cluster Kubernetes orchestration system. It allows you to run cloud-native applications across multiple Kubernetes clusters and regions. It has advanced scheduling capabilities, high availability, centralized multi-cloud management, failure recovery, and traffic scheduling. Karmada supports deployments on-prem, at the edge, and on public clouds.

This guide explains using Karmada with Vultr Kubernetes Engine (VKE). When you follow this guide, you will:

  • Create three VKE clusters in Vultr's Mumbai, Paris, and New York regions.
  • Create a fourth VKE cluster for Karmada to manage the rest of the nodes.
  • Install the Karmada management system on the cluster manager and Karmada agent on other clusters.
  • Deploy an example application and distribute the workload across all clusters.

Prerequisites

Before beginning this guide, you should:

  • Have the kubectl CLI installed and configured on your local machine.
  • Have the Helm client installed on your local machine.

Install Karmada API Server

  1. Deploy four VKE clusters at Vultr with at least three nodes each and download the respective kubeconfig files.

    • Deploy one cluster in Mumbai. Name the kubeconfig kubeconfig-mumbai.
    • Deploy one cluster in Paris. Name the kubeconfig kubeconfig-paris.
    • Deploy one cluster in New York. Name the kubeconfig kubeconfig-newyork.
    • Deploy one cluster in any other location. Name the kubeconfig kubeconfig-cluster-manager.
  2. Add the Helm repository to your local machine.

     # helm repo add karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts
  3. Next, verify the added repository.

     # helm repo list

    Sample output:

     NAME            URL                                                               
     karmada-charts  https://raw.githubusercontent.com/karmada-io/karmada/master/charts
  4. Karmada API server must be reachable from other clusters. So you will require a public IP address of the cluster manager node to expose it for external access. First, retrieve the IP address of the cluster manager.

     # kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}' --kubeconfig=kubeconfig-cluster-manager

    The example output is shown below. Note that 192.0.2.100 is an example IP address for documentation. You'll substitute that address with the actual address of your cluster throughout this guide.

     192.0.2.100
  5. Install the Karmada control plane on the cluster manager node.

     # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-cluster-manager --create-namespace --namespace karmada-system -- version=1.2.0 --set apiServer.hostNetwork=false --set apiServer.serviceType=NodePort --set apiServer.nodePort=32443 --set certs.auto.hosts[0]="kubernetes.default.svc" --set certs.auto.hosts[1]="*.etcd.karmada-system.svc.cluster.local" --set certs.auto.hosts[2]="*.karmada-system.svc.cluster.local" --set certs.auto.hosts[3]="*.karmada-system.svc" --set certs.auto.hosts[4]="localhost" --set certs.auto.hosts[5]="127.0.0.1" --set certs.auto.hosts[6]="192.0.2.100"

    > Note: Replace the IP 192.0.2.100 with the IP address of cluster manager.

    Sample output:

     NAME: karmada
     LAST DEPLOYED: Sat Nov 19 22:18:54 2022
     NAMESPACE: karmada-system
     STATUS: deployed
     REVISION: 1
     TEST SUITE: None
  6. Download the kubeconfig configuration file from the cluster manager to connect to the Karmada API:

     # kubectl get secret karmada-kubeconfig --kubeconfig=kubeconfig-cluster-manager -n karmada-system -o jsonpath={.data.kubeconfig} | base64 -d > karmada-config
  7. Edit the downloaded kubeconfig file:

     # nano karmada-config

    Find the following line:

     server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443 # <- this works only in the cluster

    Replace that line with the cluster manager node's IP address so that it will work outside the cluster network.

     server: https://192.0.2.100:32443                                    

Install Karmada Agent on Three Clusters

In this section, you will install the Karmada agent using Helm on other clusters and join them to the Karmada cluster.

  1. First, verify the active status of all clusters.

     # kubectl get pods -A --kubeconfig=kubeconfig-mumbai
     # kubectl get pods -A --kubeconfig=kubeconfig-paris
     # kubectl get pods -A --kubeconfig=kubeconfig-newyork
  2. Next, copy the agent.kubeconfig.caCrt, agent.kubeconfig.crt, and agent.kubeconfig.key value from the karmada-config file, decode the value using base64 and store the decoded values in variables named ca, crt and key.

  3. Run the following commands to install the Karmada agent on each cluster and link it to the cluster manager.

     # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-mumbai --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=mumbai --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443
    
     # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-paris --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=paris --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443
    
     # helm install karmada karmada-charts/karmada --kubeconfig=kubeconfig-newyork --create-namespace --namespace karmada-system --version=1.2.0 --set installMode=agent --set agent.clusterName=newyork --set agent.kubeconfig.caCrt="$ca" --set agent.kubeconfig.crt="$crt" --set agent.kubeconfig.key="$key" --set agent.kubeconfig.server=https://192.0.2.100:32443

    > Note: Replace the --kubeconfig value with your kubeconfig file of each cluster, agent.clusterName value with each cluster name, and 192.0.2.100 with the IP address of the cluster manager.

  4. Verify the Karmada agent installation.

     # kubectl get clusters --kubeconfig=karmada-config

    Sample output:

     NAME      VERSION   MODE   READY   AGE
     mumbai    v1.25.4   Pull   True    95s
     newyork   v1.25.4   Pull   True    7s
     paris     v1.25.4   Pull   True    31s

Create Karmada Policy for Orchestrating Multicluster Deployment

In this section, you will create a test deployment, define a policy to assign a workload, submit a workload to Karmada then distribute it across all clusters.

  1. First, create a test deployment with three replicas to distribute equally across the three clusters.

     # nano deployment.yaml

    Add the following configurations.

     apiVersion: apps/v1
     kind: Deployment
     metadata:
      name: hello
     spec:
      replicas: 3
      selector:
        matchLabels:
          app: hello
      template:
        metadata:
          labels:
            app: hello
        spec:
          containers:
            - image: stefanprodan/podinfo
              name: hello
     ---
     apiVersion: v1
     kind: Service
     metadata:
      name: hello
     spec:
      ports:
        - port: 5000
          targetPort: 9898
      selector:
        app: hello
  2. Apply the deployment to the Karmada API server.

     # kubectl apply -f deployment.yaml --kubeconfig=karmada-config

    Sample output:

     deployment.apps/hello created
     service/hello created
  3. Run the following command to verify the deployment.

     # kubectl get deployments --kubeconfig=karmada-config

    Sample output:

     NAME    READY   UP-TO-DATE   AVAILABLE   AGE
     hello   0/3     0            0           54s
  4. Let's describe the deployment with detailed information.

     # kubectl describe deployment hello --kubeconfig=karmada-config

    Sample output:

     Name:                   hello
     Namespace:              default
     CreationTimestamp:      Wed, 23 Nov 2022 14:25:56 +0530
     Labels:                 <none>
     Annotations:            <none>
     Selector:               app=hello
     Replicas:               3 desired | 0 updated | 0 total | 0 available | 0 unavailable
     StrategyType:           RollingUpdate
     MinReadySeconds:        0
     RollingUpdateStrategy:  25% max unavailable, 25% max surge
     Pod Template:
       Labels:  app=hello
       Containers:
        hello:
         Image:        stefanprodan/podinfo
         Port:         <none>
         Host Port:    <none>
         Environment:  <none>
         Mounts:       <none>
       Volumes:        <none>
     Events:
       Type     Reason             Age   From               Message
       ----     ------             ----  ----               -------
       Warning  ApplyPolicyFailed  106s  resource-detector  No policy match for resource

    As you can see, there isn't a policy to match resources. So Karmada doesn't know what to do with the deployments.

  5. You will create a policy to assign a replica to each cluster and allocate workloads to clusters.

     # nano policy.yaml

    Add the following configuration to define an equal weight for each cluster.

     apiVersion: policy.karmada.io/v1alpha1
     kind: PropagationPolicy
     metadata:
      name: hello-propagation
     spec:
      resourceSelectors:
        - apiVersion: apps/v1
          kind: Deployment
          name: hello
        - apiVersion: v1
          kind: Service
          name: hello
      placement:
        clusterAffinity:
          clusterNames:
            - mumbai
            - paris
            - newyork
        replicaScheduling:
          replicaDivisionPreference: Weighted
          replicaSchedulingType: Divided
          weightPreference:
            staticWeightList:
              - targetCluster:
                  clusterNames:
                    - newyork
                weight: 1
              - targetCluster:
                  clusterNames:
                    - paris
                weight: 1
              - targetCluster:
                  clusterNames:
                    - mumbai
                weight: 1
  6. Apply the policy to the Karmada cluster.

     # kubectl apply -f policy.yaml --kubeconfig=karmada-config

    Sample output:

     propagationpolicy.policy.karmada.io/hello-propagation created
  7. Verify whether the Karmada assigned a pod to each cluster using the following commands.

     # kubectl get deployments --kubeconfig=karmada-config

    Sample output:

     NAME    READY   UP-TO-DATE   AVAILABLE   AGE
     hello   3/3     3            3           6m20s

    To verify the Mumbai cluster, run:

     # kubectl get pods --kubeconfig=kubeconfig-mumbai

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-xlw5b   1/1     Running   0          75s

    To verify the Paris cluster, run:

     # kubectl get pods --kubeconfig=kubeconfig-paris

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-qntt5   1/1     Running   0          90s

    To verify the New York cluster, run:

     # kubectl get pods --kubeconfig=kubeconfig-newyork

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-d9pb8   1/1     Running   0          104s

    As you can see, Karmada assigned an equal number of pods to each cluster.

  8. Next, run the following command to scale the deployment to 10 replicas.

     # kubectl scale deployment/hello --replicas=10 --kubeconfig=karmada-config
  9. Verify the deployment.

     # kubectl get deployments --kubeconfig=karmada-config

    Sample output:

     NAME    READY   UP-TO-DATE   AVAILABLE   AGE
     hello   10/10   10           10          12m
  10. Verify the age of each cluster.

    For the Mumbai cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-mumbai

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-2xtlb   1/1     Running   0          83s
     hello-588ddff5d8-65qdx   1/1     Running   0          83s
     hello-588ddff5d8-p4gdr   1/1     Running   0          83s
     hello-588ddff5d8-xlw5b   1/1     Running   0          7m32s

    For the Paris cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-paris

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-bc7ct   1/1     Running   0          97s
     hello-588ddff5d8-qntt5   1/1     Running   0          7m46s
     hello-588ddff5d8-vkcf8   1/1     Running   0          97s

    For the New York cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-newyork

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-d9pb8   1/1     Running   0          8m1s
     hello-588ddff5d8-fm7df   1/1     Running   0          112s
     hello-588ddff5d8-lxk2j   1/1     Running   0          112s
  11. Now, edit the policy.yaml file so that the Mumbai and Paris clusters hold 40% of the pods and only 20% is left to the New York cluster.

     # nano policy.yaml

    Change the configuration as shown below.

     apiVersion: policy.karmada.io/v1alpha1
     kind: PropagationPolicy
     metadata:
      name: hello-propagation
     spec:
      resourceSelectors:
        - apiVersion: apps/v1
          kind: Deployment
          name: hello
        - apiVersion: v1
          kind: Service
          name: hello
      placement:
        clusterAffinity:
          clusterNames:
            - mumbai
            - paris
            - newyork
        replicaScheduling:
          replicaDivisionPreference: Weighted
          replicaSchedulingType: Divided
          weightPreference:
            staticWeightList:
              - targetCluster:
                  clusterNames:
                    - newyork
                weight: 1
              - targetCluster:
                  clusterNames:
                    - paris
                weight: 2
              - targetCluster:
                  clusterNames:
                    - mumbai
                weight: 2
  12. Apply the policy to the Karmada cluster.

     # kubectl apply -f policy.yaml --kubeconfig=karmada-config
  13. Verify the pod distribution again using the following command.

    For the Mumbai cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-mumbai

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-2xtlb   1/1     Running   0          4m14s
     hello-588ddff5d8-65qdx   1/1     Running   0          4m14s
     hello-588ddff5d8-p4gdr   1/1     Running   0          4m14s
     hello-588ddff5d8-xlw5b   1/1     Running   0          10m

    For the Paris cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-paris

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-qntt5   1/1     Running   0          10m
     hello-588ddff5d8-vkcf8   1/1     Running   0          4m30s

    For the New York cluster, run the following:

     # kubectl get pods --kubeconfig=kubeconfig-newyork

    Sample output:

     NAME                     READY   STATUS    RESTARTS   AGE
     hello-588ddff5d8-d9pb8   1/1     Running   0          10m
     hello-588ddff5d8-fm7df   1/1     Running   0          4m42s
     hello-588ddff5d8-lxk2j   1/1     Running   0          4m42s
     hello-588ddff5d8-vd9lv   1/1     Running   0          89s
  14. At this point, pods are running and distributed across three clusters. You can inspect the service in Karmada using the following command.

     # kubectl describe service hello --kubeconfig=karmada-config

    Sample output:

     Name:              hello
     Namespace:         default
     Labels:            propagationpolicy.karmada.io/name=hello-propagation
                        propagationpolicy.karmada.io/namespace=default
     Annotations:       <none>
     Selector:          app=hello
     Type:              ClusterIP
     IP Family Policy:  SingleStack
     IP Families:       IPv4
     IP:                10.101.68.149
     IPs:               10.101.68.149
     Port:              <unset>  5000/TCP
     TargetPort:        9898/TCP
     Endpoints:         <none>
     Session Affinity:  None
     Events:
       Type     Reason                  Age                  From                  Message
       ----     ------                  ----                 ----                  -------
       Warning  ApplyPolicyFailed       16m                  resource-detector     No policy match for resource
       Normal   ApplyPolicySucceed      11m                  resource-detector     Apply policy(default/hello-propagation) succeed
       Normal   SyncSucceed             11m                  execution-controller  Successfully applied resource(default/hello) to cluster mumbai
       Normal   SyncSucceed             11m                  execution-controller  Successfully applied resource(default/hello) to cluster newyork
       Normal   SyncSucceed             11m                  execution-controller  Successfully applied resource(default/hello) to cluster paris
       Normal   SyncWorkSucceed         2m4s (x9 over 11m)   binding-controller    Sync work of resourceBinding(default/hello-service) successful.
       Normal   AggregateStatusSucceed  2m4s (x9 over 11m)   binding-controller    Update resourceBinding(default/hello-service) with AggregatedStatus successfully.
       Normal   ScheduleBindingSucceed  2m4s (x13 over 11m)  karmada-scheduler     Binding has been scheduled

Conclusion

You've finished deploying multi-location, multi-cluster Kubernetes Orchestration with Karmada. You can now use Karmada to automate multi-cluster application management in multi-cloud and hybrid-cloud scenarios. For more information, check out the Karmada official documentation.