Skip to content

Installation to Kubernetes Cluster

You can run Botium Box on your Kubernetes environment, locally in Minikube, or on a cloud server as Amazon Elastic Kubernetes Service.

Persistent data storage is not part of the Kubernetes definitions in this repository. You can install MySQL in your Kubernetes environment as well. All major cloud providers support hosted MySQL installations, for example

Prerequisites

  • A working Kubernetes environment

  • kubectl connected to your Kubernetes environment

  • A MySQL-compatible database, installed locally or hosted in the cloud

    • for example, MySQL 5.7 or later
  • MySQL connectivity (check Firewalls etc)

    • MySQL hostname

    • MySQL username

    • MySQL password

Download this Git repository to your local workstation, and enter botium-box-premium-dist

git clone https://github.com/codeforequity-at/botium-box-premium-dist.git
cd botium-box-premium-dist

If you are using online Azure CLI (bash, not PowerShell), then you have to do this, and every other steps there!

Note

There is a valueable kubectl cheat sheet in the Kubernetes docs about the most important Kubernetes commands!

Pre-Installation Steps

Depending on your Kubernetes environment, you will have to do some preparation steps.

Copy the file kubernetes/02-configmap.yml.template to kubernetes/02-configmap.yml.

First create the namespace:

$ kubectl apply -f kubernetes/01-namespace.yml

Preparation Steps on Amazon AWS

Preparing Amazon Elastic Kubernetes Service (EKS)

You can create a Kubernetes cluster for Botium Box with a single command - this one creates a simple one-node Kubernetes cluster:

$ eksctl create cluster --name botium-box --version 1.14 --nodegroup-name standard-workers --node-type t3.medium --nodes 1 --nodes-min 1 --nodes-max 1 --node-ami auto

Cluster provisioning usually takes between 10 and 15 minutes.  When your cluster is ready, test that your kubectl configuration is correct.

kubectl get svc

Follow the Amazon documentation for initializing your Kubernetes environment and linking it to locale kubectl on your local workstation.

Deleting of a whole Kubernetes cluster is a one-liner as well:

$ eksctl delete cluster --name botium-box

Preparing Amazon EFS Storage

Botium Box requires a persistent volume for storing resources and test sets. Amazon provides the EFS container storage, which you have to activate in your Kubernetes cluster - see the instructions here. In short, most likely running this command is all you have to do:

$ kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

Afterwards, create an EFS file system (execute commands below To create an Amazon EFS file system for your Amazon EKS cluster) and make note of the Amazon EFS file system ID (in the form fs-blablabla). Copy the file kubernetes/aws-eks/01-storage.yml.template to kubernetes/aws-eks/01-storage.yml and add the Amazon EFS file system ID for the volumeHandle placeholder.

Run this command to deploy your storage class and persistent volume in the Kubernetes cluster:

$ kubectl apply -f kubernetes/aws-eks/

Preparing Amazon RDS for MySQL

Follow the instructions in the Amazon documentation. Make sure to enable the Public accessibility option.

Enter the MySQL endpoint, the user and the password in the file kubernetes/02-configmap.yml.

Preparing Kubernetes web dashboard (optional)

Optionally you can use Kubernetes web dashboard for basic management operations. Steps are described here.

Preparation Steps on Microsoft Azure

You can execute this steps with online or local Azure CLI. Online has the required tools preinstalled.

Online Azure CLI supports Bash and PowerShell. This steps are tested just with Bash.

You can edit files simple using online Azure CLI:

code kubernetes/02-configmap.yml

Preparing Azure Kubernetes Service (AKS) with Azure CLI

Use the (online, or local) to create a Kubernetes cluster - this command will create a simple one-node Kubernetes cluster and connect your kubectl to it:

$ az aks create --resource-group BotiumResourceGroup --name botium-box --node-count 1 --generate-ssh-keys
$ az aks get-credentials --resource-group BotiumResourceGroup --name botium-box

Deleting of a whole Kubernetes cluster is a one-liner as well:

$ az aks delete --name botium-box --resource-group BotiumResourceGroup

Preparing Azure File Storage

Run this command to deploy a suitable storage class in the Kubernetes cluster:

$ kubectl apply -f kubernetes/azure/

Preparing Azure Database for MySQL

Follow the instructions in the Multi-Container-App-Tutorial to create a MySQL database and enter the MySQL endpoint, the user and the password in the file kubernetes/02-configmap.yml.

Preparing Kubernetes web dashboard (optional)

Optionally you can use Kubernetes web dashboard for basic management operations.

ClusterRoleBinding must be created once before you can correctly access the dashboard:

kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

With browse command you can create a temporary tunnel:

az aks browse --resource-group BotiumResourceGroup --name botium-box

This command returns an URL to the Dashboard.

Accessing to the dasboard is possible just for the same Azure user who created it.

Preparation Steps on Minikube

Create a local MySQL database in Minikube:

$ kubectl apply -f kubernetes/minikube/

PRISMA_CONFIG in kubernetes/02-configmap.yml has to look like this:

  PRISMA_CONFIG: |
    port: 4466
    databases:
      default:
        connector: mysql
        host: mysql
        port: 3306
        user: root
        password: password
        migrations: true
        ssl: false

Kubernetes Dashboard

It is included in Minikube:

$ minikube dashboard

License Key

Add your Botium Box license key to kubernetes/02-configmap.yml

Deploy Botium Box to Kubernetes cluster

After all the preparation, Botium Box deployment is only one command:

$ kubectl apply -f kubernetes

Whats happening here ?

  • A Kubernetes namespace botium-box is created to include all Botium Box components

  • A Config Map is uploaded

  • A Prisma service is started and connected to your MySQL

  • A Redis service is started

  • A ZAP service is started

  • A Persistent Volume Claim is created for Botium Box persistent storage

  • A Botium Box Server is started

  • A Botium Box Agent is started

  • The resources directory is initialized

This takes some time for downloading docker images, initializing storage etc.

You can use this command to check the deployment status:

$ kubectl get pods -n botium-box

The init pod is in state Completed, all other pods should be in state Running.

Viewing

Botium Box Agent Configuration

If your license includes more than one Botium Box Agent, you can increase the replicas count in the file kubernetes/06-box.yml (for the box-agent StatefulSet) and run the kubectl apply -f kubernetes command again.

Or you can use the kubectl scale command to increase the Botium Box Agent count:

kubectl scale --replicas=5 statefulset box-agent -n botium-box

Using Botium Box

Use this command to show the IP Address or URL of your Botium Box:

kubectl get svc -n botium-box

The column External-IP for the box-server LoadBalancer shows the IP Address or URL to copy & paste to your Browser to launch Botium Box:

It can take 10-20 minute to start. Even if there is an IP instead of <pending> label, be patient.

Minikube

Opening Box in default browser:

$ minikube service box-server -n botium-box

Updating Botium Box

When there are Botium Box updates, there are new Docker images available or maybe additional configuration parameters in the Kubernetes files.

First, pull the newest changes from the Git repository to your local workstation.

Run this command to apply the changes to the Kubernetes cluster:

$ kubectl apply -f kubernetes

Depending on your Kubernetes configuration, new Docker images are not pulled automatically, so you have to force Kubernetes to get the latest Botium Box images. One way to do this is to remove the Botium Box pods and let Kubernetes create them again by first scaling down and afterwards scaling up again:

$ kubectl scale --replicas=0 deployment box-server -n botium-box
$ kubectl scale --replicas=1 deployment box-server -n botium-box
$ kubectl scale --replicas=0 statefulset box-agent -n botium-box
$ kubectl scale --replicas=5 statefulset box-agent -n botium-box -n botium-box #take correct number of Botium Box Agents here
$ kubectl scale --replicas=0 deployment prisma -n botium-box
$ kubectl scale --replicas=1 deployment prisma -n botium-box
$ kubectl scale --replicas=0 deployment redis -n botium-box
$ kubectl scale --replicas=1 deployment redis -n botium-box
$ kubectl scale --replicas=0 deployment zap -n botium-box
$ kubectl scale --replicas=1 deployment zap -n botium-box

Viewing Log Output

It’s always important to know how to view the log output of Botium Box services - either for yourself, or for us to support you. First, geht the list of pods:

$ kubectl get pods -n botium-box

And then get the log output (replace the botium-box-xxxxxxxxx placeholder with the actual name from the above output):

$ kubtectl logs botium-box-xxxxxxxxx -n botium-box