Skip to content

Architecture & Requirements

Botium Box Components

Botium Box consists of several compontents:

  • The Botium Box User Interface is a browser-based application
  • The Botium Box Agent(s) are the work-horses, which actually run the test cases. Usually, there are several Botium Agents active for parallel execution of test cases
  • The Botium Box Server is the central instance connecting the Agents and the User Interface

Required 3rd party components:

  • Recommended: NGINX as frontend webserver (upgrade to load balancer and failover)
  • Prisma as data storage engine, which is itself backed by either MySQL or PostgresQL
  • Redis for reliable messaging, task scheduling and session storage
  • Recommended: some kind of continuous integration service such as Jenkins or Bamboo

Note

All of the 3rd party components (except continuous integration) are part of the distribution of Botium Box, with a basic working default configuration.


Server Hardware Requirements

It is not possible to talk about the hardware requirements without any knowledge about

  • how many test projects you plan
  • how many test cases you plan
  • how many test runs you plan

Furthermore, there are several deployment options, from a standalone, single server deployment to a cluster or swarm of countless machines connected to the Botium Box.

For a minimum standalone, single server deployment (suitable for a PoC or Demo environment), you are usually safe with this configuration (which corresponds to an Amazon t2/t3/t3a.medium instance):

  • 2 CPU (dual-core)
  • 4 GB RAM
  • 100GB HDD

For a medium-sized single server production deployment, we recommend at least (corresponding to an Amazon t2/t3/t3a.xlarge instance):

  • 4 CPU (or quad-core)
  • 8 GB RAM
  • 200 GB HDD

Resource Requirements Per Component (Pod)

When installing Botium Box on a Kubernetes cluster or in a distributed Docker environment, here are the the minimum resource requirements for each component (multiplied by the number of instances). The recommended resource allocation is double the size.

  • Botium Box Server
    • 1 CPU core
    • 1 GB RAM
  • Botium Box Agent
    • 0,5 CPU cores
    • 1 GB RAM
  • (optional) Botium Coach Worker
    • not a required component, used for test set insights
    • 1 CPU core
    • 4 GB RAM
  • Prisma
    • 1 CPU core
    • 2 GB RAM
  • Redis
    • 0,5 CPU cores
    • 0,5 GB RAM

Additional Resource Requirements - Test Set Insights

When calculating advanced Test Set Insights (like cohesion/separation/similarity charts), you have to enable the additional Coach Worker component.

  • Coach Worker
    • 1 CPU core
    • 4 GB RAM

Additional Resource Requirements - Voice Testing

When using Voice Testing you have to enable the additional Botium Speech Processing component. Depending on the speech backend you are using the hardware requirements are different - when connected to a cloud service like Google Speech, the local resource requirements are of course way lower than when using Kaldi and MaryTTS locally.

  • Botium Speech Processing with cloud service like Google
    • 1 CPU core
    • 1 GB RAM
  • Botium Speech Processing with Kaldi and MaryTTS
    • 2 CPU cores
    • 8 GB RAM

Additional Resource Requirements - Performance Testing

For generating conversation load on the chatbot, there is CPU power required on Botium side. Our experiments show that as rule of thumb Botium will be able to simulate \~300 parallel users per CPU core. Following are the results of our experiments.

For the simulation of \~ 500 parallel users per second:

  • 2 Botium Box Agents
    • 1 CPU core each
  • Increase Redis resources (for Kubernetes installation)
    • 1 CPU core
    • 2 GB RAM
  • Sample configuration
    • 2 CPU cores
    • 8GB RAM
    • \~AWS t2/t3/t3a.large instance type

For the simulation \~1000 parallel users per second:

  • 4 Botium Box Agents
    • 1 CPU core each
  • Sample configuration
    • 4 CPU cores
    • 16GB RAM
    • \~AWS t2/t3/t3a.xlarge instance type

For the simulation \~2000 parallel users per second:

  • 8 Botium Box Agents
    • 1 CPU core each
  • Sample configuration
    • 8 CPU cores
    • 16GB RAM
    • \~AWS t3a.2xlarge instance type

Note

Parallel in this case actually is not full parallelity, but has to be seen as a total amount of requests within a given time frame (100 user requests per second for example)


Server Software Requirements - Docker

All Botium Box components are distributed as Docker containers, so you need a working Docker installation on all servers.

Check Docker

Run this command to verify that your Docker installation is actually working:

docker run hello-world


Client Requirements

Any current desktop browser will do. We support Internet Explorer, Edge, Chrome, Firefox, Safari.

Currently not supported: Using Botium Box with a Smartphone.

Networking Requirements

There are several firewall rules to be opened for running the Botium Box. Shell access (usually SSH, Port 22) is required.

Github

On first installation you have to download the Botium Box repository containing the installation files (Docker-Compose, Kubernetes) from Github. Github doesn’t have a fixed list of IP addresses, so either open up the firewall for the first download, or upload the Github files manually to the server.

Docker Hub

During installation or update of Botium Box, Docker will pull the required images from Docker Hub. As there is no fixed set of IP addresses or hostnames available for whitelisting Docker Hub, it is recommended either to temporarily open the firewall or copying the required Docker images to the server manually for Offline installation.

Botium Box Server

Botium Box listens on the default ports 80 (for HTTP - disabled by default) and 443 (for HTTPS - enabled by default) if NGinx is used, and port 4000 (HTTP) otherwise. Clients accessing the Botium Box, as well as any connected CI/CD products have to get access to these ports.

Prisma (multi server deployments only)

Please enable connectivity from the Botium Box Server to the Prisma Server on port 4466.

Redis (multi server deployments only)

Please enable connectivity from the Botium Box Server and the Botium Box Agents to the Redis Server on port 6379.

Other ports

You have to make sure that Botium Box Server and Botium Box Agents connectivity to the chatbot under test is available. This depends on the used connector. If you don’t know, we are happy to consult.

When using LDAP authentification, you have to open the port to your LDAP server (usually 389).

Server Software Requirements - Kubernetes/Openshift

See Installation to Kubernetes Cluster / Installation to Openshift

Security Requirements

We recommend to generate and use an SSL certificate for serving the Botium Box User Interface. It should match the hostname of the machine the Botium Box Server is running on.

Note

By default, a self-signed certificate for server 127.0.0.1 is included, which at least will cause security warnings in our browser.