When running large test sets, Botium Box gets slower and slower
Most likely it is one of the following reasons.
API Rate Limits
When using an API for chatbot testing, for example Dialogflow or Watson, there are typically rate limits on the number of possible API calls. It makes sense to apply Botium Rate Limiting to not overflood the API.
Persistance Layer Memory
Under heavy load, the persistence technology included in Botium Box, Prisma, is using lots of memory. You can find the amount of memory used by running docker stats:
> docker stats botiumbox_prisma_1
Output is the docker memory limit and the currently consumed memory.
This shows that docker allows up to near 4GB of memory, but Prisma is using just 1GB of memory.
By default, Prisma is using up to 1GB of memory, which is too less when under heavy load. Open the docker-compose.yml file and add a JAVA_OPTS environment variable:
... prisma: ... environment: JAVA_OPTS: "-Xmx4G" PRISMA_CONFIG: | ...
In this case, Prisma will get the full 4GB amount of memory which is in this case the docker memory limit. To increase the docker limit, consult the docker documention.