Skip to content

I want to run tests in parallel - how do I do this ?

It is a typical requirement to shorten the feedback cycle by running tests in parallel.

What are Botium Box Agents ?

The Botium Box Agents are the work-horses, which actually run the test cases (see Architecture & Requirements for details on the Botium Box architecture)

The number of agents depends on the Botium Box flavour.

  • With the free Botium Box Mini plan, there is a pool of agents available, shared by all users

  • With the paid plans, there are dedicated agents with allow parallelization

Every agent will run exactly one test case at a time.

Parallel Test Execution

What’s happening behind the scenes is:

  • There is a queue with test cases to process

  • Every agent processes the test cases it receives from the queue as quickly as possible

    • As soon as an agent is ready with a test case, it immediately pulls the next one from the queue

As an example, let’s consider a scenario where 5 Botium Box Agents are available, and you have two test sets, BIGTESTSET with 1000 test cases, and SMALLTESTSET with 100 test cases.

By default, without any further configuration, the behaviour in Botium Box is to equally distribute the load onto all available agents by creating batches of test cases. So let’s say you start to tests in BIGTESTSET, and right afterwards in SMALLTESTSET.

  • The queue is filled with five batches of 200 test cases out of the 1000 ones from BIGTESTSET

  • The workers begin to process the test cases - each worker takes one of the five batches

  • In addition, the queue is filled with five batches of 20 test cases out of the 100 from SMALLTESTSET

The outcome in this scenario is:

  • All agents are first working on test cases from BIGTESTSET in parallel

  • Afterwards, the test cases from SMALLTESTSET are processed

So, although there is parallel execution, you have to wait for the BIGTESTSET to finish until the test cases from SMALLTESTSET are processed.

Parallel Jobs Count

To prevent a scenario like above, it is possible to define the degree of parallelity. In the Test Project Settings section, under Test Execution, you can set the Parallel Jobs Count field.

This number gives the number of batches that Botium Box will push to the queue for the test set.

Let’s extend the scenario from above - for BIGTESTSET, the Parallel Jobs Count is set to 3, for the SMALLTESTSET, it is set to 1.

  • When launching BIGTESTSET, there are three batches created (334/334/332)

  • Three of the agents begin to process it, two agents are still waiting for something to do

  • For SMALLTESTSET, there are two batches created (50/50)

  • The two idle agents are processing them

So now the test cases from two different test sessions are processed in parallel.

Summary

If there is only one Botium Box Agent available, there is no parallel test execution available.

If there are more Botium Box Agents available, by default the test cases for a test session are equally distributed for parallel execution.

If you want to run test cases from multiple test sessions in parallel, you can specify the degree of parallelity with the Parallel Jobs Count field.