Skip to content

Get Started with Botium Box

Follow the steps in this tutorial to become a Botium Pro and get your first Botium Box test project up and running!

Concepts and Terminology

Before starting your Botium Box journey, it is important to understand the concepts and the terminology that is used in Botium Box and how they relate to each other.

  • Chatbots
  • Test Sets
  • Test Projects
  • Test Results

image

Chatbots

A Chatbot in Botium Box is a communication channel between Botium Box and your chatbot engine. By establishing this communication channel you enable Botium Box to talk to your chatbot and listen for responses.

A communication channel can be opened in many different ways, depending on your technology. In Botium, a technology for establishing a communication channel to your chatbot is called Botium Connector, and they are available for all relevant technologies out there, such as IBM Watson, Google Dialogflow, Amazon Lex or Microsoft LUIS

Hint

Have a look at the Botium Connectors list for all currently supported technologies.

Test Sets

Test Sets contain the data, scripts and other artifacts that represent your test cases. You can think of a test set as a folder where your test data lives.

Test Projects

You actually run the test cases by registering a Test Project and hitting the Start button. Within a Test Project you tell Botium Box:

  • which test cases you want to run
  • on which chatbot channel
  • when you want to run them
  • and what happens with the test results

So a Test Project is composed from:

  • a chatbot to define the communication channel
  • one or more test sets, which are themselves containing the test cases
  • configuration what aspects you want to test (test features)
  • a preconfigured webhook to be integrated into your CI/CD pipeline
  • and a historic view of the finished test results and test cases

Test Results

A Test Result is the outcome of a single test execution, usually by starting test execution in a test project. You can see the total results as well as drill down to the single test cases and conversation steps causing troubles.

So now that you know about the concepts, let's begin your Botium Box journey.

Step 1: Connect Botium Box to Your Chatbot

The first step is to connect Botium Box to your chatbot. From the Dashboard, open the Chatbots menu.

Enter the Register a new chatbot section.

Have a look at the Connector/Chatbot Technology selection, here you can see what chatbot technologies are supported by Botium Box. You can get an overview and more details for each technology here.

  • Enter First Steps Chatbot in the Chatbot Name field
  • Select to Botium Sample Chatbot (Echo) - a demo chatbot you can use for exploring Botium Box features.

Attention

While connecting to this demo chatbot doesn’t require further configuration, there are depending on the selected technology additional things to configure, like endpoint addresses, usernames, secrets, …

  • Click the Say Hello (Check Connectivity) button to check availability of the chatbot. Botium will send a Hello to the chatbot and expect some kind of welcome response back.

Attention

As this is a demo chatbot, it is available of course, but in other cases you can assert the technical correctness of the configuration.

  • Click the Close button
  • Click Save to save the new chatbot record
  • You can now navigate to the record by using the filter in the Chatbots list and clicking on the record:

Step 2: Using the Live Chat to Create a First Test Set

In Botium, a test set is a collection of test cases. A test case is a single, multi-step conversation with your chatbot, which should respond in a way we expect it.

Attention

See Get Started with Testing Conversational AI to get more background information, but you don’t need it to finish this tutorial.

An intuitive way for creating a test case is by

  1. having an actual chat with your chatbot and
  2. saving this conversation transcript to be repeated in exactly the same way later

That’s what the live chat in Botium Box is for - click the Live Chat button to open a chat session with your chatbot:

  • Enter hi in the input field at the bottom and click the send button (or Enter)

  • You can now see the chatbot responding to your greeting
  • Click on the Save as Test Case button to save the conversation you just had as test case

  • Let’s create a new test set by entering First Steps Test Case in the input field …
  • … and First Steps Test Case as Test Case Name
  • … and finally click on OK

  • Now navigate to the test set record we just created by using the Test Sets menu and clicking on the First Steps Test Set in the list - again, you can use the filter on the top right.

  • Click on the Test Cases section to inspect the test case we just created

  • You can see now a single test case, the one we saved before. You can use the test case designer for inspecting and changing the conversation steps.

For software developers it might be interesting that all Botium test cases are actually backed by BotiumScript, a scripting language designed for defining test cases for conversational interfaces. The BotiumScript for our first test case looks like this:

First Steps Test Case

#me
hi

#bot
You said: hi


You can read it like:

  • The name of the test case is First Steps Test Case
  • As a starter, send the text hi to the chatbot
  • We expect the chatbot to answer with the text You said: hi - if the answer from the chatbot is different, the chatbot doesn’t behave as expected, and the test case is considered failed

Step 3: Run a First Test Session

So, we are connected to the chatbot and we recorded a first test case. Now let’s run this test case.

  • Go to the Test Results menu and in the Start Test Session section, select our First Steps Chatbot and our First Steps Test Set, and click on Start Test Session.

  • You can now see Botium doing the hard work in the background, and after several seconds, the first test result is ready.
  • As expected, the test session succeeded
  • In the Test Cases section, you can see all the executed test cases - for our first test set, we only had one single test case.

  • Click on the test case to open the conversation transcript. This looks very similar to the live chat we had earlier - but the conversation happened without any manual intervention.

Step 4: Use the Test Case Designer to Extend the First Test Case

Our test case covered exactly one possible greeting a user can send - hi - nothing else. Now let’s extend this test case with other possible greetings a real user might send to your chatbot like hello and good morning.

In Botium, these synonyms are called utterance lists. Navigate back to your First Steps Test Set. Instead of using the main menu you can also use the Recently Visited list in the user menu.

  • Open the Test Cases section and click the Write New Utterances List button

  • Enter UTT_USER_HELLO as utterance list name
  • Enter the utterances hi, hello and good morning (use the + button to add an additional input field)
  • Click the Save button to save the utterance list

Now that our utterance list is ready we have to change the First Steps Test Case to send all three greetings to the chatbot instead of only the one we used in our live chat session before.

  • Click on the bubble with text hi in the First Steps Test Case to bring up the conversation step editor

  • Click on the Select From Utterance List button to bring up the selection box for the utterances lists
  • Click on UTT_USER_HELLO to use this utterance list as input for the test case
  • Close the conversation step editor

  • Note that the test case now has been changed: where before there was only the hi as input, now there is the UTT_USER_HELLO utterance list we created before
  • Save the test case and let’s run another test session the same way as above
    • Go to the Test Results menu and in the Start Test Session section, select our First Steps Chatbot and our First Steps Test Set, and click on Start Test Session.

  • See what happened now ? There are now 3 test cases, and 2 of them failed!
  • Click on Test Cases and expand one of the two failing test cases

  • You can see here the reason the test case failed:
    • We are sending the text hello to the chatbot (one of the possible greetings in our UTT_USER_HELLO utterance list)
    • The chatbot, as it echoes everything we tell him, is responding with You said: hello
    • But in our test case, we are expecting the response You said: hi from our initial live chat conversation with the chatbot
    • So two of those three test cases are failing

  • Let’s fix this test case failure by navigating back to the First Step Test Set, selecting the First Step Test Case and clicking on the You said: hi conversation step

  • Replace the text You said: hi with You said: *
  • Close the conversation step editor and save the test case

  • Run again the test session and you will see that now all three test cases are passing successfully

You noticed that it requires some clicks to start a test session, event with the same chatbot and test set. So before you actually start writing more test cases, let’s create a new test project to make this easier.

Step 5: Use the Quickstart Wizard to Create a Test Project

In the previous step, you had to select the chatbot, select the test set and only then Botium was able to start the test session. A test project is a permanent link between a chatbot and one or more test sets in Botium Box.

  • Let’s launch it in the Start menu by clicking the Let’s Go button in the Regression Testing section.

  • Enter a name for the test project and select the First Steps Chatbot, afterwards proceed to the next screen

  • In the second screen, click on Select from registered Test Sets, as we already created a test set record in the previous tutorial steps
  • Select the First Steps Test Set, and proceed to the final screen

  • On the final screen, click the Save Test Project and Start Testing button to
    • create a test project record …
    • … and start the first test session

  • When the test session is ready, you can navigate to the test project in the Test Projects menu
  • Note that in the test projects list, you can already see the result of the latest test session for this test project

  • You can now start a new test session by a single click on the Start Test Session Now button

  • You can now even see the history of all the previous test sessions for this test project in the Test Case History section (screenshot is from another test project that has more progress than the First Steps Test Project)

Conclusion and Next Steps

In this tutorial you learned basic usage of most important Botium Box features:

  • Connecting to your chatbot
  • Creating test cases
  • Running test sessions
  • View the result of test sessions