Skip to content

E2E-Testing with Botium Box and Selenium/Appium

This article shows how to set up Botium Box as an End-to-end-testing environment connected to a device lab or browser lab.

Your chatbot needs testing, you know that — otherwise you wouldn’t read this article. Apart from the conversation flow, which is best tested on API level (see Appendices for a list of supported APIs and SDKs), you want to make sure that your chatbot published on a company website works on most used end user devices out there. Botium Box helps by connecting your BotiumScript test cases with your chatbot widget using a Selenium server, the current industry standard for testing web applications.

Step 1: Connect to Device Lab or Browser Lab

Botium Box supports cloud providers, but you can connect to your self-hosted Selenium server as well.

  • Saucelabs.com cross browser and real device cloud
  • Perfecto.io cloud
  • Experitest Mobile App & Cross-Browser Testing End-to-End
  • Local, self-hosted Selenium server
  • Headless Chrome

image.png

The Selenium Endpoint has to be filled with URL of the Selenium server. For cloud providers, consult the documentation or contact the cloud provider support. You will have to enter username and password or access key as well. For local installations, consult your developers.

  • For Saucelabs, the Selenium Endpoint typically is https://ondemand.saucelabs.com/wd/hub, the Username and the Access Key you can find on the User Settings section in the Saucelabs dashboard
  • For Saucelabs Real Device Cloud (former TestObjects), you have to create a Test App first, switch to the Appium view and go to the Setup Instructions — you will find the Selenium Endpoint and the API Key there
  • For Experitest, you can find the Selenium Endpoint (typically https://cloud.seetest.io/wd/hub) and the Access Key in the Seetest Dashboard in the Automation view

See here for a guide how to setup your own Selenium Grid and connect it to Botium Box.

Step 2: Compose Device Sets

Next Step is to select the devices for running your test cases. Botium Box comes preconfigured with reasonable default device sets. There are several guidelines available what devices you should select for your use case.

image.png

For cloud providers, the available devices and browsers are shown for selection.

image.png

  • For Saucelabs and Perfecto, the device list is retrieved in real-time from the Saucelabs API
  • For Experitest, the file EXPERITEST.json in the resources folder is used to show the selection of the available devices and browers
  • For a local selenium grid, the file LOCALSELENIUM.json in the resources folder is used to show the selection of available devices and browsers

See here for a guide how to add the available browser configurations for your local Selenium Grid to Botium Box.

Step 3: Prepare Botium Webdriver Connector

To finish the configuration process, you now have to tell Botium Box how to connect to your chatbot. Make sure to select WebdriverIO as chatbot technology.

image.png

If you ever worked with Selenium, you are aware that writing an automation script usually is a time-consuming task. Botium Box speeds up the development process by making reasonable assumptions:

  • There maybe is some kind of click-through to actually open the chatbot
  • The chatbot has an input text field, or some buttons to click (however it looks)
  • The chatbot outputs reponses in some kind of list within a window (however it looks)

All those assumptions can be parameterized for adapting it to your actual chatbot website.

image.png

You can instruct Botium Box to take a screenshot of every test case on each device to get an overview of the visual appearance of the chatbot widget when running your test cases.

Configuration for Common Chatbot Widgets

For some common chatbot widgets and websites, Botium provides out-of-the-box Selenium scripts (Microsoft Bot Framework WebchatIBM Watson Assistant, and more to come …) — Botium will be able to detect the widget on the website automatically, based on Selenium selectors. Select the technology in the Webdriver Script field.

image.png

Custom Configuration with Selenium selectors

With some know-how with Selenium Botium can be adapted to almost every webchat out there. The first thing to do is to analyze your chatbot widget to find out about the Selenium selectors to use — here is a guide what to look out for. The common Selenium flow in Botium is:

  1. Open the Start Url as given in the Botium configuration
  2. Do some clicks to open the chatbot window (if it is not opened automatically)
  3. Use the Input Element Selector to send some text to the chatbot
  4. Use the Output Element Selector to detect the responses of the chatbot

For rather simple chatbot widgets, the input and output selectors can be entered in the respective fields in the Botium configuration screen. If there are dynamic actions required to open the chatbot window (Point 2 from above), you have to do some Javascript coding and register the scripts in Botium Box.

Custom Configuration with Javascript Snippets

There are several extension points where you can inject your own Javascript snippets. In most cases, the extension point you will have to adapt is the WEBDRIVERIO_OPENBOT extension point: it is responsible to open the chatbot widget and make it ready for Botium to start the conversation.

Here is an example:

  • it waits for the cookie banner and clicks it away
  • it waits for a chatbot button to appear (#StartChat) and clicks it
module.exports = async (container, browser) => {
  const ccBtn = await browser.$('#onetrust-accept-btn-handler')
  await ccBtn.waitForClickable({ timeout: 20000 })
  await ccBtn.click()  const startChat = await browser.$('#StartChat')
  await startChat.waitForClickable({ timeout: 20000 })
  await startChat.click()
}

You can find more samples in the Github repository for the Botium connector.

Adding your Javascript Snippets to Botium Box

Place the snippets in the corresponding section of the connector settings.

image.png

Step 4: Run Test Cases

When running test cases, you have to select the device set(s) to use.Apart from that, it behaves as any other test project in Botium Box. Click on Save and Start Testing to start the first test run.

Step 5: View Test Report

The generated test report shows every single detail of the test run, including the transcript, the expected outcome and, probably one of the most interesting information items, the screenshot(s) taken by Botium Box.

image.png

Frequently Asked Questions

Why should I test my chatbot user interface ? Is it not enough to test the conversation flow on API level ?

You should at least make sure to verify the visual appearance and the correct operation of your chatbot user interface on most used desktop and mobile device browsers out there. Botium Box can support you in this.

If you already have Selenium scripts for testing your website including the chatbot widget on different browsers, then this already seems to be a good setup, and it most likely won’t make sense to duplicate the tests in Botium Box.

What is the difference to just using Selenium ?

Selenium is just an API connecting custom automation code to desktop or mobile device browsers. It is the technology backing the Botium Box Webdriver connector. Botium Box adds device management, test case management, test reporting, load balancing and much more to plain Selenium.

Do I need developer skills to do E2E-testing with Botium ?

It depends. If you are using one of the well-known and widely used webchat widgets out there (for example, the Bot Framework Webchat), then the answer most likely is no, you don’t need developer skills. For custom webchat widgets, you need at least the skills to define Webdriver selectors.

I have my own Selenium server, can I use it with Botium Box ?

Sure, see here.

I have an account for mobile device cloud XYZ, can I use it with Botium Box ?

If the device cloud provides a Selenium endpoint, then most likely the answer is yes.