Best Practice: Setup a Process for Test Case Development
One of the questions coming up in the beginning of every project: Where should I keep my test case scripts ? In an Excel file ? In a Git repository ?
Fortunately, this question is quick to answer: Store your test case scripts in a Git repository!
Test cases should be handled in the same way as source code, and the state-of-the-art method for storing source code is Git. Connect your Git repository to Botium Box and use it for versioning your test cases.
In this article, we describe the recommended Botium Box structure for test case development.
Local Repository vs Linked Repository
Botium Box comes with an integrated local repository: test case scripts and excel files are stored within the Botium Box database.
Botium Box also allows to connect to linked repositories: test case scripts and excel files are stored in a remote location linked to the Botium Box.
Currently supported linked repositories are:
Git Repositories, for example Bitbucket, Gitlab or Github
Shared Folders connected to the Botium Box server
Download Links to retrieve test case content from an external HTTP server or Google Sheet
You should always keep the test cases in a linked repository, preferred is Git, as the local repository doesn't have many options for making backups, versioning etc
We will follow best practices from software development:
There is a development environment which will be used for developing the test cases
There is a production environment which will be used for running the finished and approved test cases
Snapshots of the test cases will be transfered from development to production environment on certain events (end of sprint, before go-live, …)
Preparing the Test Case Development Process
We will introduce a suggestion for the test case development process (yes, you should handle and call it like a software development process):
You are using the Live Chat, the Botium Crawler or the Convo Editor for composing your test cases
There is a Git repository for holding the test cases
We will create two test sets, one to place the test case scripts during development phase, and another one to use for the regular Botium test runs.
First Test Set: Development
This test set is for developing purposes. Make sure to select Use Local Repository only in the Content Selection Type of the Test Set Settings.
Connect your Git repository in the Linked Test Case Repositories section.
You should point to your Git main branch here. Later you can choose to not directly commit your test case scripts to the master branch but to separate feature branches.
In addition, create a test project pointing to your development chatbot and this test set.
Second Test Set: Ready2Test
Create a second test set. This time, select Use Remote Repositories only, and again connect your Git repository.
In addition, create a test project pointing to your chatbot under test and this test set.
Implementing the Test Case Development Process
As usual, this development process is running in cycles - you can and should repeat it as often as required.
1. Working on Test Cases
In the Development test set, you can use all Botium tools to create and edit your test cases, you can even combine all of them.
Use the Convo Editor or the Script Editor to manually compose your test cases step by step
Use the Test Case Wizard to generate test cases from your conversation model (works with IBM Watson, Dialogflow, QnAMaker and more)
Use the Live Chat to record your conversations with your chatbot and save it as test case
Use the Botium Crawler to automatically detect your conversation branches
using Botium Excel sheets for setting up this process is not suggested, as Git features like version tracking and pull requests work best on plain text files.
2. Verifying Test Cases
You can use the Development test project to run test sessions against your development chatbot to verify that the test set itself is complete and working. As you selected Use Local Repository only above for this test set, only the local repository with your own test set will be used for the test session, not the test cases in the linked Git repository.
You can consider the Development test set as the sandbox where the unpublished test case scripts are placed, before they are commited to Git and published to other project members.
As soon as you consider your test set to be complete, it is time to publish it to Git.
3. Export Snapshot to Git repository
Use the Export Local Repository function to export a snapshot of your test set to the Git repository now.
The Git checkin screen will look familiar to software developers:
You can choose a branch to commit your test cases to (and your Git credentials if required)
There should be a checkin comment describing the changes you have done
We recommend to not push directly to the main branch but instead push to a feature or development branch and use the established Git workflows of Pull Request / Review / Merge for bringing the test cases to the main branch.
The test case scripts are exported to the Git repository as files, and the existing test case scripts will be overwritten.
You can use all Git features in your Git repository now: feature branches, pull requests, revert changes, version history, ...
4. Setup Continuous Testing Pipeline
In your Ready2Test test set, you should now use the Update Test Case Statistics function, which will pull the main branch of the linked Git repository content to Botium Box and verify the test case content.
Now you are ready to use this test set in your Continuous Testing pipeline - Botium Box will pull the all changes on test cases from Git automatically on each test execution.
You now applied state-of-the-art software development methods to test case development with Botium Box.