Best Practice: Use Botium Crawler for Detecting Conversation Model
Please read this article about crawler, it's more up to date: How to maintain chatbot regression tests with minimum effort
What is Botium Crawler?
Botium Crawler is able to find all of the conversation flows in your the chatbot.
It detects quick replies and buttons and simulates clicks on all of the options and follows the paths until it reaches the configured depth or found and open-ended question.
In the following points the Botium Crawler is introduced by walking through an example.
Step 1: Create a Crawler project
Create from quick start
In the first step of quick start you are able to chose or register a new chatbot. Then in the second step you can configure some basic property of the Conversation Crawler. In the third step you can save the crawler project or you are able to save crawler project and start the first crawler session immediately.
Register a new Crawler project
If you have already a chatbot, you can register your new Crawler project directly. The Conversation start messages are prefilled with two typical start messages, so the only thing you need to do add the name and to select an existing chatbot.
Both from quick start and new registration you will ended on the Dashboard of your Crawler project. Here you can see the previous Crawler session results and the current execution settings.
Step 2: Advance configuration
Conversation start messages
These are the messages with the crawler starts the conversations.
As many start messages you have, as many parallel job will be started.
Maximum conversation steps
This is the depth of the conversation in the conversation flow. (One depth is a user-bot message pair.)
Number of welcome messages
There are chatbot which initiate the conversation without user interaction. In this case you have to specify how many welcome message will be sent by the bot.
If you don’t have any start message, then the crawler try to find quick replies and buttons in the welcome messages, and start the conversations along them.
Wait for prompt
Many chatbots answer with multiple bot messages. In this case you can define a timeout until the crawler wait for the bot messages after each simulated user message.
If the text or in the payload of any quick reply/button matches to any of the exit criteria, then that conversation is stopped there and marked as successfully ended conversation.
All text messages are saved as utterance. Crawler can recognize the non-unique utterances and merge them into one utterance.
Optionally you can select a specific agent, which you want to run the crawler jobs on.
Let’s see my Crawler project configuration. This banking example chatbot has two welcome messages, which contains buttons. The crawler can use these buttons to start the conversations, so I removed the conversation start messages. This bot is pretty small so the maximum conversation steps is just 3. This bot has multiple bot messages, so I raised the wait for prompt to two seconds to be able to sure that the crawler doesn’t miss any bot message.
Step 3: Start a crawler session
On the Dashboard tab or on the Results tab you can start a Crawler session. The session time depends on the size/depth of the chatbot and on the Wait for prompt settings property has a big impact as well.
In my case the session is finished within a minute.
If you click on the finished session you can see the result. In this case 5 conversation is generated (three of them reached the specified depth, but two of them finished earlier).
You can also check the whole conversation model as a graph on the Flow tab
Step 4: Open-ended questions
In case of conversations which are stopped with open-ended question you can add user responses.
And on the Open-ended questions tab you can see all your responses and edit them.
If you rerun the crawler session, the user responses will be handled by crawler like quick replies or buttons would be found on that certain path. So in my example I added ‘Tomorrow' and 'Wendesday’ as user response. In the conversation flow diagram you can see the result that these responses used by crawler on the corresponding path:
Step 5: Create or update a Testset from the crawler result
You are able to copy the crawler result into a new or an existing Testset by clicking on the 'Copy test scripts into Testset' button.
Botium Crawler will help you to detect automatically all paths in the conversation flow. Additionally all detected conversation flows can be saved as Botium test cases and utterance lists and it can be used as a base for a regression test set.