Skip to content

Add Botium Test Suite to Azure DevOps Pipeline

We prepared a step-by-step guide how to connect Botium Box to the Azure DevOps Pipeline.

Configure Botium Box Pipeline Integration

In Botium Box, open the test project you want to run from Azure DevOps Pipeline, select the Build Pipeline Integration tab and adapt the parameters:

  • Enter ${BUILD_ID} in the Build Id field
  • Select JUnit XML as output format
  • Enable the Wait with HTTP(S) response until test session completed option

Select the Curl Command Line tab and copy the command line for later use.

You can use other build environment variables in the command line script as well, you can find the list of available environment variables here. You will have to map it later.

Configure Azure DevOps Pipeline

Now there are several small steps to do in the Azure DevOps Pipeline

Create a DevOps Project

Create an Azure DevOps Project. The settings really don’t matter, choose them for your infrastructure.

Select a Git Repository

Azure DevOps Pipeline is configured in a file called azure-pipelines.yml - this file is saved in a repository of your choice. You can host it in your own Github repository, on Bitbucket or in your Azure Git repository - it doesn’t matter for Botium. So if you don’t already have a Git repository, just create a new one in Azure.

Create a Build Pipeline

Now add a new Build Pipeline, connecting it to your existing repository or to our newly created repository. Select Starter pipeline on the Configure tab.

On the last screen, change the pipeline configuration to this yaml definition:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

In line 11, replace the curl command with the one from your previous step, adding the --output $(System.DefaultWorkingDirectory)/testresult.xml flag.

What happens in this pipeline ?

  • It is bound to the “master” branch of the repository - any change in this repository will trigger the pipeline to run
  • First the Botium CI/CD integration is called to run the test project “Echo-Bot-Test-Suite“ and output the test result as JUnit XML
  • The file is saved to the working directory
  • The final task picks up the JUnit XML and publishes it as test result.

That’s it. Azure DevOps Pipeline is ready for running Botium tests!

Running Botium tests

Now you can run your first Botium test in Azure. There are several triggers available in Azure DevOps to trigger the test run.

You can fine-tune the triggers in the Triggers section of the pipeline configuration. You can either use the DevOps user interface to configure the triggers in your pipeline configuration, or you can modify your yaml definition.

Trigger Manually

The best option for your first test run is to use the Queue function to start the Botium tests manually.

Trigger from Repository

A better option for live usage is to trigger the test run from your Git repository

  • this is enabled by default: just make a commit and push from the connected Git repository (to one of the convo files, for example), and the test run will start automatically.

Scheduling Test Runs

You can also add instructions to the pipeline configuration to run the tests regularily - this is a good option for a nightly test run for monitoring purposes, for example. You can either use the DevOps user interface to add a scheduler to your pipeline configuration, or you can modify your yaml definition.

You can find several samples in the DevOps documentation above.

Viewing Test Results

You can see test results and the failed Botium test cases in Azure DevOps builds and drill down to the detailed Botium error message.

Sending Email Notification

Azure DevOps includes a highly configurable notification mechanism. You can use it to send email notifications when tests fail, succeed, or recover. The configuration screen is available in the Project Settings.

Attaching Test Report to Build

By extending your yaml definition you can download the test report from Botium Box (in several formats) and publish/persist it in the Azure DevOps pipeline. This is meant for long-term storage of test results you may need for auditing purposes, for example.

This yaml definition not only runs the test cases on Botium Box, but it also downloads the test report as PDF and as CSV file and attaches it as artifact to the Azure DevOps build:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.pdf "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=pdf"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.csv "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.pdf
    artifact: Test Result (PDF)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.csv
    artifact: Test Result (CSV)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

You can then download the test reports in the Related section from the Build Summary:

Attaching Transcript to Build

It is possible to download the Transcript (the conversation) from Botium Box for external processing/storing. You can integrate it with Azure DevOps pipeline, and store Transcript as Artifact.

Supported file formats are json, and csv. (REPORTER=csv or REPORTER=json)

Following yaml definition

  • runs test cases, downloads test report as XML,
  • creates artifact from test report
  • downloads the transcript as csv
  • creates artifact from transcript
  • publishes the test report:
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  displayName: Running tests, downloading test result
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=${BUILD_ID}&WAIT=1&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish test result as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.xml
    artifact: Test Result (XML)

- task: Bash@3
  displayName: Downloading transcript
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/transcript.csv "https://box.botium.at/api/transcript/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish transript as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/transcript.csv
    artifact: Transcript (CSV)

- task: PublishTestResults@2
  displayName: Publish test results, check failed status
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true