Menu

Jenkins and Bamboo Integration

qTest Manager allows for out-of-the-box integration with two Continous Integration (CI) tools: Jenkins and Bamboo.

The integration collects unit test results executed while building projects in your CI server and reports them to qTest Manager. In order to set up this feature, please follow steps below:

  1. Enable CI Integration feature in your qTest project
  2. Install the qTest Plugin on Jenkins or Bamboo and configure it to connect to your qTest project

Notes:

  • Our plugin does not execute your unit tests. It only collects and submit test results (if any) to qTest.
  • We support any testing frameworks which are capable of generating JUnit XML format test reports.
    • For Jenkins, if your testing framework does not generate JUnit XML format test reports, you can utilize xUnit Plugin to generate and publish test results before qTest Plugin collects and reports them to qTest.
    • For Bamboo, only JUnit XML format test reports are supported
  • We support these build tools: Maven, Ant, and Gradle

Important: Please configure your firewall's settings to allows accesses/requests from your CI Tool server to the following specific Amazon servers' elastic IP addresses of our qTest OnDemand:

  • 54.83.56.253
  • 54.83.57.47
  • 54.83.57.78
  • 54.83.57.122
  • 54.83.57.143
  • 54.83.57.153

Activating CI Integration

In order to receive Test Logs submitted from your CI server and associate them with the builds, you will need to enable this feature in your qTest project.

  1. Access to your qTest project. Click on the Gear icon and select Automation Settings.
  2. Switch on the Activation Status.
  3. Map automation execution results of your CI tools with corresponding qTest values.
  4. Click on Save button.

Mapping CI Status

  • Switching on CI Tool Integration will also activate Automation Integration feature.
  • It is required that execution results of your CI tools are mapped with correspondings qTest values. The CI tools' default execution status are as below:

    Jenkins:

      • PASSED
      • FAILED
      • SKIPPED
      • FIXED
      • REGRESSION

    See this article for further information of Jenkins' statuses.

    Bamboo:

    • SUCCESS
    • FAILED
    • SKIPPED

Organizing Test Runs

While setting up CI Tool Integration, you have different options to manage Test Runs and Test Cases.

  1. In one Test Suite per selected Release:

    One Test Suite is created under each selected Release on qTest plugin's configuration in your CI server. This is the default option.

    • For Jenkins: Test Suites are named as [Jenkins Project Name] [mm-dd-yyyy], in which [mm-dd-yyyy] is the suite's created date.
    • For Bamboo: Test Suites are named as [Bamboo Plan Key] [mm-dd-yyyy], in which [mm-dd-yyyy] is the suite's created date.
  2. In one Test Suite for all Releases: Test Runs are created under only one Test Suites regardless of selected release in your CI server's qTest plugin configuration.
    • For Jenkins: Test Suites are named as [Jenkins Project Name].
    • For Bamboo: Test Suites are named as [Bamboo Plan Key].
  3. Test Case submitted from your CI server are located under a Module named as [Jenkins Project Name] or [Bamboo Plan Key]. You can optionally select a parent module to contain the CI module. If no parent module is selected, CI module is created under root.

Jenkins

  1. Install the qTest Plugin from the Jenkins' Plugin Manager. 
  2. While configuring the build project in your Jenkins server, add Submit JUnit Test Results to qTest as a post-build action.
    • NOTE: If your testing framework does not generate JUnit XML format test reports, you can use the xUnit Plugin to generate and publish test results before qTest Plugin collects and reports them to qTest. Once the xUnit plugin is installed, add Publish xUnit test result report as a post-build action and then add Submit JUnit Test Results to qTest following it. 

  3. Important:
    • Make sure that the option Delete temporary JUnit files is unchecked.
    • For Maven projects: it is not required that there is a prior task to publish test results to Jenkins.
    • For Freestyle and Multi-configuration projects: it is required that there is a prior task to publish test results to Jenkins. If not, no test results will be submitted to qTest and Submission Status will show the status as SKIPPED for the builds
  4. Input your qTest project information so that the Jenkins results push results appropriately:
    • qTest URL: the URL to access your qTest instance. e.g. https://demo.qtestnet.com
    • API Key: the token to login to your qTest instance. You can get the Jenkins API Key from qTest Resources page:
  5.  Click on Retrieve button then select values for these fields:

    Project: select one qTest project which the build is integrating with.

    Release: select one Release which will be the target release of submitted Test Runs.

    Environment: It is not required. The selected Environment will be set as submitted Test Runs' property.

  6. Specify how the plugin will scan your test results. There are 2 options:

    Parse test results from testing tools (default): the plugin will scan for the XML files which contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the plugin will scan the whole project.

  7. Utilize test results from the CI tool: In case there is one of the prior tasks has already scanned and published test results to Jenkins, our plugin will utilize the test results without scanning again.
  8. Specify how to create Test Cases in qTest.
    • Each JUnit Test Suite (class) = a qTest Test Case: it is the default option. A qTest Test Case is created from a JUnit Test Suites (class)
    • Each JUnit Test Case (method) = a qTest Test Case: A qTest Test Case is created from a JUnit Test Case (method)

NOTE: The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one. We are working on the fix for that and will release it soon.

Bamboo

  1. Install the qTest Plugin for Bamboo from the Manage Add-ons page in Bamboo.
  2. Configure a build plan on your Bamboo server. Select a build stage under the plan. Select a build job under the stage. Add a new task to the selected build. On the pop-up, search for qTest Integration task and add it to the stage. Important: This task needs to be added to the job in which unit tests are executed. 
  3. After the task is added, you will need to configure it to connect to your qTest instance.
    • qTest URL: the URL to access your qTest instance.
    • API Key: the token to login to your qTest instance. You can get the Jenkins API Key from qTest Resources page. 
  4. Click on Retrieve button then select values for these fields:
    • Project: select one qTest project which the build is integrating with.
    • Release: select one Release which will be the target release of submitted Test Runs.
    • Environment: It is not required. The selected Environment will be set as submitted Test Runs' property

Notes:

  • The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one. We are working on the fix for that and will release it soon.
  • The plugin will scan for the XML files which contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the plugin will scan the whole project

CI Integration Reports

CI Tool Integration Report

This is an export file which contains details of tests submitted from your CI server. It includes two views:

  • Build by Build: details of all tests executed in each CI build.
  • Test by Test: execution results of each test through different CI builds

To download the export file:

  1. On Test Execution tab, click on Export button then select CI Tool Integration Report.
    • Select filters before exporting. Note: The export is not filtered by what you select on Test Execution tree. It is only filtered by values selected in this pop-up.

Test Result Submission Report

On your CI tool side, we provide a report allowing you to track the status of test submissions to qTest after each build is completed.

    • Jenkins: Select a build project. Click on qTest Plugin on the left panel.

    • Bamboo: Select a build plan. Open qTest Plugin tab.

 

 

Subscribe To Our Blog
Powered by Zendesk