Menu

Jenkins and Bamboo Integration

Overview

qTest Manager allows for out-of-the-box integration with two Continuous Integration (CI) tools:

  • Jenkins
  • Bamboo

The integration collects executed unit test results while building projects in your CI server and reports them to qTest Manager. Our plugin does not execute your unit tests. It will only collect and submit test results (if any) to qTest.

To Set up this Feature:

  1. Activate the CI Integration feature in your qTest project.

  2. Map the CI execution status with corresponding qtest values.

  3. Configure the Test Run and Test Case organization.

  4. Install the qTest Plugin on Jenkins or Bamboo and configure it to connect to your qTest project.


Supported Testing Frameworks

We support testing frameworks that are capable of generating JUnit XML format test reports.

  • For Jenkins, if your testing framework does not generate JUnit XML format test reports, you can utilize xUnit Plugin to generate and publish test results before qTest Plugin collects and reports them to qTest.
  • For Bamboo
    • qTest Bamboo Plugin does not support Bamboo Specs.
    • Only JUnit XML format test reports are supported.


Supported Build Tools

  • Maven
  • Ant
  • Gradle

 

Firewall Settings

Please configure your firewall's settings to allows accesses/requests from your CI Tool server to the following specific Amazon servers' elastic IP addresses of our qTest OnDemand:

  • 54.83.56.253
  • 54.83.57.47
  • 54.83.57.78
  • 54.83.57.122
  • 54.83.57.143
  • 54.83.57.153

Activating the CI Integration

In order to receive Test Logs submitted from your CI server and associate them with the builds, you will need to enable this feature in your qTest project. This will need to be done for each project individually.

To Activate the CI Integration:

  1. Select the appropriate qTest Project.

  2. Hover over the Gear icon and select Automation Settings from the drop-down menu.

  3. Switch on the Activation Status in the CI Tool Integration section.

  4. Map the automation execution statuses of your CI tools with the corresponding qTest values.




Mapping the CI Status

Activating the CI Tool Integration will also activate the Automation Integration feature. Therefore, it is required that execution statuses for your CI tools are mapped to the corresponding qTest values. The CI tools' default execution statuses are:

Jenkins:

  • PASSED
  • FAILED
  • SKIPPED
  • FIXED
  • REGRESSION

For further information on Jenkins' statuses read this article.

Bamboo:

  • SUCCESS
  • FAILED
  • SKIPPED

To Map the CI Statuses:

  1. Select the Add Mapping button.

  2. Enter the CI tools status and the corresponding qtest value.

  3. Save.




Organizing Test Runs

While setting up the CI Tool Integration, you have different options to manage your Test Runs and Test Cases.

  1. In one Test Suite per selected Release: One Test Suite is created under each selected Release on qTest plugin's configuration in your CI server. This is the default option.
    • Jenkins: Test Suites are named as [Jenkins Project Name] [mm-dd-yyyy], in which [mm-dd-yyyy] is the suite's created date.
    • Bamboo: Test Suites are named as [Bamboo Plan Key] [mm-dd-yyyy], in which [mm-dd-yyyy] is the suite's created date.

  2. In one Test Suite for all Releases: Test Runs are created under only one Test Suite regardless of selected release in your CI server's qTest plugin configuration.
    • Jenkins: Test Suites are named as [Jenkins Project Name].
    • Bamboo: Test Suites are named as [Bamboo Plan Key].

  3. Test Case submitted from your CI server are located under a Module named as [Jenkins Project Name] or [Bamboo Plan Key]. You can optionally select a parent module to contain the CI module. If no parent module is selected, the CI module is created under root.

Jenkins Integration

Install the qTest Jenkins Plugin

  1. From the Jenkins' Plugin Manager, search for qTest Plugin and install.

  2. Add Submit JUnit Test Results to qTest as a post-build action. 

Note: If your testing framework does not generate JUnit XML format test reports, you can use the xUnit Plugin to generate and publish test results before qTest Plugin collects and reports them to qTest. Once the xUnit plugin is installed, add Publish xUnit test result report as a post-build action and then add Submit JUnit Test Results to qTest following it. 

IMPORTANT:

  • If you use xUnit Plugin to generate and publish test results, make sure that the option Delete temporary JUnit files is unchecked.

  • Maven projects: it is not required that there is a prior task to publish test results to Jenkins.

  • Freestyle and Multi-configuration projects: it is required that there is a prior task to publish test results to Jenkins. If not, no test results will be submitted to qTest and Submission Status will show the status as SKIPPED for the builds.

Below is a screenshot of the qTest Jenkins Plugin after being added as a post-build action.

qtest-jenkins-plugin.png

Configure qTest Jenkins Plugin

  1. Navigate to qTest Manager and locate the Download qTest Resources  icon to open the page.
  2. Select the CI Tool Integration section to open.

  3. Copy the API token from the Integration with Jenkins section. This allows Jenkins to access your qTest Manager instance. 
    Note: The API token is at a user level, which means it is different between qTest users. 

    jenkins-plugin-02.png

  4. Navigate back to the Jenkins plugin configuration page.

  5. Enter the following qTest Manager information:
    • qTest URL: the URL to access your qTest instance. e.g. https://demo.qtestnet.com
    • API Key: this is the Integration with Jenkins token you obtained from the qTest Manager Resources page. 

  6. Select the Retrieve Data button to load data from qTest Manager.

  7. Select the qTest project which the build is integrating with and where you want to submit test results.

    jenkins-plugin-01.png

  8. Configure the options to submit your Test Results to qTest Manager.
    • Submit test result to a Release as settings in your qTest Manager
      • By selecting this option, the qTest Plugin will submit the test results to a Release in qTest Manager (refer to Organizing Test Runs section above to learn how to set this up). Select a Release, which will be the target release of submitted Test Runs.
        jenkins-plugin-04_environment.png


    • Submit Test Results to an Existing Container
      • By selecting this option, you allow qTest Jenkins Plugin to submit test results to a specific container in qTest Manager.
        • Container: select a container where you want your test results to be submitted, e.g. a Release, a Test Cycle or a Test Suite.  The plugin will create a Test Suite directly under the selected container. Then it creates Test Runs underneath that Test Suite. It will then submit Test Logs to those Test Runs. If the Test Suite already exists, the plugin will not create duplicate Test Suite(s) but will submit test results to that Test Suite.
        • Check this box to create a new test suite daily. If unchecked, a single test suite, with this job’s name, will be used: qTest Jenkins Plugin will create a new Test Suite(s) daily, then create Test Runs underneath of that test suite and then submit Test Logs to those Test Runs. If the Test Suite already exists (e.g. created by previous Jenkins built in the same day), the plugin does not create duplicate Test Suite but submit test results to that Test Suite. 
          submit-results-to-existing-container.png
  9. Environment:  Optional. The selected Environment will be set as submitted Test Runs' property. 
    environment.png
    • Known issue: The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one.

  10. Overwrite existing test steps: Selected by default, when test results are submitted to qTest Manager, the test case steps are also updated in the log. If the original test case contained test step details with manual test steps, these details would get overwritten, since the test step logs would create newer test case version(s). 
    NOTE: If you wish to retain test step details from the existing test case, you can deselect this option. 
    overwrite-existing-test-steps.png

  11. Parse test results from testing tools: If selected (default), the plugin will scan for the XML files which contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the plugin will scan the whole project.
    parse-test-results-from-testing-tools.png

  12. Utilize test results from the CI tool: In case there is one of the prior tasks that have already scanned and published test results to Jenkins, for example by using xUnit Plugin, qTest Jenkins Plugin will utilize the test results output from that task without scanning again.
    utilize-test-results-from-ci-tools.png

  13. Each JUnit Test Suite (class) = a qTest Test Case: It is the default selected option. A qTest Test Case is created from a JUnit Test Suites (class.)
    each-junit-test-suite-equals-a-test-case.png

  14. Each JUnit Test Case (method) = a qTest Test Case: A qTest Test Case is created from a JUnit Test Case (method.)
    each-junit-test-method-equals-a-test-case.png

Bamboo Integration

Install the qTest Bamboo Plugin

  1. Install the qTest Plugin for Bamboo from the Manage Add-ons page in Bamboo.

 

Configure qTest Bamboo Plugin

  1. Navigate to qTest Manager and locate the Download qTest Resources    icon to open the page.

  2. Select the CI Tool Integration section to expand.

  3. Copy the API token from the Integration with Bamboo section. This allows Bamboo to access your qTest Manager instance. 
    Note: The API token is at a user level, which means it is different between qTest users.

  4. Navigate back to the Bamboo Plugin Configuration page.
     
  5. Configure a build plan for your Bamboo server.

  6. Select a build stage under the plan.

  7. Select a build job under the stage.

  8. Add a new task to the selected build.

  9. On the pop-up, search for the qTest Integration task and add it to the stage. 
    Important: This task needs to be added to the job in which unit tests are executed.

    mceclip12.png

  10. After the task is added, you will need to configure the task to connect to your qTest Manager instance.

  11. Enter the following qTest Manager information:
    • qTest URL: the URL to access your qTest instance. e.g. https://demo.qtestnet.com
    • API Key: this is the Integration with Bamboo token you obtained from the qTest Manager Download Resources page. 

  12. Select the Retrieve projects button to load data from qTest Manager.

  13. Select the qTest project which the build is integrating with and where you want to submit test results.

  14. Select one Release which will be the target release of submitted Test Runs.
  15. Optional. The selected Environment will be set as submitted Test Runs' property. 

Notes:

  • The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one. We are working on the fix for that and will release it soon.
  • The plugin will scan for the XML files which contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the plugin will scan the whole project.

CI Integration Reports

CI Tool Integration Report

This is an export file which contains details of tests submitted from your CI server. It includes two views:

  • Build by Build: details of all tests executed in each CI build.
  • Test by Test: execution results of each test through different CI builds

To Download the Export File:

  1. On the Test Execution tab, select the Export button.

  2. Select the CI Tool Integration Report.

    • Select filters before exporting. Note: The export is not filtered by what you select on Test Execution tree. It is only filtered by values selected in this pop-up.

Test Result Submission Report

On your CI tool side, we provide a report allowing you to track the status of test submissions to qTest after each build is completed.

    • Jenkins: Select a build project. Click on qTest Plugin on the left panel.



    • Bamboo: Select a build plan. Open qTest Plugin tab.

 

 

Subscribe To Our Blog
Powered by Zendesk