Menu

Cisco: Gaining Visibility Across All Test Methodologies

Author: Ryan SchafferQA Engineer at Cisco CloudLock

Why We Built This Integration

At Cisco Cloudlock, all of our teams use qTest Manager to track their test cases.  We use qTest Manager to holistically assess our test matrix from a single location, however, we have many decentralized teams. 

For example, our teams that contribute to core systems and do not have a direct impact on user flows often consist only of developers with no testers within their team makeup. This meant that we needed to report unit, integration and functional tests into qTest in order to gain full visibility into each decentralized teams’ testing methodologies. 

The solution needed to be performant as well as not have a dependency on pre-created test ID’s.  We also wanted to make sure that all our test cases were linked to our JIRA tickets to show test coverage and defect traceability.  With this in mind, we created our own plugin that integrates with the qTest APIs for the below results. 

Tools

Test Framework

qTest APIs

Open Source

Summary Integration Workflow

This plugin makes use of several hooks in pytest in order to integrate with the test framework.

          Pytest_sessionstart – Execute before session.main() is called.

          Pytest_sessionfinish – Execute after the whole test run finished

          Pytest_runtest_makereport – Returns the test report for a given pytest item and call info

Install and Configure

1) Install plugin via pip

2) Configure the plugin to connect to the qTest environment

  • Plugin takes command line arguments for:
    • Project
    • qTest url
    • base test directory
    • clearing the cache

In the pytest_sessionstart hook

3) Create nested qTest Manager test cycles based on the base directory, Git repository name, and directory the test is being run from.

For example, if you run the test from the root directory of MyApp and pass the argument ‘demo’ through the command line your test cycles will look like:

          Automation

                  |_demo

                             |_MyApp

If the test cases are run from a direction in the MyApp repository it will look like:

          Automation

                   |_ demo

                             |_ MyApp[directory]

4) The plugin will then either create, open, or recreate a cache based on the state and command line options.

5) The plugin will then cache all of the existing requirements from qTest Manager for linking purposes.  This ensures that we have an up to date list to test against and it saves time during test executions.

In the pytest_runtest_makereport hook

6) As the test cases execute, the plugin then creates the directory structure based on the test case’s location in the test structure of the python test suite. Directories are created as test cycles while files are created as test suites.

7) Based on the module structure and the test case name, a combination that is guaranteed to be unique, the plugin will either retrieve the module and test case IDs from the cache or create the module that the test case resides in and the test case itself.  If the module or test case needs to be created, it will be added to the cache.

8) The plugin will parse the comments of the test item looking for JIRA IDs using a regular expression.  If JIRA ids are found, the plugin will then link the test case to JIRA requirements using.

9) The plugin will store the result along with any error messages in the test report.  This will be used to create test runs and test log details at the end of the test session.

In the pytest_sessionfinish hook

10) Grab all the test results and test log details from the report object and asynchronously send them to the qTest Manager API.

Example Project

Directory Structure

Here is the directory structure of our example project.  There is no application, but this is a common test structure you would see.  In the editor window, we see an example usage of JIRA tickets in the doc strings.  As you can see this test will generate a failure.                                             

Test Case Created in Test Design

Here is the test case that is created in qTest Manager after the test run.  You will notice that the plugin strips the prepended test_ from the test case for legibility. 

The JIRA User Story (qTest Requirements) is now linked to the test case:                                                                                             

Test Run Results

This image shows the result of the test runs that have been asynchronously posted to qTest Manager.  If you view the Test Log Details, notice the failure message captured in Test Execution.  

JIRA Traceability

We also have the qTest JIRA plugin enabled.  Here you can see that the JIRA Ticket is updated automatically through qTest integration with test runs results.  In this example, we see that a User Story in JIRA displays the failed test run status.                                                                                                               

Notes

  • Initial test runs are extremely slow due to the heavy volume of API calls needed. This run time is substantially reduced in subsequent runs.
  • Plugin assumes users add and remove test cases. In the event of test case restructuring, the cache will need to be cleared.

Results

Using this plugin, we were able to gain a much better picture of the actual testing efforts that went on from release to release.   Here is a breakdown of how each decentralized team was able to:

  • Teams comprised only of developers were able to report all their results into qTest so QA engineers and QA management could get real-time insights into the testing practices of that team. 
  • Teams that do have QA engineers were able to automate acceptance criteria with in the frameworks that they were used to developing in.  This allowed QA engineers to spend more time on test scenario’s and exploratory testing while still having confidence that the acceptance criteria was running and passing.

Also, since all test results were linked into JIRA, engineering leaders were also able to see that test cases that were being run against a ticket without having to go into qTest to see results.  This enabled teams to have effective conversations about testing efforts without having to worry about losing any historical data needed for auditing purposes. 

About Cisco Cloudlock

Cloudlock was launched in 2011 with one simple goal in mind: to transform cloud security into a business enabler. From that initial goal, we’ve built out a unified cloud security platform that helps organizations protect their sensitive data in public cloud applications such as Google’s G Suite, Salesforce, Dropbox, Box, ServiceNow, and many others.  After being acquired by Cisco in August 2016, Cloudlock now is part of the Cisco Security Business Unit, and remains a cohesive team that is headquartered in Waltham, Massachusetts, with remote offices in London and Israel.  The cloud is our passion and cloud security is our mission. Cisco is one of the largest security companies in the world, with solutions ranging from Next-Generation Firewalls to Advanced Malware Protection and DNS-layer security.

Subscribe To Our Blog
Powered by Zendesk