Dolby: Scaling Test Automation with qTest APIs

Author: Michael Murphy, Senior Manager, Technology Operations at Dolby Laboratories 

Background

Dolby Laboratories Inc. implements audio and video technologies across myriad products. It’s internal continuous integration systems is constantly running builds and test automation on a 24 x 7 basis across the globe generating tens of thousands of test results per day. Visibility to this automated check on product quality change-to-change is key.

Testing Activity Per Month
Avg Num Projects	27
Avg Num Platforms Tested	153
Avg Num of Builds	422
Avg Num Test Runs	296,016

Dolby implements test automation using common open source test frameworks with a mix of internally-developed tools all integrated together to provide a broad framework for conducting API testing, SDK testing, UI testing, performant testing and feature testing.

Previously Dolby was using Testlink to store and track test results from build to build and release to release. Each test cycle could generate thousands of results but these results could take hours to fully store when using the Testlink API or any system that only provides one-by-one updates to test runs and test cases.  It was common to see the reporting of test results take longer than the actual build and test job time on Dolby’s CI environments. This delayed teams from receiving key quality information from their test results to support decision making. Moreover, valuable compute time on CI agents was being consumed by the reporting process rather than on the builds and tests waiting in the CI queues. Dolby made a decision to move to qTest to improve their efficiency.

Dolby partnered with QAS during its initial implementation to develop an additional API method in qTest to provide the ability to bulk upload results reducing a test run reporting from hours to mere minutes thereby making their test results almost immediately available in qTest Manager and qTest Insights and through qTest JIRA Integration. Dolby Labs takes advantage of QAS products to make the reporting available on demand to all its stakeholders.

Tools Used

• qTest Manager
• Test (OSS) and internal proprietary developed test framework (Python-based)
• Proprietary python library to process XUNIT XML output.
• qTest APIs
  • Batch Submit Test Logs of Multiple Test Runs
  • all APIs

Results

	Testlink	qTest
Average Time to Report per Test Cycle	45 test runs per minute	4,000 to 7,000 test runs per minute
Report Availability per Test Cycle	1 - 5 hours post tests completed	0.5 - 2 minutes

Detailed Workflow

ACTOR: a continuous integration job OR a test engineer at a console. In general, this is done all automated as part of the build process for a given Dolby software product.

1. ACTOR deploys an AGENT to build the product and run tests. AGENT may be Dolby product hardware, embedded system board, virtual machine.
2. ACTOR deploys product and test source code onto AGENT.
3. ACTOR initiates test harness environment.
4. ACTOR initiates tests
5. TESTS generate JUNIT/XUNIT compatible XML output.
6. ACTOR tears down or resets AGENT.
7. JUNIT is picked up by ACTOR and call qTest REST API endpoints.
   1. Create a release (if not yet available)
   2. Create a build under the release
   3. Create a top-level test cycle.
   4. Call bulk submit automation test logs API endpoint and upload all results in one request. Any test cases not previously in qTest are automatically created and results are available within 1-3 minutes for review and analysis by the engineering team.

Rest API Endpoints 

• http://api.qasymphony.com/#/release/create2
  • Based on input create a release (if not exists yet).
• http://api.qasymphony.com/#/build/create1
  • Create a build for this test run under the proper release.
• http://api.qasymphony.com/#/test-cycle/createCycle
  • Create a top-level test cycle based on input.
• http://api.qasymphony.com/#/test-log/submitAutomationTestLogs2
  • One-step upload of all test results from JUNIT file formatted to the API JSON format and pushed to the Release/Build/Test Cycle identifying the proper settings for environment or configuration set used.

Builds to test runs are controlled via automated continuous integration. After initial setup, team members are not directly controlling the build nor test execution nor gathering of results and just letting it occur based on the teams’ changes and commits to source code management.

In addition, engineering team members (beyond development and QA) regularly need to review the results and various reporting to assess product quality, requirements delivery, and test coverage during daily standups and weekly broader team meetings or sprint retrospectives. Dolby leverages Insights and provides this reporting through qTest Insights’ ability to share embedded results on its own internal company engineering wiki. This makes the information available to program managers, product managers and senior executives on an almost immediate basis. Dolby also uses qTest’s JIRA integration feature so that the traceability between requirements (JIRA epics, stories, etc.) is maintained and clearly reported against continuous integration testing.

About Dolby Laboratories

Dolby Laboratories (NYSE: DLB) creates audio, video, and voice technologies that transform entertainment and communications in mobile devices, at the cinema, at home, and at work. For more than 50 years, sight and sound experiences have become more vibrant, clear and powerful in Dolby. For more information, please visit www.dolby.com.

Dolby and the double-D symbol are registered trademarks of Dolby Laboratories. DLB-G