Menu

Dolby: Scaling Test Automation with qTest APIs

Author: Michael Murphy, Senior Manager, Technology Operations at Dolby Laboratories 

Background

Dolby Laboratories Inc. implements audio and video technologies across myriad products. It’s internal continuous integration systems is constantly running builds and test automation on a 24 x 7 basis across the globe generating tens of thousands of test results per day. Visibility to this automated check on product quality change-to-change is key.

Testing Activity Per Month

 

Avg Num Projects

 27

Avg Num Platforms Tested

 153

Avg Num of Builds

 422

Avg Num Test Runs

 296,016


Dolby implements test automation using common open source test frameworks with a mix of internally-developed tools all integrated together to provide a broad framework for conducting API testing, SDK testing, UI testing, performant testing and feature testing.

Previously Dolby was using Testlink to store and track test results from build to build and release to release. Each test cycle could generate thousands of results but these results could take hours to fully store when using the Testlink API or any system that only provides one-by-one updates to test runs and test cases.  It was common to see the reporting of test results take longer than the actual build and test job time on Dolby’s CI environments. This delayed teams from receiving key quality information from their test results to support decision making. Moreover, valuable compute time on CI agents was being consumed by the reporting process rather than on the builds and tests waiting in the CI queues. Dolby made a decision to move to qTest to improve their efficiency.

Dolby partnered with QAS during its initial implementation to develop an additional API method in qTest to provide the ability to bulk upload results reducing a test run reporting from hours to mere minutes thereby making their test results almost immediately available in qTest Manager and qTest Insights and through qTest JIRA Integration. Dolby Labs takes advantage of QAS products to make the reporting available on demand to all its stakeholders.

Tools Used

  • qTest Manager
  • Test (OSS) and internal proprietary developed test framework (Python-based)
  • Proprietary python library to process XUNIT XML output.
  • qTest APIs
    • Batch Submit Test Logs of Multiple Test Runs
    • all APIs

Results

 

Testlink

qTest

Average Time to Report per Test Cycle

45 test runs per minute

4,000 to 7,000 test runs per minute

Report Availability per Test Cycle

1 - 5 hours post tests completed

0.5 - 2 minutes

Detailed Workflow

ACTOR: a continuous integration job OR a test engineer at a console. In general, this is done all automated as part of the build process for a given Dolby software product.

  1. ACTOR deploys an AGENT to build the product and run tests. AGENT may be Dolby product hardware, embedded system board, virtual machine.
  2. ACTOR deploys product and test source code onto AGENT.
  3. ACTOR initiates test harness environment.
  4. ACTOR initiates tests
  5. TESTS generate JUNIT/XUNIT compatible XML output.
  6. ACTOR tears down or resets AGENT.
  7. JUNIT is picked up by ACTOR and call qTest REST API endpoints.
    1. Create a release (if not yet available)
    2. Create a build under the release
    3. Create a top-level test cycle.
    4. Call bulk submit automation test logs API endpoint and upload all results in one request. Any test cases not previously in qTest are automatically created and results are available within 1-3 minutes for review and analysis by the engineering team.

Rest API Endpoints 

Builds to test runs are controlled via automated continuous integration. After initial setup, team members are not directly controlling the build nor test execution nor gathering of results and just letting it occur based on the teams’ changes and commits to source code management.

In addition, engineering team members (beyond development and QA) regularly need to review the results and various reporting to assess product quality, requirements delivery, and test coverage during daily standups and weekly broader team meetings or sprint retrospectives. Dolby leverages Insights and provides this reporting through qTest Insights’ ability to share embedded results on its own internal company engineering wiki. This makes the information available to program managers, product managers and senior executives on an almost immediate basis. Dolby also uses qTest’s JIRA integration feature so that the traceability between requirements (JIRA epics, stories, etc.) is maintained and clearly reported against continuous integration testing.

About Dolby Laboratories

Dolby Laboratories (NYSE: DLB) creates audio, video, and voice technologies that transform entertainment and communications in mobile devices, at the cinema, at home, and at work. For more than 50 years, sight and sound experiences have become more vibrant, clear and powerful in Dolby. For more information, please visit www.dolby.com.

Dolby and the double-D symbol are registered trademarks of Dolby Laboratories. DLB-G

Subscribe To Our Blog
Powered by Zendesk