In this blog post we will focus on Xray Test Management which is a JIRA plugin that allows teams to manage all test items as JIRA issues. We will discuss the testing lifecycle activities and break down how to effectively use Xray entities and features to plan, prepare, execute and report on the testing performed.

Test planning and design

An important aspect of test management is being able to plan effectively and structure tests in test plans and cycles in accordance with the overall project schedule.

• Let’s review the Xray entities, which provide the flexibility for users to perform the following:
  Create Tests to validate specific JIRA requirements, which then can be linked to provide end-to-end traceability.
• Add Preconditions to tests to ensure initial conditions are met, for example for a login test, a precondition can be that a user has been invited and has already registered on the system.
• Sort and organise tests within Test Sets, which essentially can be used as a collection of tests such as ‘Login tests’.
• Create a Test Plan and import Tests or Test Sets into this plan to capture the current version of these tests, this can be used for fortnightly sprints or System Integration Testing (SIT) for instance.
• Create a Test Execution which contains a list of Tests or Test Sets to be executed along with detailed steps, expected results and recorded execution status for each test. Typically Test Executions are linked from a Test Plan, however they can also be created in an ad-hoc manner.

Testers are usually delegated functional areas to script test cases for, as an example a tester may be required to analyse requirements for logging into a system. They will then script test cases in Xray specifically around the Login functionality, covering both positive and negative scenarios and ensuring the acceptance criteria has been met. These tests can then be grouped into a Login test set and order them in a logical fashion.

The test manager who oversees planning can then create specifically a SIT test plan for example, and import the Login test set as well as others, which contain the tests to build out sufficient test coverage for the system under test.

Test execution

Multiple test executions can be created from this SIT test plan, which aligns with the sprint deliverables to map out what is being tested in Cycles 1, 2 and 3 over the coming months. This is to ensure all tests are executed and there is room for defects to be identified, fixed and re-tested in subsequent cycles.

The overall SIT test plan will show the total number of test executions per test across the cycles, but the overall status will always reflect the latest version of the test executed. For instance, if a specific test fails in Cycle 1, is blocked in Cycle 2 and passes in Cycle 3, then the test plan will detail there were 3 test executions for this test and show an overall status of PASS in the test plan.

Xray provides all the bells and whistles for test executions, allowing the tester to execute tests with detailed steps and expected results, along with recording any comments, evidence/attachments, outcomes and raise associated defects in JIRA.

Also Read: The Power of JIRA for Project Management

It also supports creating both manual and automated tests, with the ability to specify tests in Cucumber BDD style language, to hook into test automation frameworks. Xray 5.0 was released in July this year, which now supports key major testing features such as parameterized tests, data-driven testing and compatibility with the Robot Framework v4.0. These tests can then also be linked to continuous integration (CI) systems such as Jenkins or GitLab.

One of the key benefits of this Xray is encapsulating requirement, test and defect entities all within JIRA and therefore having that full traceability across the project. This transparency is imperative for auditability which is vital for organisations running large complex projects.

Test Reporting

Xray provides test managers the ability to perform MI reporting with its out-of-the-box reports and gadgets, covering some key areas such as:

• Overall requirements coverage.
• Test run summary.
• Test coverage.
• Test plans metrics by status.
• Test sets list by status.
• Test executions list by status.
  
  

These gadgets can be added and sorted on a dashboard, as well as setting filters against each gadget. This allows users to present an overall dashboard to senior stakeholders on key testing metrics.

Also Read: What is value stream management?

A couple of limitations we have encountered with the Xray reporting features, is that there is only one evolution type chart (i.e. data over time) and no ability to analyse current data and forecast or predict upcoming trends. Currently Xray only has a Test Evolution gadget that shows test execution status trends for a group of Tests over a given time frame. It would be useful for test managers to view other evolution type charts for example, test scripts created over time or unresolved test executions across priority over time.

Neuro for Xray

Over at Dragonfly HQ, we’ve worked with multiple test management tools and plugins and given all the features and benefits outlined earlier on Xray, we’ve decided to take it one step further and that’s one of the main reasons why we built neuro.

Neuro is our engineering and quality management platform that turns insight into action, integrating with multiple systems including JIRA, Xray, Zephyr and Jenkins – removing the need to configure and manage reporting dashboards across each system separately. What this means for neuro users is having a single source of truth and being able to produce an overarching report capturing JIRA issues, defects, test execution and automation build outcomes all in the one consolidated view.

Some other key features of neuro are insightful out-of-the-box dashboards (including multiple evolution type charts), being able to enable predictions for charts over time, or to set project goals to closely track key milestones. For example, we can track a UAT Test completion milestone by visualising work throughput over time when constrained by a specific end date and identify when action is required to ensure the milestone is achieved.

Stay on top of your testing with Xray and neuro, by discovering neuro’s other key features and how it can drive efficiency, increase value and accelerate your growth by talking to our team today.

You might also be interested in our blogs on jira dashboards, and zephyr dashboards.