Close

How to create and manage test cases with Xray and Jira

Sergio Freire
Sérgio Freire

Head of Solution Architecture and Testing Advocacy, Xray

Step-by-step instructions on how to write, specify, organize, and execute test cases

Time: Eight minute read. Complete in one session (two hours) 

Audience: You are new to Xray Test Management. You've created a project and want to learn how to write, specify, organize, and execute test cases. 

Prerequisites:

  • You’ve installed Xray in your Jira instance
  • You’ve created or added Xray to your project

What is a test?

A test is an abstraction of a test idea/scenario and a way to verify/validate associated requirement(s). It’s one or more checks, an experiment, or an investigation with the goal of obtaining information about a quality attribute of the test target.

A traditional, manual test involves a sequence of steps coupled with conditions or variables, test inputs, and expected results. It is intended to establish the quality, performance, and/or reliability of a piece within a system. 

Usually, every requirement or objective of the test target is expected to achieve at least one test.

The success of a test is determined by comparing the expected and actual results.

 A test can be: 

  • Scripted (e.g. test case or automated test) or exploratory 
  • Specified using Gherkin (scenario)
  • Executed manually or through automation
  • Linked to/cover one or more requirements
  • Like any other Jira issue type, it can be labeled, prioritized, assigned to components, and commented on

Tests and testing in general don’t ensure quality. They provide information about quality.

100 percent coverage is a myth. Adding more tests doesn’t ensure better information about the quality that matters to you.

It’s often better to focus on assessing what stakeholders are concerned about, consider risks, and specify and perform testing based on these metrics. 

There will always be some bugs. It’s important to understand what’s valuable so testing can focus on what threatens those values.

Test process in Xray

Xray allows you to plan, design, and execute tests, as well as generate test reports. Xray uses specific Jira issues types for this process. 

Each testing phase allows you to use the following issues:

  • Plan phase: Test plan issues
  • Design phase: The specification is defined using precondition and test issue types. The organization of the tests can be defined using test sets
  • Execute phase: Test execution issues
  • Report phase: Test execution issues including built-in requirement coverage reports and custom issues using Jira Software tools

To start, it’s useful to add tests to a small project. You can use test issues to create tests for your requirements and execute ad hoc, unplanned test executions. Here’s how:

Step 1: How to write and specify tests

Test types

There are three default test types:

Each test type has one of three kinds of tests:

  • Gherkin
  • Unstructured
  • Steps

Actions you can perform on Xray tests: 

  • Create
  • Edit
  • Clone

Create

To create a new test issue follow these steps: 

Step 1: Click Create Issue at the top of the screen to open the Create Issue dialog box/page.

Create button

Step 2: Select the Project. On Issue Type, select Test.

Step 3: Type a Summary for the test and fill at least all mandatory fields marked with an asterisk.

Create issue

Step 4: When you are satisfied with the content of your test, click the Create button.

Step 2: How to organize tests

Organizing tests is a crucial part of daily testing activities. Since you might start with a large number of tests and your tests may grow over time, it’s important to organize them in an efficient and effective way that works for you. 

Xray provides two different ways to organize your tests:

  1. Use the test sets issues
  2. Use the test repository

The test set is a simple way to create different groups of tests, since it is a flat list of tests. You may have as many test sets as you wish and a test may be included in multiple test sets. Test sets are ideal if you want to have full control over certain groups of tests.

Create test sets

To create a new test set issue:

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog/page.

Step 2: Select the Project and on Issue Type, select Test Set.

Step 3: Type a Summary for the test set and complete at least all fields marked by an asterisk.

Step 4: When you are satisfied with the content of your test set, click the Create button.

The test repository is a tree-like organizational structure at the project level. It allows you to hierarchically organize tests within folders and sub-folders. This folder concept is common in some tools and resembles file organization in a computer's operating system.

Create a test repository

To create a test repository, create a new folder:

Step 1: Click on the parent folder.

Step 2: Click on the add icon at the top of the folders section.

Test repository

Step 3: Type the name for the folder on the create folder dialog.

Create Folder

Step 4: Press "Create" to create the folder.

Step 3: How to plan tests

Test planning allows you to decide your testing strategy, including the issues you want to validate, how to validate them, if tests will be manual or automated, how resources will be allocated, and when and who will execute the tests.

Depending on several factors, you may want to prioritize some tests or focus on specific requirements.

Create a test plan

Here’s how to create a new test plan Issue: 

Step 1: Click Create on the top of the screen to open the Create Issue dialog box/page or press the "C" key on the keyboard.

Create button

Step 2: Select the Project. On the Issue Type, select Test Plan.

Step 3: Type a Summary for the test plan and fill out at least all mandatory fields marked with an asterisk. 

Step 4: When you are satisfied with the content of your test plan, click Create.

Create button

Step 4: How to execute tests

Test execution occurs between developing test scripts and reporting and analyzing test results. 

A test execution is an issue type that aggregates a user-determined collection of tests. It monitors and verifies if tests work as expected in a target context and environment. The overall execution status, updated after each test is performed, informs you about the progress of the test execution, including which tests passed, failed, are being executed, or are waiting to be performed.

Create a test execution issue

Step 1: Click the + at the left of the screen to open the Create Issue dialog box.

Create Issue dialog box

Step 2: Select Project. On Issue Type, select Test Execution.

Step 3: Type a Summary for the test execution and fill out at least all mandatory fields marked with an asterisk.

Step 4: In the Create Issue dialog populate the test execution custom fields provided by Xray, including the Begin/End date and Revision

Optional: To create a series of similar test executions within the same Project, check Create another at the bottom of the dialog.

Step 5: When you are satisfied with the content of your test execution, click Create.

Learn more

For more detailed information on how to create and manage test cases in Jira and Xray, check out the Xray Academy.

See more integrations for Atlassian Open DevOps.

Sérgio Freire
Sérgio Freire

Sérgio Freire is the Head of Solution Architecture and Testing Advocacy for Xray, a cutting-edge Test Management app for Jira. He works closely with many different teams worldwide to help them achieve great, high-quality, testable products. He believes that by understanding how organizations work, processes and quality can be improved while development and testing can "merge" and act as a unique team, with a common goal: provide the best product that stakeholders need.


Share this article

Recommended reading

Bookmark these resources to learn about types of DevOps teams, or for ongoing updates about DevOps at Atlassian.

Devops illustration

DevOps community

Devops illustration

Simulation workshop

Map illustration

Get started for free

Sign up for our DevOps newsletter

Thank you for signing up