How to create and manage test cases with Xray and Jira
Step-by-step instructions on how to write, specify, organize, and execute test cases
Eight-minute read. Complete in one session (two hours)
You are new to Xray Test Management. You've created a project and want to learn how to write, specify, organize, and execute test cases.
You’ve installed Xray in your Jira instance. Get started with Xray for Jira here.
You’ve created or added Xray to your project.
You have an active Jira project with the GitLab application installed and configured. Get started with Jira here.
What is a test?
A test is an abstraction of a test idea/scenario and a way to verify/validate associated requirement(s). It’s one or more checks, an experiment, or an investigation with the goal of obtaining information about a quality attribute of the test target.
A traditional, manual test involves a sequence of steps coupled with conditions or variables, test inputs, and expected results. It is intended to establish the quality, performance, and/or reliability of a piece within a system.
Usually, every requirement or objective of the test target is expected to achieve at least one test.
The success of a test is determined by comparing the expected and actual results.
A test can be:
- Scripted (e.g. test case or automated test) or exploratory
- Specified using Gherkin (scenario)
- Executed manually or through automation
- Linked to/cover one or more requirements
- Like any other Jira issue type, it can be labeled, prioritized, assigned to components, and commented on
Tests and testing in general don’t ensure quality. They provide information about quality.
100 percent coverage is a myth. Adding more tests doesn’t ensure better information about the quality that matters to you.
It’s often better to focus on assessing what stakeholders are concerned about, consider risks, and specify and perform testing based on these metrics.
There will always be some bugs. It’s important to understand what’s valuable so testing can focus on what threatens those values.
Test process in Xray
Xray allows you to plan, design, and execute tests, as well as generate test reports. Xray uses specific Jira issues types for this process.
Each testing phase allows you to use the following issues:
- Plan phase: Test plan issues
- Design phase: The specification is defined using precondition and test issue types. The organization of the tests can be defined using test sets
- Execute phase: Test execution issues
- Report phase: Test execution issues including built-in requirement coverage reports and custom issues using Jira Software tools
To start, it’s useful to add tests to a small project. You can use test issues to create tests for your requirements and execute ad hoc, unplanned test executions. Here’s how:
Step 1: How to write and specify tests
There are three default test types:
- Cucumber: Write tests in a business-readable domain and specific language (Gherkin)
- Generic: An unstructured test, without steps. This is a way to abstract and have visibility of traditional automated tests or exploratory tests
- Manual: A traditional test case composed of a list of steps that are scripted
Each test type has one of three kinds of tests:
Actions you can perform on Xray tests:
To create a new test issue follow these steps:
Step 1: Click Create Issue at the top of the screen to open the Create Issue dialog box/page.
Step 2: Select the Project. On Issue Type, select Test.
Step 3: Type a Summary for the test and fill at least all mandatory fields marked with an asterisk.
Step 4: When you are satisfied with the content of your test, click the Create button.
Step 2: How to organize tests
Organizing tests is a crucial part of daily testing activities. Since you might start with a large number of tests and your tests may grow over time, it’s important to organize them in an efficient and effective way that works for you.
Xray provides two different ways to organize your tests:
- Use the test sets issues
- Use the test repository
The test set is a simple way to create different groups of tests, since it is a flat list of tests. You may have as many test sets as you wish and a test may be included in multiple test sets. Test sets are ideal if you want to have full control over certain groups of tests.
Create test sets
To create a new test set issue:
Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog/page.
Step 2: Select the Project and on Issue Type, select Test Set.
Step 3: Type a Summary for the test set and complete at least all fields marked by an asterisk.
Step 4: When you are satisfied with the content of your test set, click the Create button.
The test repository is a tree-like organizational structure at the project level. It allows you to hierarchically organize tests within folders and sub-folders. This folder concept is common in some tools and resembles file organization in a computer's operating system.
Create a test repository
To create a test repository, create a new folder:
Step 1: Click on the parent folder.
Step 2: Click on the add icon at the top of the folders section.
Step 3: Type the name for the folder on the create folder dialog.
Step 4: Press "Create" to create the folder.
Step 3: How to plan tests
Test planning allows you to decide your testing strategy, including the issues you want to validate, how to validate them, if tests will be manual or automated, how resources will be allocated, and when and who will execute the tests.
Depending on several factors, you may want to prioritize some tests or focus on specific requirements.
Create a test plan
Here’s how to create a new test plan Issue:
Step 1: Click Create on the top of the screen to open the Create Issue dialog box/page or press the "C" key on the keyboard.
Step 2: Select the Project. On the Issue Type, select Test Plan.
Step 3: Type a Summary for the test plan and fill out at least all mandatory fields marked with an asterisk.
Step 4: When you are satisfied with the content of your test plan, click Create.
Step 4: How to execute tests
Test execution occurs between developing test scripts and reporting and analyzing test results.
A test execution is an issue type that aggregates a user-determined collection of tests. It monitors and verifies if tests work as expected in a target context and environment. The overall execution status, updated after each test is performed, informs you about the progress of the test execution, including which tests passed, failed, are being executed, or are waiting to be performed.
Create a test execution issue
Step 1: Click the + at the left of the screen to open the Create Issue dialog box.
Step 2: Select Project. On Issue Type, select Test Execution.
Step 3: Type a Summary for the test execution and fill out at least all mandatory fields marked with an asterisk.
Step 4: In the Create Issue dialog populate the test execution custom fields provided by Xray, including the Begin/End date and Revision.
Optional: To create a series of similar test executions within the same Project, check Create another at the bottom of the dialog.
Step 5: When you are satisfied with the content of your test execution, click Create.
For more detailed information on how to create and manage test cases in Jira and Xray, check out the Xray Academy.
See more integrations for Atlassian Open DevOps.
Share this article
Bookmark these resources to learn about types of DevOps teams, or for ongoing updates about DevOps at Atlassian.