Plan to Win: Use Test to Your Advantage!
Learn how your team can leverage a diversified test strategy including automated, manual, and exploratory tests to build quality at all levels in a product.
Software Testing: Quality From the Beginning
There is a big irony with respect to quality. The quality assurance team does not assure quality. The development team assures quality. Each line of code either makes the customer experience better or worse. Developers own the quality relationship with the customer. The QA team alerts the broader organization about risk.
Why is this point so important for your software project? Teams that place the responsibility of quality on test devalue the development process. Developer absence from quality also allows issues to fester in the code base longer making them more expensive to fix. Software teams today involve a number of people: developers, testers, designers, product managers, and executive stake holders. All play a role in the quality of the final application.
Developers lie at the heart of a software project. They implement the experiences that customers will rave about at release. Too often developers don't get to own the entirety of their feature life cycle. Traditional test methodology holds that testing is a separate process out of step with the developer. Testing really is a collaborative process between the developer, the team, and the customers. Newer methodologies like test driven development promote more ownership for developers and collaboration with the team. Developers that own and prove the quality of a feature have fewer bugs, fewer round trips with QA, and are faster to market. A structured series of tests owned by the developer and test teams help protect code from incoming bugs. Developers that write tests are also more likely to keep their quality up.
Many Agile methodologies promote the idea of demonstrating working code to the product's stakeholders. Having the developer in the center of that experience reinforces a culture of quality. No one likes to see a bug come up during a demo. Visible demonstrations of code encourages an extra level of polish.
Engage Your Users
One of the leading tenets of Agile software development is to always maintain a shippable product. Doing so allows a steady flow of code from the development team out to the user base. Regular releases out to your customer base gives users the ability to provide feedback on the software to drive future development. Development and test teams that quickly respond to customer feedback are more successful.
Make feedback easy. Your users are busy. Therefore take the time to ensure that it's easy to give feedback on the product. Atlassian's JIRA Capture, Issue Collector, and JIRA Mobile Connect can enable your users to give valuable feedback in a low cost fashion.
As development engineers take on more of the quality process, the natural question comes "Can we eliminate test?" In a word, no. Test engineers still are valuable to the project because of their visibility into quality issues across the product. Effective test engineers understand and empathize with the customer as users of the product under development each and every day. Test engineers can:
- Give early feedback - Test engineers should be included early in the development process providing feedback on feature scope, architecture, and sustainable development.
- Enable the larger team to run faster - Software test engineers should be responsible for the product's automated test framework by responding to the needs of the development team
- Inspire creativity - Development engineers don't always find the right test cases when building a feature. Software test engineers supplement the test process by adding in new ideas for test cases. Test engineers also bring new testing practices to the organization.
- Test flexibly - Software test engineers focus on exploratory testing and serve as a clear set of eyes when the development team needs quick, critical feedback. This is helpful on less common use cases or areas that are harder to test in an automated fashion.
- Engage with the outside - As issues come in from customer support, software test engineers can triage to prioritize the severity and risk- allowing developers to focus on implementing fixes.
- Mix development and test - Effective teams grow and retain talent by sharing responsibilities across the team. Rotate the software test engineers with software development skills into feature development roles to give them a new perspective. Likewise, encourage developers to take part in improving the test automation infrastructure.
Build a Solid Software Test Strategy
An effective testing strategy includes automated, manual, and exploratory tests to efficiently reduce risk and tighten release cycles. Tests come in several flavors:
- Unit tests validate the smallest components of the system, ensuring they handle known input and outputs correctly. Unit test individual classes in your application to verify they work under expected, boundary, and negative cases.
- Integration tests exercise an entire subsystem and ensure that a set of components play nicely together.
- Functional tests verify end-to-end scenarios that your users will engage in.
So why bother with unit and integration tests if functional tests hit the whole system? Two reasons: test performance and speed of recovery. Functional tests tend to be slower to run, so use unit tests at compile time as your sanity-check. And when an integration test fails, it pin-points the bug's location better than functional tests, making it faster for developers to diagnose and fix. A healthy strategy reqires tests at all levels of the technology stack to ensure each part, as well as the system as a whole, works correctly.
The Benefit of Automation
The amount of change in software projects today far outpaces the ability of test teams to validate changes manually. How do highly effective teams manage this amount of change? They use automation! Computers are very good at running repetitive tasks quickly. That makes them perfect for testing! Test automation harnesses the power of computers to run test suites quickly, saving human capital for more engaging tasks in the organization. As tests are run more frequently, issues are caught earlier and fixed while the development engineer has context in that space.
For more benefits and best practices, check out our sister page on test automation.
Feature and Test Code: 2 Hearts That Beat as 1
Developers are acutely aware of how the feature they are implementing should work. Likewise, QA specialists are attuned to user behavior and able to imagine the nitche use (and mis-use) cases your application might be subjected to. Together, developers and QA are a testing powerhouse!
Engineers who are involved in the coding and maintenance of their own tests tend to find bugs sooner - even before committing code. And QA specialists help ensure developers are writing the right tests, especially at the integration and functional test level. Bugs cost everyone time. Finding them early and fixing them quickly makes the whole program more efficient.
As your automation test suite grows, some of the same issues that come from manual testing arise - namely, the tests take to run. Engineers can combat the bloat in three ways: faster performance, more focused testing, and horizontal scaling.
- For each test in the system, ensure that the test uses minimal resources. Can complex database queries be simplified? Can their results be stored stored statically?
- Selenium is great for mimicking user workflow but it is slow. Can server side and client components be tested independently by their API contracts?
- Equivalence Partitioning can make tests more effective when executed against different configurations.
- Tests that do not depend on each other can be run in parallel using multiple build agents.
Good Architecture Tests Well
Software today no longer lives in complete isolation. In a networked world software has to be flexible and integrate with a wide variety of other applications. Automated software tests exercise the interfaces of your product so when issues do arise, it's easier to track down where the actual defect lays.
Manual debugging is expensive. But automated unit and integration tests can validate server-side and client-side features independently using an API contract. Tools like Selenium can mimic user actions in functional test to validate important inter-system workflows.
Test Your Tests
One of the challenges to automated tests is knowing whether you are testing the right things. Code coverage tools like Clover analyze the test coverage but don't tell you if you're testing the riskiest areas. For new bugs that come into the system, developers need to ask, "Why did I not find this bug?" Ultimately, it's hard for a single person to find all the test scenarios for a piece of feature code.
Test code reviews can help flush out additional needed test cases as well as find flaws in existing ones. For example, how effective is the test pictured here? Does it really test that the item was added to the collection?
Testing the return code is not enough. Automated test should do more to ensure that the action actually happened - they should make additional calls to the application that verify the result is as expected.
Keep the Neighborhood Clean!
As the number of software tests increases there are bound to be test failures. And it's easiest to spot a house in disrepair when the rest of the neighborhood is clean. As the entire neighborhood deteriorates, finding that same house gets harder.
Your automated tests are no different. When one test signals failure, its easy to get the attention of the development team. If lots of tests continually fail the urgency to keep things running well drop significantly. Atlassian's Bamboo is a continuous integration that alerts the team when the build health deteriorates, and supports test quarantine to keep failing tests out of the way while repairs are in progress.
Why Test Manually?
Manual tests are a structured set of steps to test an area of a product, have an expected result and are documented as a set of test suites in a larger test plan. Manual software tests have the benefit that they are easy to design because they directly mirror the customer experience. And they are structured to have a high degree of reproducibility as users run them over a series of builds.
Rapidly changing products often benefit from some manual testing as the code is often short lived. Always test items foundational to the product in an automated process. Testing is all about reducing risk in delivering the product. If an area of code is foundational in nature to the system, it's a clear candidate for automation.
Play to Your Team's Strength
Black box test methodology assumes that the tester has no insight into the product under test - it is a "black box". Black box testing focuses on the inputs and outputs of the system without looking into its internals. A black box tester will ride the bike and report "it feels right".
White box testing looks at the component parts individually and validates that each component performs to spec. When testing a bicycle, a white box tester would look at the tires, wheels, pedals, frame, and handlebars each independently to see if they met the criterion for those individual parts.
Grey box testers work in a black box fashion but use underlying information about the system to drive tests.
So, why have a black box approach at all in the test strategy? Most users of an application will come to it with a black box approach. Exploratory testing has a lot of the benefit of black box test methodology that can be scaled across the larger numbers of people.
Value the Human Experience
Manual tests are human in nature - they mirror the human experience of your product. While exploratory testing is gaining ground against traditional manual tests, a regimen of manual tests fills the gap between exploratory tests and automated tests. If a bug arises from an exploratory session and it can't be easily automated, it's a candidate for the manual test suite. Manual tests ensure eyes are looking at the product and validating it. Effective rotation of manual tests across the team helps to ensure that there is good knowledge transfer as well as keeping a fresh set of eyes on the product.
Effective test planning weighs the cost of a failure against the cost of protection from that failure. If a software bug appears for a customer in the field, what is the impact? How common is that use case? If a customer found that bug, how quickly could the team fix it? Use manual tests when exploratory tests and automated tests are not sufficient, but always work to reduce the amount of manual tests to be run. Start with the most important tests. They usually are the ones that reduce the highest amounts of risk.
Testcase + Issue Tracking = Goodness
Test tracking and issue tracking work hand in hand to keep your software at its healthiest. Testers and developers benefit from seeing integrated reports of test results and any issues that come from testing such as linking issues to the test case that failed, and tracking the status of a test over multiple builds.
Manual tests that fail on a consistent basis are clear candidates for automation. They will reduce more risk in your program than tests that consistently pass. Atlassian's JIRA integrates with a number of test case managers including Zephyr which installs from the plugin manager in JIRA's administration console.
One of the benefits of exploratory software testing is that anyone can join in to help test because all they need to do is wander about the product in a free form manner. Exploratory testing is not random, yet they aren't scripted like manual tests, either.
Exploratory testers test workflows that make sense to them, which means you can learn a lot about the way people use your product in the wild. Exploratory test sessions can even motivate their users by offering rewards for the most number of issues, best defect, or doing something unexpected with the product.
Leverage Different Users
When released, your application will have a varied set of users. Some users will be domain experts while others will be beginners. Exploratory software testing captures all types of users and is key for validating user experience, as opposed to focusing on a specific area of code.
Well-designed products scale across their entire user base. For this reason, exploratory test results will be of keen interest to product owners, user experience designers, as well as the engineering teams.
Intuitively See Risk
There are a number of ways that a team can declare a product complete. List driven teams will see that all the required user stories are complete. Data driven teams will look at the issue tracker, test results, and other metrics to see if a product is ready. Exploratory tests invoke an emotional response to a product. Exploratory testers who are on the product team will feel if a product is ready. This can be some of the most valuable data about a product's health. Products with more risk in them have less positive emotional responses. After each test session, understand the human element of the experience and learn from it.
What's the User Story?
Exploratory tests fit well with the Agile concept of user stories. A user story is a plain English description of how software should work. Engineers implement software based on user stories. Software test engineers and other users can easily turn a user story into a Test Session. Test Sessions are short, focused tests that can evaluate a user story for correctness.
JIRA Agile (formerly GreenHopper) and JIRA Capture (formerly Bonfire) work together for the entire Agile lifecycle. First, teams define and track user stories in JIRA Agile. Then JIRA Capture supports single- or multi-user test sessions, including the test results, right alongside the user story.
Make Feedback Easy
It's essential that exploratory testers remain product focused. That means giving feedback and reporting bugs needs to be simple and effective. If feedback is not comprehensive enough, it's not actionable by the engineering team. If feedback is too cumbersome for users, the development team won't get feedback from enough users.
JIRA Capture allows your users to give high fidelity feedback easily and also includes key environment back to the engineering team. JIRA Capture installs right into the browser and can capture feedback for native and web based applications. Development engineers can create templates that introspect a user's application, ensuring that the relevant data like browser version attached to each and every bug report.
Feedback Loop to Other Tests
When issues are found in exploratory testing it means that the scripted testing failed to catch an issue. Learn from that experience. Analyze why the product failed. Review existing tests to see if a test should have caught that failure. Fix the lazy test if you can. If it's a completely new test case, add an automated test if possible. And do some brainstorming around that test failure. Bugs often cluster - if there is one undiscovered bug, there are likely a few others in the same area.
When the team uses Bonfire, it is easy to capture the effectiveness of the testing and make testing at all levels more effective.
Blogs & Articles
- Automated Testing ...for the win!
- How to mock up your Unit Test environment to create alternate realities
- Selenium Tips: Uploading Files in Remote WebDriver
- Why Your Next Cloud App Will Probably Suck Without….Unit Testing
- Release Engineering at Facebook
Manual and Exploratory Software Testing
- Test Management with JIRA
- Let’s explore - Exploratory Testing!
- Capture great feedback with JIRA Capture
- QA Innovation at Atlassian
- What is Exploratory Testing?
- Exploratory Testing (from The Turning Point)
Building better software in every industry, every day.