Exponential growth, hundreds of permutations, and other complexities made testing a nightmare for Loblaw Digital, the software arm of Canada’s largest retailer. By integrating Jira Software with Adaptavist Test Management for Jira, their engineers customized a speedy, simple, and scalable solution.
Justin Watts and his engineering team at Loblaw Digital – the software arm of Loblaw, Canada’s largest retailer – couldn’t find a test management solution that would address their complex needs – until they discovered they could integrate Jira Software with Adaptavist Test Management for Jira (TM4J).
Justin, Director of Productivity Engineering, had been disappointed with the firm’s test management tools for a while. When Loblaw Digital grew from six employees to over 250 in just six years, it was time to make a change.
Engineering needed a solution that would support the unique role Loblaw Digital plays within its parent organization and meet the team’s three main goals: reduce toil, increase productivity, and get stuff done. After trying several tools to no avail, they finally found one that hit the mark.
In close collaboration with Adaptavist, an Atlassian Solutions Partner, Loblaw Digital customized an end-to-end, scalable testing solution that has reduced testing time, increased productivity, improved communication, and empowered the company to deliver more effectively on their mission to help customers “Live Life Well.”
A plethora of people, platforms, and permutations
Two organizations, hundreds of employees, numerous existing platforms, dozens of brands, three break points, two languages, multiple browsers, two testing methods.
Our time to deliver has 10x-ed multiple times.
Director of Productivity Engineering
These are just a few of the variables that made Loblaw Digital’s testing procedures so complex. Even their role within the larger organization is more involved than the typical in-house agency relationship. On top of building digital properties for over 20 brands, Loblaw Digital also owns the profit and loss statement for all of those properties. “Because we maintain the platforms we build, we value testability, reusability, and scalability,” Justin explains. “We own the technical debt, so it’s in our own best interest to care just as much about the journey as the end result.”
Since Loblaw is in the retail space, the company also has to plan for the end-to-end customer experience, from the user placing the order, to Loblaw completing it, to the customer potentially picking it up in person. These considerations made Justin’s wish list for a test management system even longer.
After extensive conversation, the engineering team narrowed their criteria to five main requirements for their new solution:
- World-class manual and automatic testing capabilities
- Native UI and API interfaces
- Integration with one of their existing platforms, like Jira Software
- The ability to track hundreds of permutations
- Clear, shareable test results for both technical and non-technical stakeholders
It was a tall order, but the team had faith that they could find the right partners with the right tools to make their dream a reality.
The hunt for the perfect fit
Justin and his team searched for a solution for well over a year. Loblaw Digital was already using Jira Software for every project, as well as Confluence for documentation and collaboration, so they hoped to find a tool or app that would extend their Atlassian stack.
The group started by visiting the Atlassian Marketplace to explore third-party integrations, but didn’t like that most of them required navigating back and forth between interfaces to complete tasks. They also tried apps that embedded directly in Jira Software’s UI, but found them awkward because the app would treat test cases like issues, which seemed more like a workaround than a solution.
Eventually, the team stumbled upon Adaptavist Test Management for Jira (TM4J), a Jira Software app that functions as a complete solution. ”The best test management system is the one you don’t realize you’re using,” Justin says. “Adaptavist...felt like a test management product within Jira Software.”
Justin bought TM4J Server sight unseen, then started experimenting with it. Within a month, he and his team had customized a solution that fit their needs to a tee.
A partner in crime for pushing TM4J to its limits
When Cristiano Caetano, Product Manager at Adaptavist, saw what Loblaw Digital was doing, he couldn’t wait to learn more. “We thought, ‘What the heck are these guys trying to do with the API?!’” he laughs. While some companies might have tried to stop Loblaw Digital, Cristiano and his team enthusiastically added fuel to their fire. “We saw they were trying to push the product in ways we hadn’t even imagined, so why not help and enable them to keep going?”
Adaptavist reached out to Justin to better understand how his team was using the tool, and so began a beautiful partnership that continues to this day. Caetano explains, “They give feedback to us, and we have a conversation to incorporate missing features and ideas to make it better.”
Having one suite and one set for both automated and manual tests brought us down to two-week releases from monthly, then to weekly. We’re now at daily, which means this system scales.
Director of Productivity Engineering
From time-consuming and complex to fast and easy
With TM4J and Jira Software, Loblaw Digital has been able to transform the testing process used by their internal team as well as the entire parent organization.
For example, take permutations, which presented one of the most complex and frustrating testing challenges. Because of all the variables involved, every test has 384 permutations. With other test management tools, Loblaw Digital would need to write and store 384 tests. Jira Software and TM4J allow for the use of “environments” to conduct tests and upload contextual results.
“We would generate test reports for every permutation, but it’s very hard to collate and look between the reports,” Justin explains. “We realized if we put the tests on an X-Y axis (tests over permutations) and used green or red squares, the same patterns were visible with a single look.”
This revelation not only made test results faster and easier for the engineering team to read, but also made them more understandable for non-technical stakeholders. Adaptavist was so impressed with the reports that they added this feature to TM4J.
Loblaw Digital has also leveraged TM4J and Atlassian to increase productivity by creating a Slack app, affectionately named Jeanie, that automates the process of running tests and reporting results. Justin’s team benefits from not only the automation itself, but also the resulting artifacts. “TM4J is really awesome because it’s generating an artifact in Jira Software, so anyone can go back and see what tests were run, when they were done, and view tons of fancy reports,” he says. “Atlassian is helping us be good neighbors and give feedback to everyone else while letting us operate the way we want to without having to click around multiple UIs.”
Another challenge Loblaw Digital solved with TM4J and Jira Software was tracing defects. Justin reports that one of his favorite things about using Jira Software for tests is the ability to bring together issues, stories, and tests into a “traceability matrix.” “What’s awesome about that is we can generate a report for every test we run and every story it relates to. That way, we can make sure bugs don’t accidentally slip through,” he says. “It also warns you if there’s something that’s unaddressed. This is great for stakeholders and keeps us honest.”
Speed, simplicity, and scalability – all from one solution
Loblaw Digital had three main goals for their test management solution: reduce toil, increase productivity, and get stuff done. Their Jira Software + TM4J solution has exceeded expectations on all fronts.
Tedious work is now easier because the tools fit the way the team operates. Justin explains, “Jira Software is unopinionated. I love that about it. We are able to build and operate quirky patterns that make perfect sense to us.”
With smooth, flexible workflows, the team is more productive and completes work faster. “Our time to deliver has 10x-ed multiple times.” Justin says. “Having one suite and one set for both automated and manual tests brought us down to two-week releases from monthly, then to weekly. We’re now at daily, which means this system scales.”
We have transitioned from Waterfall, to Scrum, to Kanban effortlessly without changing tools. We’ve been able to grow with the product and not feel like it’s in our way.
Director of Productivity Engineering
While the results have been undeniable so far, Justin is even more excited for what’s ahead. The team is now running around 200,000 tests a day, which is speeding up the delivery of new features and shortening regression times by 250%. What’s more, they believe they can make the process five times faster.
“We were worried we would pick a tool that would only help us for part of the journey,” admits Justin. “But we have transitioned from Waterfall, to Scrum, to Kanban effortlessly without changing tools. We’ve been able to grow with the product and not feel like it’s in our way.”
Justin has so much confidence in the solution’s scalability that he didn’t flinch when the parent company took notice and wanted to adopt Loblaw Digital’s tools into the larger enterprise. “Our usage of Atlassian has made them envious,” Justin chuckles. “We have a 2000-user license because anyone who collaborates with us in the enterprise wants to use it.”
Empowering employees to provide more value to the business has trickled down into more value to customers. With the support of Atlassian and Adaptavist, Loblaw is proud to be delivering on their promise to help customers “Live Life Well” faster and better than ever before.