In DevOps circles, there’s a myth about the software development process: real software teams automate everything. Look at Facebook. Look at Google. Look at Etsy. Companies killing it at DevOps. Hundreds of deployments a day. Push-button, amazing, magical automation.
And like many myths, there’s some truth to this one. Continuous integration, automated testing, and cloud-deployment have made delivering code arguably the easiest part of the software development process. But dig a little deeper and you learn that great software teams haven’t taken humans out of the loop at all – they’ve moved them to different points.
Companies like Facebook and Google have two overwhelming advantages over most of us:
- Thousands of tech-savvy employees who use their products every day on their own time and often for work; and
- Billions of users who use their software without paying for it, and therefore can’t complain all that much if things don’t work flawlessly.
So it’s hardly the case that they don’t use human testers in their software development process. Google dogfoods, for example. For some of their products employee testing is mandatory, and for just about all of their products a segment of the customer base gets new products before they go live for everyone. If you’re reading this, there’s a one hundred percent chance you’re an unwitting tester for one of those companies.
What do you do if you’re not Facebook or Google, you don’t have thousands of employees who can test your software internally, or billions from whom you can choose “just a couple million” to test it externally before you roll it out?
Dogfooding: it’s not just for dev teams
Dogfooding means using your own product before your customers do. If you can possibly do it, you should, because developers don’t always think like end-users and you’ll learn a lot from real users. But that’s a catch-22 of the software development process – the people who like to dogfood products are developers who often don’t see bugs or usability issues with their software because of familiarity bias toward products they’ve worked on.
So if you’re in a medium-to-large sized company, you may have a bigger internal audience than you think, but you have to look outside of your dev team to find it. This may mean changing the way you think about getting feedback. A few tips:
- Consider having feedback contests, where the prizes go to charities supported by the winners. People like to compete and incentives matter, but in most cases the money available might not be enough to motivate people on its own. So create solidarity and good cheer instead.
- Make sure it’s easy to give feedback. People outside your dev team will not necessarily fill out all of the JIRA fields that your team does, so consider a simplified form or use a chat tool, like HipChat.
- Give people special bonuses for testing from home, or in a real-world situation if it’s a mobile app. Perfect network connections and big beautiful screens are probably not the reality for your users.
If you don’t have enough time or people, or your app is not amenable to true dogfooding in your environment (say you’re a startup building an enterprise app or you’re developing something people don’t use every day like a car rental app), you have other options for getting human feedback quickly and early enough in the software development process to help:
- Set up a contest on your staging site, with real prizes or charity backing, and make it goal-oriented: finding the best deal on a car rental or the hidden insight using the data mining tool.
- Use crowdtesting, giving the crowdtesters scenarios to follow with usability feedback as a requirement. HSE24, A German online retailer with sites throughout Europe uses crowdtesters from test IO to protect against cultural bias in their designs and ensure that location-specific features work as intended.
Feature flagging: you don’t need a billion users
Once rank-and-file employees are done testing their apps, Google and Facebook release the software to production. But that’s not the same as releasing it to customers. In fact, a key part of the modern software development process now involves human testing in production – colloquially known as “feature flagging,” turning on new features for a subset of your user community. At any point, Facebook might be running hundreds of different experiments on its users, with entire countries serving as de facto testers (thanks, New Zealand!).
While feature flagging helps Facebook slice and dice its billions of users and run hundreds of experiments, it can be useful for much smaller organizations with fewer users. By putting code behind a feature flag, you can monitor your code for runtime errors, monitor goal attainment metrics, and survey your customers for feedback on how a “secret” feature is working for them. Making feedback and bug reporting easier for those users with embedded tools is a great idea, particularly for mobile apps.
Even with high levels of test automation, front-end issues often slip through – especially on mobile. If you’re worried about the quality of the software you’re about to release to your customers, you can use feature flags for final QA testing. Consider having your internal team – or a corps of crowdtesters – use the feature flag version before you release it to your customers. That way, QA is not a bottleneck to moving code through your pipeline, but you can use testers for what they’re best at – discovering issues from the customer’s point of view before your customers do.
Shipping to your customers: just because you can, doesn’t mean you should
The DevOps revolution of the past few years has put the focus on rapidly shipping software, but we need to remember that software is ultimately for people. In the old days when shipping software was expensive and difficult, people sometimes called QA the department of product prevention; but in the age of automation where products emerge more iteratively, perhaps the rarest quality is human insight – does the benefit of releasing this to a lot of customers outweigh the risks?
Dogfooding and feature flagging are ways of injecting more human testing and insight into the software development process. Great dev organizations will be as good at harvesting that insight as they are at shipping code.