This is a guest post by Shani Shoham, President and COO for Testim.io, a test automation platform that uses machine learning to create self-healing stable tests.


What is continuous testing? In broad terms, it means testing from the beginning to the end of each development cycle. It means running unit tests, functional tests, performance tests, compliance tests, and everything in between, as often as possible, at every step of the development cycle. With the developers on the far left, and the production on the far right, the term that has come to prominence around continuous testing is “shifting left.”

Everyone talks about shifting left in the development process, but what does it actually mean, and what are the benefits? Perhaps it’s best to quantify the problem, first.

According to Agile Modeling, the farther right in the production lifecycle a bug is caught, the more it costs to fix. Finding a bug in the requirements phase costs around $1: it can be fixed with the stroke of a pen or keyboard. When the bug is caught in the design stage, you’ve already multiplied your costs by a factor of ten. Pushing that bug to development means it will cost another 10x: $100. Your developer will likely take an hour or two of their time to fix this issue, and their time costs a lot more because programmer time is always in high demand.

Things start to snowball from here on. Catching a bug in testing costs $1,000 on average, because now it’s being pushed back to a developer who has likely already expunged the needed systems info from their brain in favor of the next project’s internals. This means fixing a bug could require an entire day from a developer, or even worse, an entire day from two developers.

But fixing a bug in production is where things go completely sideways. At $10,000 per bug fix, you’re deeply impacting your bottom line. That’s a number large enough to wipe out an entire day’s profits for the application you’re fixing. Maybe even an entire week’s profits. You’ve now entangled your operators, administrators, managers, testers, and developers in the fixing of this bug, diverting entire departments away from roadmap work to fire extinguishing.

One way to keep this from happening is to automate as much of your testing process as possible. While your testing team should have a goal of catching everything before pushing to production, they can also help push discovery to the left by providing the most automation possible.

Automating early and broadly means developers checking in code can be alerted to issues immediately, confining bug fixes to their development process, rather than to some timeframe after they’ve checked in code and moved on to another project. With bug finding as a gatekeeper to successful builds, you’re preventing issues from being pushed forward before the developer forgets how to fix them.

This dovetails into fast builds. If you’ve got a team that would die without five-minute-or-shorter builds, then you’ve also mandated faster automated tests. In such a team, your ROI is literally tied to the time your members spend on each story, each issue, each sprint. With short sprints, testing and fixing time is also shortened.

This can be a good thing or a bad thing. For clumsy teams, it can mean pushing more bugs to the right. But for a well automated team with a well greased continuous integration and continuous delivery pipeline, it can mean faster iteration times, better feedback loops, and shorter time fixing issues. However, this can only happen when the majority of your tests can be run on every build in an automated fashion.

If you’re only running the bulk of your regression tests just before you ship to production, you’ve already backed yourself into a corner when a fix will cost at least $1,000. Get those tests to run faster, that build to be more efficient, and your team to be looking farther left, and you can cut those costs and the time it takes to fix issues.

Of course, testing continuously requires a lot of discipline and hard work.

Here are 5 tips to help you start shifting left

1. Get live traffic

Testing in a continuous fashion requires a continuous view of the incoming data your system runs on. Whether that’s customer traffic, inter-machine data transfers, or simple transactions, the best way to test is with actual traffic data.

For a start, you can’t just send actual over-the-wire data through your systems without scrubbing it, inspecting it, and masquerading it so that no sensitive information is being bandied about in your regression tests.

Next, you’ll need to ensure the traffic you’re sending to your servers is extracted from the proper systems at the proper times. If your servers are undergoing an update, maintenance, or even a DDOS attack, even a few of those packets worming their way into your test streams could cause havoc.

There are many systems out there that can help capture and groom traffic for internal usage, but perhaps another route is to simulate your own traffic based on the real thing. This can be done with the help of a TCP dump, or PCAP, and with the assistance of some basic text generation tools, like Markov chains and name randomizers.

It’s a more work-intensive route, but in the long run building your own traffic harness based on real traffic is a sure way to keep tests honest.

2. Test at scale, in the cloud

If you’re not already using the cloud to test your software, you’re losing both time and money while missing out on capabilities. Remember the days of building and maintaining a test lab? The cloud allows that lab to be dynamically constructed, deconstructed, modified, and expanded with the push of a button.

Using the cloud to test means being able to burst up for performance testing before major releases, saving video of failed tests forever, and having to pay zero dollars for servers when you’re not actually running any tests.

The scale of the cloud can even give you a chance to replicate your real-world load scenarios, ensuring your servers are up to the task of the quarterly sales report, or the end-of-year customer buying spree.

But perhaps the biggest benefit of testing in the cloud is the ability to save results, logs, and other seemingly ephemeral test data. Once saved, big data analytics can be used on this information, and eventually, machine learning algorithms. The future of software testing and QA is decidedly in the cloud, and barreling towards a future of robot-like automation that can learn and remember forever.

In the past, test results have been fretted over for a few days, then discarded in favor of new ones. Long term existence of bugs is tied directly to the bug report that originally found it, not to the actual functionality that has been impeded. This information is timely, but seemingly useless over the long haul.

That is not that case anymore. Machine learning is built on large pools of data. Unstructured data can be stored and analyzed. Long term disk space is cheap, and cold storage even cheaper in the cloud.

These three factors combine to push the future of innovation in software testing. It’s a mundane task for a human tester to notice when a text box has been moved 200 pixels to the right, but for some testing tools this is a breaking change that requires human intervention. Simple movements in the UI can ruin ancient tests that look for world-ending bugs, while negative state changes in the application can be caught by unit tests, but often not by traditional UX, user acceptance, or basic functionality tests.

3. Keep regressing

Imagine a regression test that can understand when a button has been moved and won’t fail every time. It may sound like a pipe dream, but the capability exists in machine learning and AI-based test automation platforms. Your regression tests don’t have to be static, dull, and ancient assets, relegated to some dusty closet on the test chain labeled “everything else.” Those tests can be living, breathing entities (again with the help of machine learning), and the gathering of data. Regression tests are often seen as assets that aren’t desirable to deal with. When they fail, usually a time consuming process to get to the root cause of the issue, even when it’s just a simple fix. Those are the types of situations that make testers have heart attacks: This problem is either huge, or it’s just that a button was moved in the UI. Waiting to figure out the difference is like opening the casket of a vampire. Is Dracula really in there waiting to kill us all, or is it just an empty old box full of dust and a cape?

Using machine learning to revise and rework regression tests brings about a grim reality: it ensures that Dracula is always in the box. If your regression tests can become self-healing entities that are maintained by something resembling an artificial intelligence, then they won’t go screaming in the night for anything short of the actual walking dead come back to life to feed on the blood of the lifecycle.

4. Get creative

This tip can get a bit messy and should only be attempted by experts who are comfortable in the disciplines of both CI/CD and microservices. It also goes hand-in-hand with tip number 1. If your services are stable, secure–and most important–can fail gracefully and reliably, you can test in production.

Before you go insane, harking back to the fundamental rules of QA, there is a method to this madness. When your Web-based application is built on RESTful services, it is entirely feasible to branch out some A/B testing scenarios to include new features.

Got a new drop-down menu on the front page? A/B testing with a small portion of B users getting the box, but not having it drawn on screen, can at first test the existence of the new code outright. Turning the new drop-down box all the way on for that small subsection can slowly test the microservice behind it, ramping up the number of users as the success rate nears 100%.

Honestly, you’ve already been testing in production, you just don’t know it. You have found bugs in production before, right? That was testing in production.

With the right mindset, teams can use this tactic to perform staggered roll-outs of new code, and be sure that when they’ve pushed something 100% to production, there won’t be any new problems found by the users in the wild because they’ve already been discovered by those B users in small groups. This avoids a show-stopper bug getting in the way of the entire line of business.

However, don’t use this as a crutch, or an excuse to not perform normal test batteries. This is a chance for your team to perform more tests, not less, and add more citizen testers, not reduce headcount.

5. Test the environment, not just the code

Using the cloud to test software means being able to control the entire environment. This means the application, the load balancers, the databases, the microservices, and even the external APIs upon which your system is built.

That means using all of those things in your tests. Don’t just test the application at load, test it at load with the load balancers. If you’re using Google Maps, use Google Maps and test with Google Maps. This may make things take longer, but perhaps it’s a chance for you to test your caching layer?

Everything in the stack is a moving piece in a larger puzzle and the wider you cast your testing net, the easier it will be to trace back a problem. Everyone wants full-stack engineers on their teams, but if everyone is a full-stack engineer, then every test must be focused on the full stack, not just on the smaller picture.

With so many moving pieces rotating, grinding, and being updated, leaving even the smallest piece of the puzzle outside of the test environment can lead to variation at deployment time. And variation between the end environment and the test environment is where bugs get through to production.

5 tips for shifting left in continuous testing