Hello again! Glad to see you’ve come back for more tips on making Bamboo awesome for Git and Hg teams. Today we’ll talk all about tests. Grab a chocolate and get comfortable.
Comparing test status against upstream
When we left off last time, we were talking about slick ways to manage merges to upstream branches. Here’s another scenario: let’s assume that in your team sometimes a bad code hits the upstream. For example people do commit small fixes directly to the upstream branch. Or there is necessity for a really quick hot fix, which turns out bad. Or whatever. And the result is that sometimes the upstream branch becomes ‘red’ anyway. And if you do work on branches and use the ‘dry-run-the-merge’ technique, you will surely find yourself in the situation that Bamboo will notify you about breakages in your branch. And you have this common concern: “is this really my fault? is it faulty code on my branch? is it a faulty code from the upstream?”. Fortunately, Bamboo can provide you the information about the test status across the branches:
Ah, Bamboo is telling me that exactly the same test is currently failing on the upstream branch latest build. I don’t need to navigate through Bamboo – I have the relevant information right on my branch result. Good. I can use that information to focus on fixing the stuff that I’ve broken for sure – I mean, all the tests that aren’t marked as “Failing on the upstream branch”. The rest of breakages probably comes from dry-merging with the upstream and were introduced by another developer, who presumably is right now working on fixing them. I don’t need to tread on his/her shoes. And if my branch doesn’t have ‘mine own’ breakages, I can play bold and force Bamboo to merge my branch to the upstream anyway – with quarantines.
Quarantined test is a test that is considered by Bamboo as ‘successful’ regardless of its state – failing or passing. That mechanism can be used to temporarily ‘disable’ some tests giving you the time to fix them while simultaneously allowing the rest of the team to work on their stuff without ‘inheriting’ the failures. In the example above we can quarantine the ‘inherited’ failing test:
…and run again the branch, making it possible for Bamboo to merge back my branch into the upstream successfully. However, quarantines usefulness is more than this example.
Consider a situation where your tests contain a few so-called ‘flakey’ tests. These tests tend to fail occasionally, usually due to the bad patterns used in the test code, which emerge long time after the test was written and checked in. This is very distracting to the developers, as when a ‘flakey’ test fails you never know if this is real breakage (your fault – fix your stuff), or the random flakey-failure (move along, the next build will succeed). If you spend a few hours trying to figure if it is your error when it isn’t – that’s bad, you’ve lost hours. But when you ignore the failure thinking it is the random one, and you leave a faulty code in the upstream – that’s even more bad. Surely your fellow developers won’t like you for leaving the red builds.
Some counter-measure is to quarantine a suspected test when you’re in doubt. That way you might introduce a bug into the codebase, but if you keep an eye on the builds status you might notice that after a few hours (or on the next day) either the test is still failing and you can unleash the test (or fix it your code!), or either if you were hit by the flakey-ness occurrence, the test will be green and you might just unleash it. Either way, if your team workflow allows you to have broken code occasionally (for example you’re far away from the release date), you probably have saved some time (either your time or your team’s time) that could be used to do other things, like improving the tests infrastructure. Or having a chocolate.
Another thing with the quarantines is that they are inherited by newly detected branches. So if you happen to check in the code that fails the upstream build, but that doesn’t necessarily mean a fatal disaster (like for example you just introduced minor bug that actually can be ignored for a while, or like when you checked in a new test that illustrates an existing bug in the application), you might want to acknowledge the existence of that situation but not prevent the other developers from working on their things. So you quarantine the failing tests, and when people start to work on new things by branching from the HEAD of the upstream, they won’t be bothered with the fact that some test is failing (after all you know that the test is failing and you’re working on fixing the situation, right? so why should they bother?). They can work on their branches as if the test was passing and Bamboo can merge their stuff back into the upstream smoothly if necessary. It will track who has quarantined the test and when it was, no information is lost. However, if that is not enough you can use JIRA to store additional notes about the tests.
Linking tests to JIRA issues
Imagine a situation where one of the developers spot a flakey test. He/she can even start working on fixing the test, but in the distributed teams it might be difficult to broadcast that information properly to the other teammates. And it might happen that another developer hits the same flakey test in a different time zone and will start working on fixing the issue too. What if you could leave a note directly under the test. Something like “This is random build failure and the fix is currently in progress”. Surely that would help with the team communication. Fortunately you can link (or create new) JIRA issues directly from Bamboo test:
That way you can avoid stepping on another developer’s shoes. And further more, like in the picture above – if you find yourself in the situation that you’d broken six tests, five of which have linked issue – you know that you probably must focus on fixing the test with the missing linkage, because presumably this is the real breakage. Or even better – you can raise a new JIRA issue for that so you won’t forget about fixing the test, and focus on doing much more important things right now. Like having a chocolate or a two.
While there are far more features in Bamboo that make software developer life a less miserable experience, I’ve decided to pick these ones as I believe they suit DVCS teams quite well (do you agree? what are your favourite DVCS-oriented features in Bamboo/your CI?). I don’t think that you need to follow these tips to be a successful developer (you don’t even need to agree with all that I’ve written here!), but the general glimpse is that you should be able to find in Bamboo the features that will aid you in the day-to-day software developer job. And I just wanted to share my notes on Bamboo as a whole, as some of you might find a few of them fresh, interesting and maybe beneficial.
I hope you had enjoyed reading this blog. Thanks for reading!
Anyone can be good, but awesome takes teamwork.
Find tools to help your team work better together in our Git Essentials solution.