“Asynchronous Build Tiers”… could I possibly think of a dryer, technical-for-the-sake-of-sounding-technical term for what I’m about to describe? Geez. I’m almost embarrassed. It’s actually an exciting concept, though.
Let’s say your team commits code changes Chicago-style (early & often). Let’s also say that you have several layers of automated tests: unit tests, integration/API level tests, and UI-based tests that, in total, take a couple hours to run. Rather than making your devs wait 2 hours to get any feedback on their changes, you can break those tests into builds that run on different intervals, giving you two or more feedback loops. It’s worth noting that this approach also helps un-clog your build queue by using your build agents more strategically, as you’ll see.
And I Care… Why?
The point of fast feedback is to get build results to your developers so quickly that they aren’t inclined to start on a new task while they wait. Because incessant switching between tasks is just plain inefficient.
One way of acheiving that is to set up a build that executes with every commit, and does just 3 things: checkout source code, compile, run unit tests.
This often takes 10-20 minutes –enough time to make a few refactoring tweeks (or a sandwich), but not enough time to dig into the next user story. That’s your inner-most feedback loop. It serves as a sanity check on whatever was just committed.
The integration- and UI-level tests do need to be run frequently. But run with each & every commit? Probably not. Scheduling them to run at intervals throughout the day is usually sufficient. And on days when that’s not, they can be launched on demand with the push of a button. That’s your outer feedback loop.
True, you wait a bit longer on average for UI-level feedback. However, triggering these tests every 2-3 hours instead of with every commit keeps your build queue flowing more freely and lets those inner-loop builds churn through lickity-split.
Products that get consumed on multiple platforms benefit from testing against each of them. In this case, you might consider an outer-outer loop: a nightly build that executes all your integration and UI tests against all N permutations of operating systems, DBs, browsers, etc. By doing this in the middle of the night when your build agents are idle, you can marshal all your resources and throw it against this rather large effort. Because you’re running the same tests against your baseline platform during the day, you’re unlikely to uncover true show-stoppers here. The outer-outer loop is intended to ferret out platform-specific bugs. These are fairly rare, so looking for them just once a day is both effective and efficient.
Show Me The Codez
Ok, so what does setting this all up look like? We use a system like this at Atlassian and I was tempted to pull some screenshots of the real deal. But frankly, our build plans in Bamboo contain enough cryptic acronyms n’ stuff that I set up some simplified builds to illustrate. (read: sanitized for your comprehension)
The basics are simple enough. Set up an inner-loop build that checks out, compiles and unit tests. It should also spit out a deployable artifact, assuming all the compilation and tests succeed. As a final step, send the artifact to temporary storage where the outer- and outer-outer loop builds can grab it. (Coming soonish, Bamboo will be able to pass artifacts between Plans and temp storage won’t be necessary. In the meantime, SCP or a script will do.)
If possible, set up the transfer in such a way that, when a build passes all the tests in the outer-most loop, you can trace that artifact back to the original inner-loop build -and revision number- that produced it. Bamboo ships with build-specific variables you can use to append an identifier to the artifact’s file name or name a directory to temporarily store it.
Next, when it’s time for the integration and UI tests to execute, the outer-loop build pulls down the most recent artifact and deploys it to QA before proceeding with the tests.
It’s important to define those files as shared artifacts so a downstream step can send them back to the file server if all the tests in this build pass.
After successful test runs, I send the artifact back to the temp server. The destination directory is different this time, though, so I have an easy way to clearly separate builds that passed this second loop and are worth putting through the last round of tests.
Now it’s time for the outer-outer loop, which is basically a ‘rinse & repeat’ (including the intra-Plan artifact sharing). Pull down the latest artifact that made it through the previous loop, deploy it out, and test away. I also added a manual stage that handles the production deploy and follows it with smoke tests. This way, we can guarantee we don’t release builds that haven’t been fully tested.
As my grandpa used to say, there’s more than one way to skin a cat. (Which I never understood, because he actually liked cats…) Instead of scheduling the outer loop builds on a chron as I’ve done, you can make them parent/child builds and select the option to only build the child (the outer loop) if the parent (the inner loop) is passing. There are also some fancy options for blocking child builds when the parent is building or in the queue. Depending on the frequency of commits and the duration of your test runs, the timing might line up just fine.
Hopefully this post has served up a tiny slice of inspiration for something you can get creative with and really make your own. And if you haven’t checked out Bamboo yet, there’s no time like the present. With the artifact and deployment enhancements we’ve got on the roadmap, stuff like this is only going to get easier.