As a software company boasting 15 products and add-ons, as well as dozens of plugins for them, Atlassian builds a lot of code. I mean, a LOT of code. We’re also famous for eating our own dogfood, so of course we use Bamboo to orchestrate it all. Conducting the orchestra is a small and dedicated team of nerds we creatively call “Build Engineering”. This team manages multiple Bamboo instances, each with well over 100 Plans and scores of agents to do the work.
The agents allow us to keep our build queues short, even at peak-activtiy times like the days leading up to a product release, and the few minutes leading up to the beer fridges being restocked. With plenty of agents on hand, we can run multiple Plans simultaneously, and within each Plan, run multiple jobs in parallel –all in the name of faster feedback times and higher developer productivity. Agents also help us test our products as though each test run was a pre-launch deep-dive regression. With a variety of OS, database, and browser configurations distributed amongst agents, we can execute our tests against the many permutations represented out there in our install base. A lot to keep track of, but the benefits more than outweigh the cost. And speaking of costs (specifically, reducing them)…
In his never-ending quest to automate himself out of a job, our own Adrián Deccico of the Build Eng team (and HungBuildKiller fame) recently came up with a 2-prong solution for administering all those agents.
- First, use Amazon EC2 instances for as many agents as possible. Because Amazon charges by the hour, this is far more cost effective than buying and provisioning enough bare iron to support the number of agents needed, especially since it’s rare that all the different platform configurations need to be building at the same time.
- Second, automate creating the images (which are analogous to a recipe or blueprint) needed to configure the EC2 instances. Not only is manual creation of images time-consuming, it is also error-prone and difficult to trace back.
Automating the process lets the Build Eng team keep pace with the steady flow of new JDKs and browsers the product teams want to test against, as well as make back-up copies of the configurations that would otherwise only exist on Amazon’s network.
The team already had a head-start on this because they use Puppet, a leading configuration management tool. Remember all those OS/DB/browser permutations? Each one is recorded as a Puppet manifest. When it’s time to provision a new machine, all you need to do is select a manifest, tell Puppet to do it’s thing while you grab another cup of coffee, and by the time you’re done stirring in the sugar, you’ve got a freshly-minted box to play on.
So: EC2 instances for the build agents, Puppet to manage their configurations, and a bit of home-grown black magic the Build Eng team named “Bellatrix” (in honor of Bellatrix Lestrange from the Harry Potter series).
Bellatrix is a suite of utilities that automates interactions with EC2 instances. Given a Puppet manifest, it will fire up a blank EC2 instance, invoke Puppet to provision the instance according to the manifest, and burn a copy of the configuration as an AMI (Amazon Machine Image) for use later by Bamboo. Bellatrix has it’s own Bamboo Plan containing the series commands to make all that happen; the Plan is triggered whenever a change is pushed to our repository of Puppet manifests, hosted on BitBucket. Having lifted nary a finger, we get a fresh set of the AMIs needed to test products on multiple platforms.
Currently, our internal Bamboo instances are updated manually with the new AMI codes. Not a huge burden since they don’t change very often. But the Build Eng team is serious about automating themselves out of their jobs, and plans to automate this step as well by building custom endpoints on top of Bamboo’s REST API. (Naturally, I’ll brag about that on their behalf in a future post when it’s up and running!)
Let’s Get All Hogwarts on This
If you want to start bewitching AMIs on your own project, you’re in luck: Bellatrix is available for download at http://pypi.python.org/
For those of you in the San Francisco Bay area (or looking for an excuse to visit), Dan Bode and Teyo Tyree of Puppet Labs will be giving a presentation at this year’s Atlassian Summit, May 30th – June 1st. Focusing on deploy automation, their talk will introduce you to Puppet, and use real-world continuous delivery examples to give you a peek “under the hood”.
Special thanks to Adrián for building this spellbinding tool, and helping me wrap my head around its awesomeness!