Customizing your cloud

In Bamboo 2.2, we’ve released support for running builds in the cloud with the Elastic Bamboo feature. Of course, like so many other features with Atlassian products, we wanted to make the feature useful for ourselves right from the start. However, whilst the basic premise of being able to unlock elastically compute resources for our builds is undoubtedly promising, the approach was not without its own challenges.
For a start, builds were not entirely “environment agnostic”, in that they assume some setup on the environment has already happened. Some Confluence functional test builds, for example, needed to have certain databases installed on the environment the build is happening in. The Selenium tests also required Firefox.
The other issue is what some of our engineers on the team have dubbed the “download the entire internet” problem. Whilst each Confluence build doesn’t quite download the entire internet, the Maven build does have a sizable amount of dependencies needed to be pulled down from the public repositories. Most of these dependencies are quite static. But because we blow away our instances every time, these downloads need to happen every single time an instance starts up (which can, potentially, be every build). Taking up to several minutes to download, and, with EC2 data transfers in costing 10 cents a gigabyte, this is certainly not an acceptable level of overhead from a cost and time perspective.
In short, what we needed was some way to customize our elastic instances easily. This will allow us to enable the environment for the build, and “pre-warm” our elastic instances, taking away the need to “download the internet” on every start up.

.codeSnippet pre {
overflow: auto;
}
.codeSnippet {
border-width: 1px;
border-style: dashed;
padding: 5px;
margin: 10px;
}

Customizing Images

One obvious solution, of course, is to create a customized image (Amazon AMI). We can create an AMI which looks exactly like the remote agent machines we have set up before. The image can include the various required resources for the build. In addition, it can include the maven libraries pre-loaded onto the image to minimize startup time and transfer costs. This solution would work, and indeed, is the only solution for builds with very specific build requirements (e.g. must run on a specific operating system).
However, we felt that the need to create a customized image was cumbersome. For a start, creating a customized image was no easy feat. We wanted the process to be as painless as possible. Also, as part of the Elastic Bamboo feature, Bamboo ships with a default image set up with all the necessary infrastructure to ensure the instance can connect to the Bamboo server and ready to accept builds immediately on start up. By creating a customized image, we will now need to maintain this infrastructure ourselves. Any upgrades to the default image in the future will have to be ported manually to this customized image.

An alternative with Amazon Elastic Block Store (EBS)

EBS is Amazon’s persistence mechanism for the EC2 service. Essentially, you can think of it as a mountable drive you can attach to an EC2 instance. Similar to how an image is a “mould” for a running instance, EBS allows you to create a snapshot which is a “mould” for a running volume.
In the end, we found using EBS was the simpler way to customize. EBS allowed us to separate the infrastructure of Elastic Bamboo, tied to the image, from the customization needed specifically for our builds. An EBS snapshot would contain all the files necessary for customization. When an Elastic Instance started up, it would attach the volume, and then run a special script inside that volume (customise-extras.sh) which we can modify. This would perform the “customization” steps on the image.

An example – setting up Confluence builds in the cloud with EBS

True to the Atlassian dog-fooding tradition, we wanted to send our Confluence builds into the cloud. Confluence builds included both unit test builds (which had no external dependencies), and functional test builds needing database servers and Firefox (for Selenium).

Starting out

To start things off, we were able to get the Confluence unit tests to run on an Elastic Agent with hardly any customization needed. However, we did want to “pre-warm” the image, so we included in our EBS volume:

• a pre-populated Maven repository
• a settings.xml file, which uses the Maven repository on the EBS volume.

When the instance started, the settings.xml will override the default settings.xml on the instance. This was enough to get our unit tests were running in the cloud – in comparable times to local builds.

Supporting PostgreSQL

The next step was to set up something more complicated – a functional build with dependencies on a PostgreSQL database. To do this we needed to install PostgreSQL on the Elastic Instance. For this, we created a simple setupPostgres.sh script, which used the package management tools provided by Fedora to install and configure PostgreSQL on the agent:

1. Use yum to install the PostgreSQL server packages.
2. Initialise the PostgreSQL server environment by creating the database directories and default config files.
3. Create a new pg_hba.conf file which trusts all local connections and all connections coming from localhost.
4. Starts PostgreSQL.

setupPostgres.sh

#!/bin/sh
yum install -y postgresql-server
service postgresql initdb
cat > /var/lib/pgsql/data/pg_hba.conf << EOF
local all all trust
host all all 127.0.0.1/32 trust
EOF
/etc/init.d/postgresql start

This script is invoked from the customise-extras.sh script on startup of an instance.

Supporting Selenium Tests

The last step was to set up the support for Selenium tests in Confluence. Just like supporting PostgreSQL, we created a simple setupFirefox.sh script, which used the Fedora package management tools to install Firefox and enough of X to get a vnc server running:

1. Install the following packages:
   • vnc-server: the vnc server used by the selenium test server
   • xorg-x11-server-Xvfb xterm xorg-x11-server-utils twm xorg-x11-fonts-*: these packages cover the xorg dependencies to get firefox to run.
2. Copy some pre-prepared vnc authentication files into the bamboo home directory and sets their permissions accordingly.
   • vncpasswd the password file used by the vnc server (copied to /home/bamboo/.vnc/passwd)
   • vncxstartup the script executed by the vnc server when a connection is made (copied to /home/bamboo/.vnc/xstartup)
3. Install Firefox into /opt/firefox (not using yum). We needed to install Firefox separately because the package installed by the Fedora 8 package management appears to be too old (2.0.0.8) for some of our tests.
   • The tar is extracted to the appropriate directory
   • The .bashrc file is customised to include the Firefox directory when searching for libraries. This will ensure Firefox finds its own libraries.

setupFirefox.sh

#!/bin/sh

yum install -y vnc-server xorg-x11-server-Xvfb xterm xorg-x11-server-utils

twm xorg-x11-fonts-* yum install compat-libstdc++-33

if [ ! -d /home/bamboo/.vnc ]; then

mkdir /home/bamboo/.vnc

fi

cp /mnt/bamboo-ebs/bin/vncpasswd /home/bamboo/.vnc/passwd

cp /mnt/bamboo-ebs/bin/vncxstartup /home/bamboo/.vnc/xstartup

chown -R bamboo.bamboo /home/bamboo/.vnc

chmod 600 /home/bamboo/.vnc/passwd

chmod u+x /home/bamboo/.vnc/xstartup

rm -rf /opt/firefox

zcat /mnt/bamboo-ebs/bin/firefox-2.0.0.20.tar.gz | tar -xvf- -C /opt

grep -q LD_LIBRARY_PATH /home/bamboo/.bashrc

if [ "$?" == "1" ]; then

echo "export LD_LIBRARY_PATH=/opt/firefox" >> /home/bamboo/.bashrc

fi

That's it! We now have Elastic Instances which have both PostgreSQL and Firefox setup, ready to run our Confluence builds!

Final words

We found using EBS this way was very flexible, and allowed us to successfully build in the cloud for both our Jira and Confluence builds. Because all the customization is done by the EBS volume, we were able to use the default supplied Elastic Bamboo image. Updating the snapshot is much simpler as well, and required a simple re-snapshot command generateSnapshot.sh on a running EC2 instance. This would generate a new EBS snapshot that we can use for future instances started by Bamboo.

The best part of all this is that in Bamboo 2.2, we have included this ability to customize your instance by attaching to an EBS volume. All the handy scripts and commands are included in the distribution as well. For more details, check out the Bamboo 2.2 [documentation](http://confluence.atlassian.com/display/BAMBOO/About+Elastic+Bamboo).