Build engineering can be fun!There’s this idea floating around: an idea that builds are the devil. That they’re unreliable, tedious and confusing. I won’t try to refute this… but my secondment has taught me that builds are so much more!

I began at Atlassian as a developer on the Confluence development team where my work primarily involved delivering features such as Confluence-Jira integration and the Confluence Space IA (Left Sidebar). So when I was approached about joining the Confluence build engineering team for 3 months I didn’t know what to expect. Confluence build engineering was a new team created for the purpose of maintain and improving Confluence builds ranging from test automation to automatic deployment.

Learnings, learnings, and more learnings

If I had to choose a word to describe builds, it would be “mentor”. Relentless, but with much to offer. And it goes far beyond the technical stuff.

By far my biggest learning was the consequences of technical debt. Builds are in their current state because we allow them to be: when we neglect them, they turn on us. And we on the Confluence team had been guilty of this at times. Joining the build engineering team was an opportunity to make things right. Without a strict product roadmap and sprint schedule governing my deadlines, I could work on what needed to be done and I had the time to do it well. This, in itself, was a tremendously satisfying experience!

My time on the builds team was also a lesson in self-management because there is much more freedom over how the team’s backlog is shaped. You will often stumble across something that needs attention, you will raise an issue for it, and you are probably the one who will decide how and when it will be done. For me, this translated to going to work and being given the opportunity to be the change I seek–every single day. This heightened sense of ownership played a key role in my enjoyment of the secondment.

But that doesn’t mean I was at liberty to choose my own adventure all the time. Multi-tasking is something build engineers learn to master and, with that, there is often a need to compromise. You’re often already juggling multiple issues when another comes long and taps you on the shoulder. It starts off being quite overwhelming but I soon learned to manage my priorities–a skill that will serve me well no matter what team I’m on.

Build engineering can be fun!

So what’s the work like on the build engineering team? Well, it’s certainly not all about flaky builds!

Housekeeping: For starters, I worked on a tool to report on build health. (Yes, there’s feature work involved!) There’s still work to be done in terms of figuring out how we can convert build health results into actions, but the foundations have been laid. I also set up builds for our new enterprise support policy, which entails supporting older versions of Confluence for up to 12 months–these older versions need builds.

Build health tool
Tool reporting on build health for the past two months

Quarantined tests: While looking over the quarantine list of our Bamboo instance I came across tests which have stabilized, flaky tests which hadn’t been fixed for months, and even tests which no longer exist. What a confusing mess! There’s more work to be done on this front too, but with the help of other build engineers, we made a tremendous effort to bring these forgotten ‘treasures’ back from the grave and improved test coverage. Yay!

…and also hard

Culture: There needs to be a shift in how we perceive our builds and how we respond to them. Ideally, the build engineering team should not exist to just fix builds–the same builds–over and over again. They should be there to provide tools and promote good practices so the broader team can more easily maintain quality builds. Quite like how our QA (Quality Assistance) team doesn’t do all the testing–rather, they equip and empower developers to be great testers.

Build visibility: Continuing on with the first point, in order to enable self-servicing of the builds we need to be able to communicate the right amount of information to the broader team. Too much information will be ignored (ever set up an inbox filter for all the notifications you get from your repo manager & CI system? yeah.) and too little means regressions can go unnoticed. The sweet spot in the middle is different for every team, and we’re still finding ours.

image2014-10-9 11-10-15
Visibility: reporting build health on Hipchat

Like ogres, broken builds have layers: There’s always another issue lurking underneath the initial flakiness. This is due to an accumulation of breakages as broken commits are made on top of existing breakages. Lets not let this happen! If you commit to a broken build, don’t expect someone else to fix it. Make it your responsibility to chase them up. Otherwise, how else will you know that your commit didn’t also break anything?

I confess, I was responsible for bringing down one of our internal Bamboo instances (only once!). At this point I knew I was a full member of build engineering (smile)

Final thoughts

I won’t go as far as saying that build engineering is a blissful paradise–doing so might raise the suspicion that working on builds has driven me insane! But I will say that this has been my most rewarding secondment. At the end of the day, reliable and extensive builds translates to better quality products. That means quality in the functionality of the features which we’re building (the CI bit) and quality in the timely deliveries of said features to customers (the release bit). Being able to contribute to this has been extremely fulfilling.

checkThanks to Steve Haffenden, Ben Mackie, Niraj Bhawnani and Raymond Su for being such an awesome and dedicated team to work with. Special mention to Kenny MacLeod for jumping in during times of strife. You’re all my heroes!

If you are passionate about the same things as me then you’ll definitely find gratification in build engineering, even if it’s only for a few months.

Build engineers and developers: I’d love to hear your thoughts! Why do you get satisfaction out of work with builds? Leave a comment and share your experience!

Culture, quarantine, and bringing down Bamboo: my ...