The business case for continuous delivery

Follow the overall trend in the software industry: increasing the ability to react.

What emotions does the word "release" trigger in you? Relief? Elation? A fist-pumping sense of accomplishment? If your team is still living with manual testing to prepare for releases and manual or semi-scripted deploys to perform them, your feelings may be closer to "dread" and "blinding rage."

That's why software development is moving towards continuity. The recent emphasis on continuous integration, built-in testing, constant monitoring, and analytics feedback all point toward an overall trend in the software industry: increasing the ability to react. As organizations explore what these changes mean for them, they invariably discover continuous delivery, commonly known as CD.

Make no mistake: CD is not the exclusive domain of "unicorn" companies and tech darlings. Every team – from the humblest start-up to the stodgiest enterprise – can and should practice continuous delivery. This article takes a look at the business case for making this switch. It will discuss the work that lies ahead and the benefits it will yield, as well as the decisions and trade-offs needed to ship software using CD methodologies.

Every team can – and should – practice continuous delivery.

Top four business benefits of continuous delivery

Faster reaction times

The most obvious benefit of CD is that it allows quick reactions to stimuli, both external and internal – whether it's a market segment that takes a weird left turn, morphing that P5 feature into a company-saving imperative, or a researcher that uses your flagship application as an example of what-not-to-do at the security conference du jour. What business doesn't want that?

Reduced risk

For many organizations, there's a non-trivial amount of pageantry around shipping a release. And why shouldn't there be? New features are finally out to customers. Bugs are fixed. Everyone's happy, right? The dark secret in many organizations is that shipping a release takes a huge amount of effort on the part of the QA and build/release teams. Under a continuous delivery model, software is "released" constantly (though not necessarily to customers). Therefore, the ceremony and the risk around releasing is reduced. If you're relying on your release infrastructure daily, you will notice (and resolve!) its deficiencies much quicker than if the process is only undergone once every few weeks or months.

Exposed inefficiencies and costs

Whether it's the latest web service or classic ISO images, every software organization incurs a cost to ship the bits out the door. Often times, these costs aren't entirely clear to the business. There can be all sorts of forces that obscure how much it really costs to ship a release. Continuous delivery makes this process apparent not only to the business itself, but to the decision makers. When constructing a pipeline, it will become clear where humans are involved, what bottlenecks exist, and what low-hanging automation fruit there is to pick. And once constructed, the pipeline creates a virtuous cycle. Clear incentives for a healthy software delivery dynamic now exist, replacing a nebulous discontent with how long and costly it is to release.

Flexible release options

Moving to a continuous delivery model requires the build-out of infrastructure, both in an operational context and a software architecture context. But once that's done, it provides the business with more flexibility in how it delivers features and fixes. Specific sets of features can be released to specific customers, or released to a subset of customers, to ensure they function and scale as designed. Features can be tested and developed, but left dormant in the product, baking for multiple releases. Your marketing department wants that "big splash" at the yearly industry convention? With continuous delivery, it's not only possible, it's a trivial request.

Lightbulb

Continuous delivery isn't just for unicorns

Given all the business benefits, continuous delivery may seem almost magical. It's not.

It's just a shift in thinking about how to design, develop, and deliver software, followed by focused investment on the initiatives required to implement it. Where many organizations falter with their CD transformation is this first step: defining and committing to that investment.

Because CD requires a large overhaul in technical processes, operational culture, and organizational thinking, it can often feel like there's a large hurdle to getting started. The fact that it requires a hefty amount of change and build-out of areas of a company's software delivery infrastructure that may have been neglected over the years can make it an even more bitter pill to swallow. That's the bad news. The good news is that we humans are exceedingly adaptable, especially when incentivized by the prospect of a software release process that less closely resembles Hell.

The other piece of good news is that transforming any aspect of your business to be CD-ready will pay off in other ways too. So even if you can't get your company to back an all-encompassing CD transformation, there's still loads of value in transforming whatever you can.

The largest initial investment is an agreement that the transition toward CD is a worthy business goal. An important one. One which is not attainable without the participation of all, and one which will require some serious (but ultimately fruitful) trade-offs to be made during that transition period.

"Let's spend the next few months working on automated testing and build/release infrastructure, and redesigning how our application is written. We can postpone our feature development," said no product manager ever. Their outlook makes sense given their responsibilities to react to an ever-shifting market and the pressures to outmaneuver the competition.

The way to get product managers and business stakeholders on board is to show them how much faster they'll be able to speed new features to market once the investment in CD starts to bear fruit. Imagine a world in which PMs can ship ultra-lightweight versions of features, instrumented with event analytics, shortly after the idea pops into their heads. The analytics send back data on users' adoption of and competence in using that feature, which the PM can use to further refine it. This process takes months without CD. But using CD, the iteration cycles can be as short as a few days or weeks.

This isn't to say that all feature development must cease until CD is up and running. Rather, all areas of the business – product management, engineering, and engineering support (QA, ops, build/release) – must buy into the long-term benefits CD will bring. That part isn't hard. But real discipline is required to keep it top-of-mind to ensure the initiative doesn't stagnate or get hauled out to the proverbial scrap yard altogether.

And let's be real: there are other areas that will need investment in your pursuit of continuous delivery. And while CD may be the catalyst here, the changes will bring other benefits, too.

Automated testing

This includes not only a focus on unit tests, but integration and system tests. They need to be driven in an automated fashion, by a system like Bamboo. One of the hugest investments a company I worked with made was requiring an associated unit (and, where it made sense, integration) test for every bug – (every bug!) – in a particularly important software component. Regardless of where you are in implementing CD, these tests will help identify regressions immediately and contribute to shorter and less stressful release cycles right off the bat. Expect to work on this aspect for 6 - 18 months if you're starting from scratch.

Application architectural changes

Moving toward a service-oriented architecture is a must. If your software isn't SaaS, you'll need to work toward a component architecture with the necessary versioning and release engineering for the components to be individually shipped. So-called "feature flagging" and "canary releases" can then make use of these componentized services/modules. It pays for itself by decoupling these complex systems so their interfaces are better understood, which allows teams to release their components on their own cycles – plus you get CD-readiness on top of that. Most teams make this transition over 4 - 8 release cycles.

Build/test farm

Each checkin for the Firefox web browser triggers thousands of tests, requiring over 200 hours of CPU compute time – to say nothing of the build time. It took a notable hardware (and, in some cases, cloud) investment to be able to support this for each commit. Your current build/test farm will need enough capacity to support CD, which will also tighten your developer feedback loop and make it easier to test against different operating systems, browsers, etc. Adjusting your test farm capacity is an ongoing process, but cloud providers like Amazon Web Services can take a lot of the pain away.

Emphasis on culture

CD fundamentally requires some amount of cultural rewiring because the core structure of CD is a delivery pipeline through which changes flow, not a set of human-controlled gates that must be passed. This structure's very nature can help to start chipping away at organizational silos. So it's up to the CD cheerleaders to make sure the people in those silos are prepared for that. Otherwise, it can feel like their silo is coming down on top of their heads. (And they'll understandably fight to do everything to keep it from doing so!) This is another ongoing process, the most intense period of which will last about 2 - 6 months.

Focused staffing

Getting traction on CD transformation may require hiring an engineer or consultant whose sole focus is the infrastructure work required to move to CD, which is also a great excuse to bring fresh expertise onto the team. If current engineers can't be insulated enough to make the needle move, the additional hire allows the organization to move forward, and as their work comes together, help the team transition to the brave new CD world. Even a temporary consultant can provide breathing room to make the culture changes easier, so the best contract duration is also about 2 - 6 months.

Starting your CD journey

Continuous delivery requires change from teams across the organization. It requires focused investment in tooling, hardware, and people. All of that takes time. And all of it is possible.

The first step is acknowledging that trying to transform to continuous delivery "on the cheap," as part of someone's "20% time", or "in our slack time between sprints and releases" is a recipe for spending time (and therefore money) on failure. But viewing it as an investment that will pay future dividends in making the business able to react – in real time – to the dynamic world that is the software industry, which should help offset the cost of that investment on the balance sheet. And if you factor in the employees that won't burn out due to "midnight oil" releases, and people who get to work on making their own worlds better instead of propping up a building that's constantly falling apart, the value becomes obvious.

Posted by J. Paul Reed

8 min read