Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this post we want to share how the JIRA development team at Atlassian has accomplished this using Build Stages in Bamboo.
Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it to one day. Additionally, we have taken functional testing time from 8 hours to 40 minutes and use it as an inner loop Continuous Integration feedback build.
There are two primary feedback loops that the JIRA development team utilizes:
- Generic “CI” Build – developers get feedback on their code changes quickly.
- Tested Platform Matrix – feedback about how our release is validated against different types of distributions and the different environments they can run on, which include different operating systems and database configurations.
What did you do before Bamboo Stages?
Running our tests in serial, the CI build would take 8 hours to run our functional test suite, which consisted of around 3000 tests. However, we only did this to deploy the JIRA artifacts to Maven because it is important to run all of our tests prior to deployment. This is key for JIRA artifact consumers to make sure they can pin point true integration failures instead of tracking down failures within JIRA. In addition to this build, in order to get faster feedback, we had 15 different build plans to run our functional tests but had no way to act upon those results. Finally, we also ran a unit test build as a fast CI loop that ran in 15 minutes, however only providing unit level testing. We had two builds for functional testing, one for validating deploying artifacts to Maven and the other to give us a faster feedback cycle with those tests. This meant we were potentially testing the same changeset twice and burdening our build hardware. For the Maven deployment functional test build that took 8 hours, at maximum, we could deploy our artifacts once a day. Sometimes this deployment would be delayed two or three days if those tests stayed in a failed state. The Tested Platform Matrix build would take an entire week and might not pass within that week. For most, if not all development teams, those feedback times are unacceptable.
What are we trying to run that takes 8 hours of serial execution?
- 12,567 JUnit Tests
- 3020 Functional Tests
- 639 Selenium UI Web Browser Tests
By relying on an 8 hour development testing cycle and a week release testing cycle, we added higher risk in our ability to release JIRA on time. Alot of time was spent figuring out what commit broke what test, and sometimes what day it happened. We needed help with running our tests in parallel and we needed to do it on a massive scale in order to shorten these long feedback loops. Bamboo stages allow us to run more than one build “job” within a stage and more than one stage in a build plan. The JIRA development team has done work to break the test suite into smaller parts, called batches, to launch each of these batches as a job within a stage. Of course, we need somewhere to execute all of these jobs in parallel. This is where another important capability of Bamboo helps us out with the ability to run multiple remote agents on different types of hardware.
The process is not done yet. Once our CI Build passes and “goes green”, what if we wanted to…
- deploy these tested artifacts to Maven?
- deploy our tested JIRA distribution to QA Environments for manual testing?
- build other JIRA Plugins so they can consume and validate against these tested artifacts?
- run more tests, specifically our Tested Platforms Matrix?
We’d like to ensure that we can build and promote through these phases in the following diagram going from the center left in development to release validation on the outside of the circles. The two primary loops are signified in yellow for development testing in our CI build and blue for release testing in the Tested Platforms Matrix and other important consumers of JIRA.
Bamboo Stages now help us to solve our goals
Like many other development teams, the JIRA team wants to do all of these great tasks and accomplish these goals. The focus is to have our development CI loop as the producer bridge to launch into other outer loop CI activities. However, we can’t act until we can aggregate all of the jobs that run our functional tests into one result, to decide if we want to proceed further into greatness. This is the other key feature in Bamboo stages.
Previously there were 15 functional test build plans all running on their own after a dependent parent build triggered from commit detections. However, it was impossible to launch other build plans against the results of these 15 builds. Because of this, we would not be able to deploy our tested Maven artifacts from those results. Now, with stages, our CI build plan can not only have these same 15 jobs but we deploy the artifacts from that commit revision to Maven if and only if all of the functional test jobs pass. Additionally, Bamboo Stages also makes adding jobs to each stage an easy process. This is important because we plan on growing our functional test batches from our current set of 20 to 40 with a goal of having each job execute in less time.
Here is a screenshot of JIRA Development teams CI build plan showing the use of stages on a large scale.
Next we modified our other child build plans, who needed to consume JIRA artifacts, to depend upon the result of the CI build plan. For example, the final stage in the CI plan is to deploy our JIRA artifacts to Maven only if all the previous stages’ jobs passed. Child plans dependent on our CI build, then build and validate important plugins. Some of these, like the the JIRA Fisheye, JIRA Workflow and Activity Streams plugins, then build against the JIRA artifacts that were just deployed in the parent CI plan only if all functional tests passed. All this was done with the use of Bamboo Stages. Additionally, another child plan of our CI plan, then deploys the JIRA Standalone distribution to a clean test remote environment for QA manual testing. The best news of all is that the CI build now executes all functional and unit tests on average in 40 minutes and we have ambitions to make this a 10 to 20 minute build in its entirety.
Our Tested Platforms Matrix is the second set of longer running tests that will drive us closer to release readiness. We run exactly the same functional tests in this set of builds. However, the difference here is we run against each distribution that we allow customers to download against most of our supported environments.
What does this mean?
- 4 distributions – Standalone, War, Source, and a Windows Installer.
- 3020 functional tests that were previously defined
- MySql, Postgres, MS Sql Server, and Oracle databases that, except SQL Server, run on two operating system platforms, Linux and Windows.
|Postgres||MySql||Oracle||MSSQL Server 2005||MSSQL Server 2008|
Adding up all of the cells in the Tested Platforms Matrix shows us that we execute 24,160 functional tests.
These 24,160 functional tests are executed against the same Maven artifacts that were deployed from any previously passing CI Build. We can use staging again to help organise the number of jobs launched against the matrix and also aggregate results. Because of our ability to deploy JIRA artifacts through our development CI build, we can be assured our Tested Platforms Matrix builds run at least once a day. By using staging, the execution of the matrix can help us complete the entire set of build plans in 5 hours. This allows us to have faster assurance from the Tested Platforms Matrix that we are ready promote what we created from a passing build in CI to make these same artifacts be release candidate artifacts for customer consumption.
Bamboo stages have been an ultimate win in condensing the amount of time to execute our functional tests in JIRA. It has given us the ability to not only help us launch our functional tests in parallel but also to importantly aggregate the results of those tests into a single pass fail result. This result has enabled us to then, with confidence, deploy and publish our JIRA artifacts for further testing in validation in JIRA and other products that depend on JIRA.
Further enhancements can be made to what I have shown you. We can use Bamboo’s artifact passing in Bamboo 3.0 to build once and re-use many times so we can pass the JIRA artifacts from one stage to the next. This means that all of the functional test batches I described will no longer have to checkout JIRA source and compile it. They can re-use the passed artifacts and give us the purity of testing against the exact same artifacts that were built. By using this re-use feature we also can trim even more time from both our CI build and our Tested Platforms Matrix! I’ll let you know how that goes!