At Atlassian we believe in eating your own dogfood. We’ve therefore setup Bamboo to continuously build and integrate our own products.
Over time we keep adding new types of builds (plugins, tools, etc) and existing builds start taking more time (number of testcases increases, more CPU intensive testcases, etc). We therefore decided to setup a new Bamboo box dedicated to running all the JIRA automated tests on a number of different JDKs and app servers. Some of the stats from this new box were pretty amazing and worth sharing here.
On our old build box all we could really afford was to run one functional test build of JIRA for the Enterprise edition. This build usually took around 70-100 mins to complete. Pretty slow really and we were seriously considering cleaning up our TestHarness to make it run faster.
On the new box, all functional tests run in about 20 mins . It’s a pretty late model quad core Xeon. Even with 3-4 functional builds running in parallel the build times didn’t increase much more than 1-2 minutes for each build. Pretty impressive. All of JIRA compiles in about 10-15 seconds. The Unit tests take about a minute to run in total.
One could argue that spending 3+ days trying to clean up the JIRA TestHarness such that it performs better is a complete waste of time, as we can simply buy a faster box again in 2-3 years time. Question is I guess what costs more: A new box or 3+ days of development with no visible benefit at all for customers.
The other bit that I found interesting was the performance increases between JDK 1.4, JDK 1.5 and JDK 1.6:
- JIRA Functional Tests for Enterprise JDK 1.4: 25 mins
- JIRA Functional Tests for Enterprise JDK 1.5: 19 mins
- JIRA Functional Tests for Enterprise JDK 1.6: 16 mins
Note: For JIRA we have 2 automated test suites, Unit tests (about 2600 of them) and Functional tests (about 430). Functional tests startup an instance of JIRA and make HTTP requests against this instance testing various parts of the UI.