With the JIRA Bugfix effort using Kanban, we track a metric called Lead Time, which in our case is the period of time between the moment Atlassian decides to implement a fix (or feature) to the time it is made available to the customer. Adapting Kaizen, the culture of continous improvement, we look for waste and strive to eliminate it with the goal of reducing this Lead Time. One area that was highlighted for improvement was the Continuous Integration feedback loop. A metric that we found useful in measuring this was Build Lead Time, the period of time between a developer checking in his change into source control and receiving a green CI build. There are various factors that impact this metric including:

  • Compilation / test failures
  • Number of developers / checkins which increases the risk of compilation / test failures
  • Infrastructure issues
  • Flaky tests
  • Hung builds

Another factor that impacts Build Lead Time is Build Queue Time. This is the period of time between a build kicking off and for build capacity to become available to run that particular build. Ideally, the build should start immediately, but this is not always the case due to various factors including waiting for Elastic Agents to start or waiting for an Agent with the correct capabilities to finish a previous build before starting the queued build. While trying to improve the Build Lead Time, we noticed that it was difficult to retrospectively assess the reasons why there were large Build Queue Times. We were also slow in acting on hung builds which also resulted in an increase in Build Lead Time. So we’ve developed a Wallboard specifically to track the agent & build queue health of our Bamboo instance. At a glance it shows:

  • Number of idle, busy & hung agents
  • Elastic instances starting & shutting down
  • How many builds are queued & how long they have been waiting for, highlighting builds that have been queued for a long period of time

The colours used on the Wallboard include:

  • Blue indicating idle or starting capacity
  • Orange indicating builds in progress
  • Red indicating immediate attention is required, for example a hung build or a build that has been queueing for a long period of time

Here are some screenshots of the Wallboard running internally on the JIRA team’s instance of Bamboo. The benefit of the Wallboard is that it gives visibility to the whole team as to how often all our agents are in use. In this example we can see our Bamboo instance being underutilised:

AgentSmithNormal.jpg

Here are 5 builds have been waiting to start for 28 seconds:

AgentSmithSmallBuildQueueUpdated.png

This example makes it quite obvious to the team that the agents are in high use and that we’re waiting for more Elastic Agents to start. The queued builds are red to show that they have been waiting for over 10 minutes and are now considered overdue as they are now greatly impacting Build Queue Time and the Build Lead Time:

AgentSmithLargeOverdueQueue.jpg

Here is a hung build requiring attention:

AgentSmithHungBuild.jpg

By providing a realtime visualisation we’ve found that we can quickly respond to issues that greatly impact Build Lead Time & Build Queue Time.

If you’re interested in trying out this plugin for yourself, you can find it on the Plugin Exchange here

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now