What is Continuous Integration
Build your team's agility with faster feedback. Because you only move as fast as your tests.
Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process is comprised of automatic tools that assert the new code’s correctness before integration. A source code version control system is the crux of the CI process. The version control system is also supplemented with other checks like automated code quality tests, syntax style review tools, and more.
Continuous integration articles
Importance of CI
In order to understand the importance of CI, it’s helpful to first discuss some pain points that often arise due to the absence of CI. Without CI, developers must manually coordinate and communicate when they are contributing code to the end product. This coordination extends beyond the development teams to the rest of the organization, as well. Product teams must coordinate when to sequentially launch features and fixes and which team members will be responsible.
The communication overhead of a non-CI environment can become a complex and entangled synchronization chore, which adds unnecessary bureaucratic cost to projects. This causes slower code releases with higher rates of failure, as it requires developers to be sensitive and thoughtful towards the integrations. These risks grow exponentially as the engineering team and codebase sizes increase.
Without a robust CI pipeline a disconnect between the engineering team and the rest of the org can form. Communication between product and engineering can be cumbersome. Engineering becomes a black box which the rest of the team inputs requirements and features and maybe gets expected results back. It will make it harder for engineering to estimate time of delivery on requests because the time to integrate new changes becomes an unknown risk.
What CI does
CI helps to scale up headcount and delivery output of engineering teams. Introducing CI to the aforementioned scenario allows software developers to work independently on features in parallel. When they are ready to merge these features into the end product, they can do so independently and rapidly. CI is a valuable and well-established practice in modern, high performance software engineering organizations.
How CI can be used
CI is generally used alongside an agile software development workflow. An organization will compile list of tasks that comprise a product roadmap. These tasks are then distributed amongst software engineering team members for delivery. Using CI enables these software development tasks to be developed independently and in parallel amongst the assigned developers. Once one of theses tasks is complete, a developer will introduce that new work to the CI system to be integrated with the rest of the project.
CI vs Continuous Deployment vs Continuous Delivery
Continuous integration, deployment, and delivery are three phases of an automated software release pipeline. These three phases take software from idea to delivery to the end user. The integration phase is the the first step in the process. Integration covers the process of multiple developers attempting to merge their code changes with the master code repository of a project.
Continuous Delivery is the next extension of continuous integration. The delivery phase is responsible for packaging an artifact together to be delivered to end users. This phase runs automated building tools to generate this artifact. This build phase is kept ‘green,’ which means that the artifact should be ready to deploy to users at any given time.
Continuous Deployment is the final phase of the pipeline. The deployment phase is responsible for automatically launching and distributing the software artifact to end users. At deployment time, the artifact has successfully passed the integration and delivery phases. Now it is time to automatically deploy or distribute the artifact. This will happen through scripts or tools that automatically move the artifact to public servers or to another mechanism of distribution, like an app store.
Benefits and challenges of continuous integration
CI is a valuable asset to a software producing organization. CI benefits are not limited to the engineering team but greatly benefit the overall organization. CI enables better transparency and insight into the process of software development and delivery. These benefits enable the rest of the organization to better plan and execute go to market strategies. The following are some of the overall organizational benefits of CI.
CI enables organizations to scale in engineering team size, codebase size, and infrastructure. By minimizing code integration bureaucracy and communication overhead, CI enhances agile workflows. It allows each team member to own a new code change set through to release. CI enables scaling by removing any organizational dependencies between development of individual features. Developers can now work on features in an isolated silo and have assurances that their code will seamlessly integrate with the rest of the codebase.
Improve the feedback loop
Faster feedback on business decisions is another powerful side effect of CI. Product teams can test ideas and iterate product designs faster with an optimized CI platform. Changes can be rapidly pushed and measured for success. Bugs or other issues can be quickly addressed and repaired.
Overall engineering communication and accountability is improved with CI. By introducing pull request work flows tied to CI, developers gain passive knowledge share. Pull Requests allow Developers to observe and comment on code from other team members. Developers can now view and collaborate on feature branches with other developers as the features progress through the CI Pipeline. CI can also be used to help QA resource expense. An efficient CI pipeline with high-confidence automated test coverage will safeguard from regressions and ensure that new features match a specification. Before new code is merged it must pass the CI test assertion suite which will prevent any new regressions.
The benefits of CI far outweigh any challenges in adoption. That said, it is important to be aware of the challenges of CI. The real challenges of CI arise when transitioning a project form no CI to CI. Most modern software projects will adopt CI from early inception stages and alleviate the challenges of later adoption.
Adoption and installation
The challenges of continuous integration are primarily around team adoption and initial technical installation. If a team doesn't currently have a CI solution in place, it can require some effort to pick one and get started. Thus, considerations need to be made around the existing engineering infrastructure when installing a CI pipeline.
Technology learning curve
CI functionality comes with a list of supportive technologies that may be learning curve investments for the team to undertake. These technologies are version control systems, hosting infrastructure, and orchestration technologies.
CI best practices
Test Driven Development
Once a project has established a CI pipeline with automatic test coverage, it is a best practice to constantly develop and improve the test coverage. Each new feature coming down the CI pipeline should have an accompanying set of tests to assert that the new code is behaving as expected.
Test Driven Development (TDD) is the practice of writing out the test code and test cases before doing any actual feature coding. Pure TDD can closely involve the product team to help craft an expected business behavior specification, which can then be transformed into the test cases. In a pure TDD scenario, developers and product team will meet and discuss a spec or list of requirements. This list of requirements will then be converted into a checklist of code assertions. The developers will then write code that matches these assertions.
Pull requests and code review
Most modern software development teams practice a pull request and code review workflow. Pull requests are a critical practice to effective CI. A pull request is created when a developer is ready to merge new code into the main codebase. The pull request notifies other developers of the new set of changes that are ready for integration.
Pull requests are an opportune time to kick off the CI pipeline and run the set of automated approval steps. An additional, manual approval step is commonly added at pull request time, during which a non-stakeholder engineer performs a code review of the feature.. This allows for a fresh set of eyes to review the new code and functionality. The non-stakeholder will make edit suggestions and approve or deny the pull request.
Pull requests and code review are a powerful tool to foster passive communication and knowledge share among an engineering team. This helps guard against technical debt in the form of knowledge silos, where specific engineers are the only stakeholders for certain features of a code base.
Optimize pipeline speed
Given that the CI pipeline is going to be a central and frequently used process, it is important to optimize its execution speed. Any small delay in the CI workflow will compound exponentially as the rate of feature releases, team size, and codebase size grows. It is a best practice to measure the CI pipeline speed and optimize as necessary.
A faster CI pipeline enables a faster product feedback loop. Developers can rapidly push changes and experiment with new feature ideas to help improve the user experience. Any bug fixes can be quickly patched and resolved as discovered. This increased execution speed can offer both an advantage over other competitors and an overall higher-quality experience to your customers.
Getting started with continuous integration
The foundational dependency of CI is a version control system (VCS). If the target code base for a CI install does not have a VCS, step one is installing a VCS. The absence of a VCS should be very unlikely on modern codebases. Some popular VCSs are Git, Mercurial, and Subversion.
Once version control is in place, finding a version control hosting platform is the next move. Most modern version control hosting tools have support and features built in for CI. Some popular version control hosting platforms are Bitbucket, Github, and Gitlab.
After version control has been established on the project, integration approval steps should be added. The most valuable integration approval step to have in place is automated tests. Adding automated tests to a project can have an initial cost overhead. A testing framework has to be installed, then test code and test cases must be written by developers.
Some ideas for other, less expensive CI approval mechanisms to add are syntax checkers, code style formatters, or dependency vulnerability scans. Once you have a version control system setup with some merge approval steps in place, you’ve established continuous integration!
CI is not purely an engineering specific business process. The rest of the organization, marketing, sales, and product teams will also benefit from a CI pipeline. Product teams will need to think how to parallelize execution of simultaneous streams of development. Product and engineering will work closely to determine the qualifying business functionality expectations that will make up the automated test suite.
Marketing and sales will be able to reference the CI pipeline to coordinate with customer facing communications efforts and events. CI gives a level of transparency to the rest of the organization on how engineering execution is progressing. This transparency and communication utility integrates gracefully with an agile project development workflow.
If your organization has a multiple-developer software team and is not using CI, this is an opportunity to truly enhance your team’s quality of life. CI will help your engineering org execute quicker and more effectively. Most software developers expect an organization to have CI established, so when it comes to acquiring and retaining best-in-class developers, CI is certainly worth the investment.
CI is a standard fixture of modern high efficiency software development organizations. The best companies have robust CI pipelines and don’t think twice about further efficiency investments. The benefits of CI are not limited to the engineering team and applicable to the whole organization.
Many 3rd party tools exist to aid in CI management and installation.There are many options to choose from. Some popular options are Codeship, Bitbucket Pipelines, SemaphoreCI, CircleCI, Jenkins, Bamboo, Teamcity and many others. These tools have their own in depth setup guides and documentation to help get started.
Some of the best CI tools are provided by Atlassian. Bitbucket pipelines and Bamboo are great utilities to bring a project up to speed with modern CI features. Jira is one of the world's most popular agile project management tools. Jira tightly integrates with other Bitbucket projects and when coupled with a CI pipeline, can give a very transparent view into execution health of an organization.