Is GitOps the next big thing in DevOps?
Many organizations now see DevOps as part of their digital transformation strategy, since it encourages a culture of shared responsibility, transparency, and faster feedback. Yet as the gap between development and operations teams shrinks, so do the processes.
So it goes with Git, the most widely used version control system in the world today. As companies embrace DevOps methodologies, so to the tools, which has created an evolution to GitOps, a set of practices that allow developers to perform more IT operations-related tasks.
What is GitOps?
At its core, GitOps is code-based infrastructure and operational procedures that rely on Git as a source control system. It’s an evolution of Infrastructure as Code (IaC) and a DevOps best practice that leverages Git as the single source of truth, and control mechanism for creating, updating, and deleting system architecture. More simply, it is the practice of using Git pull requests to verify and automatically deploy system infrastructure modifications.
In addition to Git as a key DevOps mechanism, GitOps is also used to describe tools that augment Gits default functionality. These tools were primarily used with operating models for Kubernetes-based infrastructure and applications. There is ongoing development and discussion within the DevOps community to bring GitOps tools to other non-Kubernetes platforms, such as Terraform.
GitOps ensures that a system’s cloud infrastructure is immediately reproducible based on the state of a Git repository. Pull requests modify the state of the Git repository. Once approved and merged, the pull requests will automatically reconfigure and sync the live infrastructure to the state of the repository. This live syncing pull request workflow is the core essence of GitOps.
The history of GitOps
Git is a mission critical tool for software development that enables pull request and code review workflows. Pull requests promote visibility into incoming changes to a codebase and encourages communication, discussion, and review of changes.
Pull requests are a pivotal feature in collaborative software development and changed the way teams and businesses build software. Pull requests bring transparency and measurability to a formerly opaque process. Git pull requests helped enable the evolution of DevOps processes into software development. System administrators, who typically were hesitant to change, are now embracing new software development practices such as agile and DevOps.
Systems administration as a craft has a sloppy history. System administrators previously would manage hardware manually by either connecting to and provisioning machines in a physical server rack or over a cloud provisioning API. In addition to the manual provisioning process, large amounts of manual configuration work was a regular routine. Administrators would keep custom collections of imperative scripts and configurations, cobble them together, and place them in various places. These scripts could break at any time or get lost. Collaboration was challenging as the custom tool chains were not regularly documented or shared.
The DevOps movement arose from this primordial swamp of systems administration. DevOps borrowed the best ideas from software engineering and applied them to systems administration, where the cobbled-together tools became version-controlled code.
IaC is one of the biggest revelations of DevOps. Previously system administrators favored custom imperative scripts to configure systems. Imperative software follows a sequence of steps to achieve a desired state, such as:
Imperative software is often error prone and easy to break by changing the sequence of events. Modern software development has trended away from imperative patterns and towards declarative software patterns.
Declarative software follows a declaration of an expected state instead of a sequence of commands. Here’s a comparison of imperative vs declarative devops statements.
While the imperative statements might read:
- Install an operating system on this machine
- Install these dependencies
- Download code from this URL
- Move the code to this directory
- Do this 3 times for 3 other machines
The declarative version of this would simply read: 4 machines have software from this URL, installed at this directory.
IaC encourages and promotes declarative system administration tools over custom imperative solutions. This led to the emergence of technologies like Docker Containers, Ansible, Terraform, and Kubernetes, which utilize static declarative configuration files. Human readability and consistent reproducible state are the beneficial outcomes. These configuration files were naturally added to Git for tracking and review. This is close but not quite GitOps.
Many of the traditional system administration problems have been solved at this point in DevOps history. Configuration files and tools are now stored in a central location, documented and accessible by many team members. Commits and pull requests were used to track modifications to the configuration and foster collaboration discussion and review. The only remaining problem with this stage is that the configuration still feels disconnected from the live system. Once a configuration pull request is approved and merged to the repo, the live system is manually updated to match the state of the static repo. This is the exact problem GitOps solves.
The GitOps idea was first hatched and shared by WeaveWorks, an enterprise Kubernetes management firm and has since proliferated throughout the DevOps community. GitOps is an extension of IaC and declarative configuration discussed above. GitOps adds some magic to the pull request workflow that syncs the state of the live system to that of the static configuration repository.
The benefits of GitOps
GitOps shares many of the same benefits as an agile feature branch software development workflow. The first major benefit is ease of adoption due to the usage of common tools. Git is the de facto standard of version control systems and is a common software development tool for most developers and software teams. This makes it easy for developers familiar with Git to become cross functional contributors and participate in GitOps.
Using a version control system lets a team track all modifications to the configuration of a system. This gives a “source of truth” and valuable audit trail to review if something breaks or behaves unexpectedly. Teams can review the GitOps history and see when a regression was introduced. Additionally this audit trail can be used as a reference for compliance or security auditing. The GitOps history can be used as proof when things like encryption certificates are modified or updated.
GitOps brings transparency and clarity to an organization's infrastructure needs around a central repo. Containing all systems configurations in a central repository helps scale contribution input from team members. Pull requests made through hosted Git services like Bitbucket have rich tools for code review and discussion commentary. This builds passive communication loops that allows the full engineering team to observe and monitor infrastructure changes.
GitOps can greatly increase productivity for a DevOps team. It allows them to quickly experiment with new infrastructure configurations. If the new changes don’t behave as expected, a team can use Git history to revert changes to a known good state. This is incredibly powerful since it enables the familiar “undo” functionality in a complicated infrastructure.
How GitOps works
GitOps procedures are performed by an underlying orchestration system. GitOps itself is an agnostic best practice pattern. Many popular GitOps solutions today primarily use Kubernetes as the orchestration system. Some alternative GitOps tool sets are coming to market that support direct Terraform manipulation.
To achieve a full GitOps install, a pipeline platform is required. Jenkins, Bitbucket Pipelines, or CircleCi are some popular pipeline tools that are complementary to GitOps. Pipelines automate and bridge the gap between Git pull requests and the orchestration system. Once pipeline hooks are established and triggered from pull requests, commands are executed to the orchestration piece.
A new pattern or component that is specifically introduced with GitOps is the GitOps “operator,” which is a mechanism that sits between the pipeline and the orchestration system. A pull request starts the pipeline that then triggers the operator. The operator examines the state of the repository and the start of the orchestration and syncs them. The operator is the magic component of GitOps.
Imagine a team identified a performance bottleneck or a spike in traffic and the team notices the load balancer is not working as expected. They look into the GitOps repo that holds the infrastructure configuration and find a specific file that configures and deploys the load balancer. They can review it on their online Git hosting site. After some review and discussion they identify that some of the configuration values for the load balancer are not optimal and need to be adjusted.
A member of the team opens up a new pull request that optimizes the load balancer values. The pull request is reviewed and approved by a second team member and merged into the repository. The merge kicks off a GitOps pipeline, which triggers the GitOps operator. The operator sees the load balancer configuration was changed. It confirms with the systems orchestration tool that this does not match what is live on the teams cluster.
The operator signals the orchestration system to update the load balancer configuration. The orchestrator handles the rest and automatically deploys the newly configured load balancer. The team then monitors the newly updated live system to see it return to a healthy state. This is an ideal GitOps scenario. Let’s expand on it further to demonstrate GitOps utility.
Let’s imagine that instead of slightly tweaking the load balancer values to be more optimal, the team makes an aggressive decision to deploy an entirely new load balancer type. They feel the current load balancer is fundamentally flawed and want to try an alternative option. The workflow is the same as the value tweak. The team creates a pull request that introduces an entirely new load balancer configuration and deletes the old configuration. It is approved and deployed through the pipeline.
Unfortunately the team finds that this new type of load balancer is incompatible with some other services within their cluster. The new load balancer causes critical traffic failures and halts user operations. Luckily because the team has a complete GitOps pipeline they can quickly undo these load balancer changes. The team will make another pull request that reverts the repository back to the old known functional load balancer. This again will be noted by the GitOps pipeline and automatically deployed. It will rapidly improve the infrastructure and improves the reliability score of the team.
GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure. Though primarily focused on Kubernetes cluster management, the DevOps community is applying and publishing GitOps solutions to other non-Kubernetes systems. GitOps can bring many benefits to an engineering team including improved communication, visibility, stability, and system reliability. One of the core requirements for a GitOps experience is a modern hosted Git platform like Bitbucket.