When approaching a GitOps workflow you’ll soon have to choose between push- and pull-based approaches. In this blog post I’ll explain the high-level differences of each approach with pros and cons.
What is GitOps?
GitOps stands for Git Operation: in this workflow all the infrastructure configurations are stored in a Git repository, which represents the source of truth of the configuration. When you need to modify something, you commit changes to your repository, and once changes have been pushed you need to apply them to the target environment. And here’s when the difference between push and pull based approach enters the room.
A common misunderstanding is that push and pull are related to the Git actions of those names, but in reality git pull and git push have nothing to do with the deployment and it’s not relevant whether you push directly on the main branch, open a pull request waiting for human confirmation, use a test branch, or whatever else you feel like doing in the deployment workflow.
To start, let’s suppose we have an OpenShift cluster to manage in a GitOps way.
The Push-based Approach
In a push-based approach there is an external entity, usually a CI/CD job, which reacts to the changes and “pushes” them into the production environment. For example we could have a Jenkins job which:
- Receives a webhook from Bitbucket when changes are pushed
- Pulls the proper branch
- Executes a command (for example an
oc apply) to deploy the changes
This will achieve the goal of reflecting the changes in the Git repository onto the target system, but has several drawbacks:
- Security: An external entity (i.e. the CI) is required to deploy changes, and needs privileged or even superadmin access to your cluster.
- Self-Healing: If something goes wrong during the deployment the system does not automatically roll back to the previous stable configuration, unless something specific is implemented in the pipeline. Furthermore, if some change is applied automatically, it will not be reverted to the Git state.
- Scalability: If you have multiple clusters to manage, you’ll have to extend your CI accordingly.
Pull-based approach
In a pull-based approach it’s the target system itself which detects changes on the Git repository and deploys them. As usual there is no free lunch: you need something to perform these actions like Flux or ArgoCD. In our cluster we use Argo.
This brings several advantages with respect to the CI/CD job:
- Security: You don’t have to store privileged credentials outside the cluster.
- Self-Healing: Argo can detect issues and automatically roll back to the previous working version, furthermore manual changes are no longer possible (without disabling self healing) so that you can be sure that the repository is really the only source of truth.
- Scalability: ArgoCD is installed within the cluster and can manage everything automatically after the initial setup, including its own updates. This means that the system is decentralized and, for example, your testing environment is completely independent from your production environment.
Of course there are still limitations, for example you may not notice a misconfiguration immediately, and you may not have full control over timing: ArgoCD periodically checks for changes and applies them, but the process may take some minutes and in the meantime you don’t have any feedback. So you’ll need to introduce ArgoCD monitoring to promptly detect issues and out of sync configurations in order to be sure that changes are deployed properly.
Conclusion
In conclusion a pull-based approach seems preferable, and in general it is. Unfortunately this is doable only when you have tools like ArgoCD which takes care of the whole deployment cycle. Unfortunately there are several cases in which such tools are not available: for example with RPM packages installation or more in general, handling environments different than Kubernetes.
In those cases you still need a CI/CD or an equivalent tool like Ansible Automation Platform to apply changes and still be able to rely on a push-based approach. In those environments it’s the DevOps engineer’s responsibility to implement check and rollback automation.
These Solutions are Engineered by Humans
Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth Phoenix.
Alessandro Valentini
Author
