Kubernetes ClusterAPI + ArgoCD = easy end-to-end declarative GitOps for platform teams

TL;DR: Combining ClusterAPI with ArgoCD (both to finish bootstrapping the clusters as well as to manage the operator’s manifests) is a game-changer for Kubernetes platform teams to more easily provision and manage all their clusters via declarative GitOps in an end-to-end and consistent way — even across hybrid and/or multi-cloud environments.

There are two common patterns that really elevate the Kubernetes and Cloud Native ecosystem beyond ‘just’ container orchestration — operators to extend its API to know how to manage nearly anything as well as GitOps for true continuous deployments (CD). These have led to Kubernetes now being the standard foundation for building internal platforms for platform engineering teams in many organisations.

Operators extend the Kubernetes API to be able to do things like understand how to deploy a PostgreSQL database on the cluster. And GitOps (via common tools ArgoCD and Flux) have flipped deployments on their head — with clusters now watching git repos and pulling changes in as soon as they are merged rather than a pipeline having to push the changes with imperative CLI commands.

GitOps workflow

Cluster API — an operator to deploy Kubernetes clusters from Kubernetes

A Kubernetes platform team inevitably ends up running many different Kubernetes clusters for different teams or services within their organisations. Often those are provisioned by tools like Terraform or AWS CloudFormation/CDK or MS Azure ARM/Bicep. But this differs from cloud to cloud (even Terraform needs different providers that are very different from cloud to cloud) — and so it doesn’t really facilitate hybrid or multi-cloud scenarios seamlessly.

The Cluster API operator is here to solve that. You have one management Kubernetes cluster whose job is to provision and manage your other clusters. You give it declarative YAML manifests saying what you want each cluster to look like and it actions them — and it can do it across everything from vmware or KVM VMs to the major cloud providers.

ClusterAPI workflow

Adding in ArgoCD GitOps as the last step in the Cluster API bootstrap

One of the things you can ask ClusterAPI to do is deploy Helm Charts. And, since ArgoCD is well packaged in Helm charts, you can use that capability to not just finish the initial bootstrap but also to initialise the ongoing GitOps for your cluster as the final step.

Example of this all in action

In this example we are using ClusterAPI to provision a GKE Autopilot cluster, install ArgoCD and then have that install some workloads.

NOTE: This would work just as well with EKS on AWS or AKS in Azure — but I have some GCP credits (check out their generous Innovators Plus program to get some too) so built my lab environment there.

The same management cluster could even provision and manage clusters in all three simultaneously. Or on bare VMs instead of the more managed GKE/EKS/AKS. See all the full list of ClusterAPI providers here.

The example is on GitHub here.

Really all there is to it once you’ve set up ClusterAPI on a K8s cluster is to give it this YAML file:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capi-gke
  namespace: default
  labels:
    argoCDChart: enabled
spec:
  controlPlaneRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: GCPManagedControlPlane
    name: capi-gke-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: GCPManagedCluster
    name: capi-gke
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPManagedCluster
metadata:
  name: capi-gke
  namespace: default
spec:
  network:
    name: default
  project: project-435400
  region: australia-southeast1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPManagedControlPlane
metadata:
  name: capi-gke-control-plane
  namespace: default
spec:
  location: australia-southeast1
  project: project-435400
  enableAutopilot: true
  releaseChannel: stable
---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: argocd
spec:
  clusterSelector:
    matchLabels:
      argoCDChart: enabled
  repoURL: https://argoproj.github.io/argo-helm
  chartName: argo-cd
  namespace: argocd
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: demos
spec:
  clusterSelector:
    matchLabels:
      argoCDChart: enabled
  repoURL: https://argoproj.github.io/argo-helm
  chartName: argocd-apps
  valuesTemplate: |
    applications:
      demos:
        namespace: argocd
        finalizers:
        - resources-finalizer.argocd.argoproj.io
        project: default
        sources:
          - repoURL: https://github.com/jasonumiker/gke-autopilot-capi-argocd-example.git
            path: demos
            targetRevision: HEAD
        destination:
          server: https://kubernetes.default.svc
          namespace: default
        syncPolicy:
          automated:
            prune: true
            selfHeal: false
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true

NOTE: Since this is a GKE Autopilot the Kubernetes control plane version upgrades are automatic and I can’t pin a version here. If this was another Kubernetes (including a non-Autopilot GKE) I could (and really should) specify/pin the K8s version in the GCPManagedControlPlane object. Then not only would this deploy the specific version I ask for initially but also allow me to trigger an upgrade when I want to by incrementing that controlPlaneVersion parameter and letting the Cluster API action that for me.

First ClusterAPI provisions our cluster:

ClusterAPI provisioning our GKE Autopilot Cluster

NOTE: The default naming convention is [K8s namespace of our manifest]-[our GCPManagedControlPlane manifest’s name]. So, in this case, we put the manifest in the default namespace and called our cluster capi-gke-control-plane.

Then after ArgoCD is installed by ClusterAPI it provisions our workloads completing the full cluster setup:

ArgoCD doing its GitOps thing

And it isn’t just doing this as a one-off — any future changes merged to this git repo will get deployed by this Argo to this cluster going forward as well.

And we can even GitOps the ClusterAPI manifests too!

We can even take this all one step further and install ArgoCD on the ClusterAPI management cluster. Then we don’t need to run a kubectl apply -f manifest.yaml on the above manifest — but can instead just merge it to git and Argo will pull it in for us GitOps style.

Now that is end-to-end GitOps — and running your K8s platform the modern K8s way!

I have continued this series — getting into more detail about how to actually set up the git repo and Argo CD as a platform team for optimum multi-cluster GitOps in my next blog post — https://jason-umiker.medium.com/argo-cds-app-of-apps-an-efficient-and-easy-way-for-a-platform-team-to-manage-clusters-and-f23877834d89