Kubernetes GitOps with Jsonnet, Tanka, and Argo CD

Why Jsonnet/Tanka?

Most k8s configuration I do at $JOB is done by pointing Argo CD to Helm charts and values files. So why not just continue this pattern for the kube-prometheus stack? There is a perfectly good community-maintained Helm chart for it!

Customizability

Using the kube-prometheus Jsonnet directly makes it easier to extend and configure. The kube-prometheus stack is so large and complex, that Helm’s approach of templating out values that chart consumers might want to change doesn’t work too well, since it is hard to account for all the different ways someone might want to tweak such a complex deployment. Jsonnet’s approach of being able to compose/overwrite any part of the configuration seems better suited in this case.

Additionally, when deploying with Jsonnet it is trivial to incorporate mixins into the deployment, which seem to be the most popular way for publishing custom Grafana dashboards and Prometheus rules/alerts.

Tanka

I’ve been keeping an eye on the Tanka project for some time now, specifically because it seems to solve one of my pain points when working with Helm charts, which is their rigid extensibility. If I want to change a resource in a Helm chart in a way that the chart creator did not forsee and pull out into a templated value, I have to fork the chart and make the change myself. Tanka allows making arbitrary Jsonnet changes on top of rendered Helm charts without having to fork them.

Configuring Tanka to render kube-prometheus Jsonnet into YAML

Install Tanka and Jsonnet Bundler

Initialize a new project:

mkdir kube-prometheus && cd kube-prometheus && tk init

Figure out which release of kube-prometheus you should use based on your k8s cluster version:

export KUBE_PROM_RELEASE='0.9'

Install it with Jsonnet Bundler

jb install "github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@release-$KUBE_PROM_RELEASE"

Copy that release’s example.jsonnet into your Tanka project:

wget -O environments/default/main.jsonnet "https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/release-$KUBE_PROM_RELEASE/example.jsonnet"

Ask Tanka to render the YAML

tk show environments/default

You should see a bunch of k8s resource YAMLs that make up the kube-prometheus stack. Tanka can be configured to connect to a cluster and deploy the rendered resources, but we want to do that part with Argo CD so we can get the nice UI and GitOps workflow.

Configuring Tanka as an Argo CD CMP

We can take advantage of Argo CD’s CMP(config management plugin) pattern to run the official Tanka image as a sidecar in order to render the kube-prometheus YAML.

First, create a ConfigManagementPlugin manifest to configure the plugin, and put it in a ConfigMap so we can mount it to our sidecar container.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-tanka-cmp
  namespace: argocd
  annotations:
    app.kubernetes.io/part-of: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: tanka
    spec:
      version: v1.0
      init:
        command: [jb, install]
      generate:
        command: [sh, -c]
        args: ["/usr/local/bin/tk show --dangerous-allow-redirect $TANKA_PATH"]
      discover:
        fileName: "./jsonnetfile.json"
      allowConcurrency: true
      lockRepo: false

Note that in the discover section, I configured this plugin to be used for any projects with a jsonnetfile.json in the root.

Next, patch the argocd-repo-server deployment and add the following container & volume definitions:

containers:
- name: tanka-cmp
  command: [/var/run/argocd/argocd-cmp-server]
  image: grafana/tanka:latest
  securityContext:
    runAsNonRoot: true
    runAsUser: 999
  volumeMounts:
  - mountPath: /var/run/argocd
    name: var-files
  - mountPath: /home/argocd/cmp-server/plugins
    name: plugins
  - mountPath: /home/argocd/cmp-server/config/plugin.yaml
    subPath: plugin.yaml
    name: tanka-cmp
  - mountPath: /tmp
    name: tmp
volumes:
- name: tanka-cmp
  configMap:
    name: argocd-tanka-cmp

Finally, we are able to create an Argo CD Application that uses the plugin:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus
  namespace: argocd
spec:
  destination:
    namespace: monitoring
    name: [your cluster name]
  project: [your Argo CD project]
  source:
    repoURL: [your project repo URL]
    path: [path to kube-prometheus project]
    targetRevision: HEAD
    plugin:
      env:
      - name: TANKA_PATH
        value: environments/default
  syncPolicy:
    syncOptions:
    - CreateNamespace=true

That’s it! We are all set to manage our kube-prometheus stack with Jsonnet, Tanka, and Argo CD!

Versions

Below is a list of the versions of stuff I used when getting the above to work:

Name	Version
Kubernetes	1.21
Argo CD	2.3.0
Tanka	0.20.0
kube-prometheus	release-0.9