1. Introduction

The ConfigSeeder® Admission Controller is a Mutating Kubernetes Admission Controller that can automatically inject one of the ConfigSeeder Connectors in a kubernetes object at creation time.

Manually configuring these connectors can be laborious and in some cases nearly impossible (for example if you use Helm Charts that are provided by a 3rd party).

The ConfigSeeder Admission Controller simplifies injecting connectors. This documentation describes:

  • How to set up the ConfigSeeder Admission Controller

  • How to let the ConfigSeeder Admission Controller inject Connectors in various Kubernetes objects

Be aware that the ConfigSeeder® Admission Controller is currently in the status of a technical preview and should not be used in production.

1.1. Supported Kubernetes objects

The ConfigSeeder® Admission Controller can inject the connectors in different Kubernetes objects:

  • CronJob (Creation of the CronJob, Job, or Pod)

  • Deployment (Creation of the Deployment or Pod)

  • Job (Creation of the Job or Pod)

  • Pod (Creation of the Pod, regardless of the owner ressource)

  • StatefulSet (Creation of the StatefulSet or Pod)

1.2. Preconditions

The following preconditions must be met so that the admission controller can be used:

  • Basic ConfigSeeder® installation must be up and running (ConfigSeeder Management)

  • The ConfigSeeder extension Kubernetes Connector must be licensed

  • Permission to deploy a mutating webhook configuration (See setting up the Mutation Webhook Configuration)

2. Admission Controller Configuration

2.1. Configuration of the injected connectors

2.1.1. Concept

2.1.2. Global / Admission Controller wide options

The ConfigSeeder Admission Controller can be configured by the following options:

All keys are prefixed with ADMISSIONCONTROLLER_. This prefix is neglected in the table to save space.
Key Mandatory Value

CONFIGSEEDER_CONNECTIONTIMEOUT

yes

Read timeout for accessing ConfigSeeder, used by the injected containers.

Defaults to 10000 (ms)

CONFIGSEEDER_READTIMEOUT

yes

Connection timeout for accessing ConfigSeeder, used by the injected containers..

Defaults to 10000 (ms)

CONFIGSEEDER_URL

yes

URL of the ConfigSeeder, used by the injected containers.

CONFIGSEEDER_TENANTKEY

no

Key of the ConfigSeeder tenant to access, used by the injected containers.

Defaults to `default'.

IMAGE_KUBERNETESCONNECTOR

yes

Image of the Kubernetes Connector that should be injected into the pods.

IMAGE_OSCONNECTOR

yes

Image of the OS Connector that should be injected into the pods.

LOG_FILENAME

no

Name of the logfile, the controller should write, used by the injected containers. Only relevant if togeter with log target file and console_and_file.

Defaults to admission-controller-yyyy-MM-dd.log.

LOG_JSON

no

Flag to define if the log output should be formatted as json, used by the injected containers.

Defaults to true

LOG_LEVEL

no

Log Level used by the controller, used by the injected containers.

Defaults to INFO

LOG_TARGET

no

Log Level used by the controller, used by the injected containers.

  • console: Only log to console

  • file: Only log to a file

  • console_and_file: Logs to the console and a file

Defaults to console

2.1.3. Options configurable per pod/deployment

2.1.3.1. Labels
Key Mandatory Value

CONFIGSEEDER_CONNECTIONTIMEOUT

yes

Read timeout for accessing ConfigSeeder, used by the injected containers.
2.1.3.2. Annotations

All keys are prefixed with admissioncontroller.configseeder.com/. This prefix is neglected in the table to save space.
Key Mandatory Connector Value

injectApiKeySecret

no

All

Name of the Secret holding the API Key required for accessing ConfigSeeder.

Defaults to

  • kubernetes-connector-apikey when injecting a kubernetes-connector

  • os-connector-apikey when injecting an os-connector

injectConfigMapCertificates

no

All

Name of the ConfigMap holding the certificates for verifying the TLS connection to ConfigSeeder.

Only required if

  • The connection to ConfigSeeder is secured with TLS

  • Self signed (CA) certificates are used

No default, if the annotation is not present, no additional certificates will be injected

injectConfigMap

no

All

Name of the ConfigMap holding the configuration for the injected connector.

No default. As an alternative, the configuration group(s), environment and other configuration can be provided with dedicated annotations.

injectConfigurationGroupKeys

no

All

Key(s) of the configuration group that should be read by the injected connector.

This annotations should only be used if no admissioncontroller.configseeder.com/injectConfigMap is specified.

injectEnvironmentKey

no

All

Key of the environment that should be read by the injected connector.

This annotations should only be used if no admissioncontroller.configseeder.com/injectConfigMap is specified.

injectLogLevel

no

All

Log level that should be used by the injected connector.

This annotations should only be used if no admissioncontroller.configseeder.com/injectConfigMap is specified.

injectStatemanagerName

no

Kubernetes

Name of the configmap in which the state of the kubernetes connector will be stored.

Defaults to kubernetes-connector-state

injectStatemanagerNamespace

no

Kubernetes

Namespace which the ConfigMap holding the state of the kubernetes connector will be stored.

Defaults to the current namespace

injectPathPrefix

no

OS

Location, where the OS Connector should inject data to. A volume of type emptydir is created together with volumeMounts to the specified location for all containers listed in the pod.

Defaults to /injected-config

3. Installation

3.1. Overview

The installation of the ConfigSeeder® Admission Controller consists of two parts:

  1. Deploy the Admission Controller

  2. Setup the Mutating Webhook Configuration (tell Kubernetes to use the Admission Controller)

3.2. Deploy the ConfigSeeder® Admission Controller

The easiest way to deploy the Admission Controller is to use the provided Helm Charts:

  1. Get the Helm Charts from https://configseeder.com/download/

  2. Provide a admission-controller-values.yaml with the following content:

    (Only required if the defaults should be overwritten)

    admissionController:
  configseeder:
    url: "https://configseeder-management.cs-demo-management-h2.svc:8443"
    tenantKey: "default"

  # images and it's subproperties define which image & version of the connectors should be injected
  images:
    kubernetesConnector: "configseeder/kubernetes-connector/kubernetes-connector:1.3.4-alpine"
    osConnector: "configseeder/os-connector/linux-connector:1.1.4-alpine"

  3. Create a rsa key and server certificate for the admission controller

    • Add the following SAN:

      • admission-controller.configseeder-admission-controller.svc.cluster.local

      • admission-controller.configseeder-admission-controller.svc

    • Store the key and certificate in a Secret of type TLS and name tls-secret in the namespace configseeder-admission-controller

      kubectl -n configseeder-admission-controller create secret tls tls-secret --cert="admission-controller-crt.pem" --key="admission-controller-key.pem" --dry-run=client -o yaml | kubectl -n configseeder-admission-controller apply -f -

  4. Install the Chart using the following command:

    helm -n "configseeder-admission-controller" upgrade -i "admission-controller" "configseeder-admission-controller-version.tar.gz" --values-admission-controller.yaml

3.3. Setting up the Mutation Webhook Configuration

The following snippet shows the mutating webhook configuration for the ConfigSeeder® Admission Controller:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: configseeder-admission-controller
webhooks:
- name: "admission-controller.configseeder.com"
  objectSelector:
    matchExpressions:
    - key: admissioncontroller.configseeder.com/injectConnector
      operator: In
      values: [ "kubernetes-connector", "os-connector" ]
  rules:
  - apiGroups: [ "apps" ]
    apiVersions: [ "v1" ]
    operations: [ "CREATE" ]
    resources: [ "deployments", "statefulsets" ]
    scope: Namespaced
  - apiGroups: [ "batch" ]
    apiVersions: [ "v1", "v1beta1" ]
    operations: [ "CREATE" ]
    resources: [ "cronjobs", "jobs" ]
    scope: Namespaced
  - apiGroups: [ "" ]
    apiVersions: [ "v1" ]
    operations: [ "CREATE" ]
    resources: [ "pods" ]
    scope: "Namespaced"
  clientConfig:
    caBundle: REPLACE_WITH_BASE64_ENCODED_CERTIFICATE
    service:
      namespace: configseeder-admission-controller
      name: admission-controller
      path: /mutate
  admissionReviewVersions: [ "v1", "v1beta1" ]
  sideEffects: None
  timeoutSeconds: 1

Setup Instructions:

  1. Get the certificate used to sign the admission controller certificate

  2. Encode the certificate using base64 (cat rootca.crt | base64 -w0)

  3. Replace REPLACE_WITH_BASE64_ENCODED_CERTIFICATE with the encoded certificate

  4. Deploy the mutating webhook configuration (kubectl apply -f configseeder-admission-controller.yaml)

4. Examples for injecting the connectors

4.1. Generic examples

4.1.1. Preconditions

In the examples below, the following assumptions apply:

  • The Connectors will

    • be configured by the ConfigMap connector

    • use the certificates stored in the ConfigMap connector-certificates for verifying the TLS connection to the ConfigSeeder

    • use the api key stored in the ConfigMap connector-apikey to access ConfigSeeder

  • A connector of type os-connector (linux connector) will be injected (it works the same for the kubernetes connector)

4.1.2. Inject when creating a Pod

The Admission Controller can inject a connector whenever a new pod is created (and regardless of the owner of the pod).

The following example shows how to configure a pod so that a linux connector is injected as init-container:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  annotations:
    admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
    admissioncontroller.configseeder.com/injectConfigMap: connector
    admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
    admissioncontroller.configseeder.com/injectPathPrefix: /data
  labels:
    admissioncontroller.configseeder.com/injectConnector: os-connector
spec:
  containers:
    - name: alpine
      image: alpine
      command: ["/bin/sh", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]

The Admission Controller will be activated because of the label admissioncontroller.configseeder.com/injectConnector and inject the os-connector. The resulting pod will look something like this:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
    admissioncontroller.configseeder.com/injectConfigMap: connector
    admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
    admissioncontroller.configseeder.com/injectPathPrefix: /data
  labels:
    admissioncontroller.configseeder.com/injectConnector: os-connector
    ...
  name: my-pod
  ...
spec:
  containers:
  - name: alpine
    image: alpine
    command: ["/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
    - mountPath: /data
      name: injected-config
      readOnly: true
    ...
  initContainers:
  - env:
    - name: CONNECTOR_CONFIGSEEDER_APIKEY
      valueFrom:
        secretKeyRef:
          key: apiKey
          name: connector-apikey-test
          optional: false
    - name: CONNECTOR_FILEPROPERTIES_PATHPREFIX
      value: /data
    - name: CONNECTOR_CONFIGSEEDER_SERVERURL
      value: https://configseeder-management.cs-demo-management-h2.svc:8443
    - name: CONNECTOR_CONFIGSEEDER_TENANTKEY
      value: default
    - name: CONNECTOR_CONFIGSEEDER_CONNECTIONTIMEOUT
      value: "10000"
    - name: CONNECTOR_CONFIGSEEDER_READTIMEOUT
      value: "10000"
    - name: CONNECTOR_CONFIGSEEDER_CERTIFICATEFILENAME
      value: /certificates.pem
    envFrom:
    - configMapRef:
        name: connector-test
        optional: false
    image: registry.gitlab.com/configseeder/os-connector/linux-connector:staging-alpine
    imagePullPolicy: Always
    name: os-connector
    resources:
      limits:
        cpu: 500m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 10Mi
    volumeMounts:
    - mountPath: /data
      name: injected-config
    - mountPath: /certificates.pem
      name: connector-certificate-volume
      readOnly: true
      subPath: certificates.pem
    ...
  volumes:
  - configMap:
      defaultMode: 420
      name: connector-certificates-test
      optional: false
    name: connector-certificate-volume
  - emptyDir: {}
    name: injected-config
  ...

4.1.3. Injecting connectors to Deployments, Jobs, and StatefulSets

The Admission Controller can inject a connector whenever a new Deployment, Job or StatefulSet is created or whenever a pod is created because of one of the mentioned kubernetes objects.

4.1.3.1. Inject connectors at kubernetes object creation time
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  annotations:
    admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
    admissioncontroller.configseeder.com/injectConfigMap: connector
    admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
    admissioncontroller.configseeder.com/injectPathPrefix: /data
  labels:
    admissioncontroller.configseeder.com/injectConnector: os-connector
    app.kubernetes.io/name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-deployment
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-deployment
    spec:
      restartPolicy: Always
      containers:
        - name: alpine
          image: alpine
          imagePullPolicy: Always
          command: ["/bin/sh", "-c", "--" ]
          args: [ "while true; do sleep 30; done;" ]
4.1.3.2. Inject connectors at Pod creation time
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app.kubernetes.io/name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-deployment
  template:
    metadata:
      annotations:
        admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
        admissioncontroller.configseeder.com/injectConfigMap: connector
        admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
        admissioncontroller.configseeder.com/injectPathPrefix: /data
      labels:
        admissioncontroller.configseeder.com/injectConnector: os-connector
    spec:
      restartPolicy: Always
      containers:
        - name: alpine
          image: alpine
          imagePullPolicy: Always
          command: ["/bin/sh", "-c", "--" ]
          args: [ "while true; do sleep 30; done;" ]

4.1.4. Inject when creating a CronJob

The Admission Controller can inject a connector whenever a new CronJob is created or whenever a pod is created because of the CronJob.

4.1.4.1. Inject connectors at CronJob creation time
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-cronjob
  annotations:
    admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
    admissioncontroller.configseeder.com/injectConfigMap: connector
    admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
    admissioncontroller.configseeder.com/injectPathPrefix: /data
  labels:
    admissioncontroller.configseeder.com/injectConnector: os-connector
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    metadata:
      labels:
        # without this label, the `admissioncontroller.configseeder.com/injectConnector` label is set. Kubernetes bug?
        just: a-label
    spec:
      backoffLimit: 0
      ttlSecondsAfterFinished: 604800
      template:
        metadata:
        spec:
          # Don't restart if an error occures
          restartPolicy: Never
          containers:
            - name: alpine
              image: alpine
              imagePullPolicy: Always
4.1.4.2. Inject connectors at Pod creation time
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-cronjob
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    metadata:
      labels:
        # without this label, the `admissioncontroller.configseeder.com/injectConnector` label is set. Kubernetes bug?
        just: a-label
    spec:
      metadata:
        annotations:
          admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
          admissioncontroller.configseeder.com/injectConfigMap: connector
          admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
          admissioncontroller.configseeder.com/injectPathPrefix: /data
        labels:
          admissioncontroller.configseeder.com/injectConnector: os-connector
      backoffLimit: 0
      ttlSecondsAfterFinished: 604800
      template:
        metadata:
        spec:
          # Don't restart if an error occures
          restartPolicy: Never
          containers:
            - name: alpine
              image: alpine
              imagePullPolicy: Always

4.2. Real world examples

4.2.1. Inject configuration data in an existing application

4.2.1.1. Situation

  • Configuration Data should be provided to an application

  • The application is installed via a Helm Chart

  • The Helm Chart shouldn’t be modified or can’t be modified

  • There is a ConfigSeeder® Setup with ConfigSeeder Management and the Admission Controller up and running

4.2.1.2. Solution

Depending on the exact requirements (and how the Helm Chart of the application is built), there are different solutions:

  1. Provide Configuration Files via OS Connector by setting the Admission Controller label and annotations to the Chart

  2. Provide ConfigMaps & Secrets via Kubernetes Connector running as a Helm pre-* Hook Job

Solution Description

1

Provide Configuration Files via OS Connector by setting the Admission Controller label and annotations to the Chart

Preconditions:

  • The applications Helm Chart must allow setting custom labels and annotations

  • The application must be configurable via files

  • The application must use one of the supported kubernetes objects (Deployment, StatefulSet, …​)

Solution description:

  • Add custom labels and annotations to the value.yaml and install the application

  • The admission controller will inject the connector before kubernetes applies it

The custom labels and annotations will look something like this:

# customAnnotations allows to set custom annotations (key value pairs: - key: value)
customAnnotations:
  admissioncontroller.configseeder.com/injectConnector: os-connector
# customLabels allows to set custom labels (key value pairs: - key: value)
customLabels:
  # admissioncontroller.configseeder.com/injectApiKeySecret points to the secret that contains the api key required for accessing ConfigSeeder. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
  # admissioncontroller.configseeder.com/injectConfigMap points to the configmap that holds the configuration of the connector. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectConfigMap: connector
  # admissioncontroller.configseeder.com/injectConfigMap is optional and points to the configmap that holds the certificates used for verifying the tls connection to ConfigSeeder. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates
  # admissioncontroller.configseeder.com/injectPathPrefix defines the directory to which the files retrieved from ConfigSeeder will be stored.
  admissioncontroller.configseeder.com/injectPathPrefix: /data

2

Provide ConfigMaps & Secrets via Kubernetes Connector running as a Helm pre-* Hook Job

Preconditions:

  • The application must be configurable via ConfigMaps and Secrets

Solution description:

  • Create the Helm Chart configuration-prehook-job

    • There should only be a Job contained in the Chart

      (and if required security elements like a ServiceAccount, Role Binding for a Pod Security Policy, …​)

    • Set the required annotations to mark the job as a helm hook

  • Make it possible to add custom labels and annotations

  • Add the Helm Chart configuration-prehook-job to your applications /charts directory

The Job could look like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Values.naming.job }}
  annotations:
    {{- include "custom.annotations" $ | indent 4 }}
  labels:
    {{- include "custom.labels" $ | indent 4 }}
  spec:
  # Don't do any retries
  backoffLimit: 0
  # Automatically clean up after 7 days
  ttlSecondsAfterFinished: 604800
  template:
    spec:
      # Don't restart if an error occures
      restartPolicy: Never
      containers:
        - name: alpine
          image: alpine
          imagePullPolicy: Always

The custom labels and annotations will look something like this:

customAnnotations:
  # Set the annotations required by helm to recognize the job as a hook (see https://helm.sh/docs/topics/charts_hooks/)
  "helm.sh/hook": pre-install,pre-upgrade,pre-rollback
  "helm.sh/hook-weight": "1"
  "helm.sh/hook-delete-policy": before-hook-creation
  # enable injecting the kubernetes connector
  admissioncontroller.configseeder.com/injectConnector: kubernetes-connector

customLabels:
  # admissioncontroller.configseeder.com/injectApiKeySecret points to the secret that contains the api key required for accessing ConfigSeeder. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectApiKeySecret: connector-apikey
  # admissioncontroller.configseeder.com/injectConfigMap points to the configmap that holds the configuration of the connector. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectConfigMap: connector
  # admissioncontroller.configseeder.com/injectConfigMap is optional and points to the configmap that holds the certificates used for verifying the tls connection to ConfigSeeder. Can be created by a kubernetes connector.
  admissioncontroller.configseeder.com/injectConfigMapCertificates: connector-certificates