1. Introduction

Nowadays many microservice applications are deployed in the form of containers in a Kubernetes Cluster. There are multiple ways to configure these containers. The configuration can be …

  1. shipped within the image itself

  2. passed on a mounted volume

  3. passed as environment variables

    1. sourced from a configmap

    2. sourced from a secret

  4. retrieved from an external service

If you are using ConfigSeeder to manage your configuration data, you can either fetch the required configuration data with an init-container (Scenario 2, OS Connector or SimpleConfigDownloader) or retrieve it directly with a ConfigSeeder Client (Scenario 4). In either case, an API Key is required to retrieve configuration data from ConfigSeeder.

These API Keys can be managed (created and replaced before expiration) manually or automatically by Kubernetes Connector. The Kubernetes Connector reads Assemblies of type Secret: API Key from ConfigSeeder and manages API Keys within Kubernetes Secrets.

At the moment, the Kubernetes Connector is limited to manage API Keys. Soon (planned for the end of September) the Kubernetes Connector will have the ability to provision configuration data stored in ConfigMaps and Secrets (Scenario 3a/b).

1.1. Use cases

The kubernetes-connector can be used for the following scenarios:

  • Provide API Keys with access to configured Configuration Groups within Secrets.

Not yet implemented:

  • Provide configuration data stored in ConfigMaps and Secrets.

2. Quickstart

2.1. Introduction

2.1.1. Content of the quickstart manual

The kubernetes-connector can to provide API Keys for accessing configuration data to a kubernetes cluster. This quickstart guide gives an introduction to the use of the kubernetes-connector.

You will understand, how

  • Secrets containing an API Key are defined in ConfigSeeder

  • multiple environments/namespaces are supported

  • labels and annotations can be set

This quickstart guide also offers links to more detailed information.

Please be advised that the kubernetes-connector is under heavy development and currently available as a tech preview. The structure of the configValues used to configure kubernetes objects can be changed in the future. Expect that other things like the described behavior will also be changed.

2.1.2. Preconditions

  • ConfigSeeder up and running

  • Licence which supports the use of the kubernetes-connector

  • Connectivity from the machine or cluster running kubernetes-connector to the ConfigSeeder

  • Access to a kubernetes cluster and permission to manage kubernetes objects like namespaces, service accounts, secrets and rbac roles & bindings.

2.2. Prepare ConfigSeeder Management

  1. Create the Environments TEST and PROD

  2. Create the Configuration Group kubernetes-connector-job-demo

  3. Create the Configuration Group demo-application

2.3. Setup Kubernetes Connector

The following actions are required to set up Kubernetes Connector:

  1. Create Namespace in which the Connector should be running

    ns.yaml
    apiVersion: v1
kind: Namespace
metadata:
  name: cs-demo-k8s-connector

    kubectl apply -f ns.yaml

  2. Create Serviceaccount with which the Connector will connect to the Kubernetes Master

    sa.yaml
    apiVersion: v1
kind: ServiceAccount
metadata:
  name: cs-demo-k8s-connector
  namespace: cs-demo-k8s-connector

    kubectl apply -f sa.yaml

  3. Give the ServiceAccount permission to store state-information (in this quickstart guide it is expected, that the state will be stored in the same Namespace in which the Kubernetes Connector(s) are running). Also, prepare a ClusterRole with access to Kubernetes Object which will be managed by Kubernetes Connector

    rbac.yaml
    kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector-state
  namespace: cs-demo-k8s-connector
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create", "get", "update", "delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector-state
  namespace: cs-demo-k8s-connector
subjects:
  - kind: ServiceAccount
    name: cs-demo-k8s-connector
    namespace: cs-demo-k8s-connector
roleRef:
  kind: Role
  name: cs-demo-k8s-connector-state
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "update", "delete"]

    kubectl apply -f rbac.yaml

  4. Create Serviceaccount with which the Connector will connect to the Kubernetes Master

    sa.yaml
    apiVersion: v1
kind: ServiceAccount
metadata:
  name: cs-demo-k8s-connector
  namespace: cs-demo-k8s-connector

    kubectl apply -f sa.yaml

  5. Give the ServiceAccount permission to store state-information (in this quickstart guide it is expected, that the state will be stored in the same Namespace in which the Kubernetes Connector(s) are running). Also, prepare a ClusterRole with access to Kubernetes object which will be managed by Kubernetes Connector

    rbac.yaml
    kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector-state
  namespace: cs-demo-k8s-connector
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create", "get", "update", "delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector-state
  namespace: cs-demo-k8s-connector
subjects:
  - kind: ServiceAccount
    name: cs-demo-k8s-connector
    namespace: cs-demo-k8s-connector
roleRef:
  kind: Role
  name: cs-demo-k8s-connector-state
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "update", "delete"]

    kubectl apply -f rbac.yaml

  6. Create the Namespaces in which the Kubernetes Connector shall manage Kubernetes objects and give the ServiceAccount permission to manage the objects. In this quickstart guide, the namespaces cs-demo-application-test and cs-demo-application-prod are used.

    ns-example.yaml
    apiVersion: v1
kind: Namespace
metadata:
  name: cs-demo-application-test
---
apiVersion: v1
kind: Namespace
metadata:
  name: cs-demo-application-prod
---
apiVersion: v1
kind: Namespace
metadata:
  name: cs-demo-ansible-prod
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector
  namespace: cs-demo-application-test
subjects:
  - kind: ServiceAccount
    name: cs-demo-k8s-connector
    namespace: cs-demo-k8s-connector
roleRef:
  kind: ClusterRole
  name: cs-demo-k8s-connector
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector
  namespace: cs-demo-application-prod
subjects:
  - kind: ServiceAccount
    name: cs-demo-k8s-connector
    namespace: cs-demo-k8s-connector
roleRef:
  kind: ClusterRole
  name: cs-demo-k8s-connector
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cs-demo-k8s-connector
  namespace: cs-demo-ansible-prod
subjects:
  - kind: ServiceAccount
    name: cs-demo-k8s-connector
    namespace: cs-demo-k8s-connector
roleRef:
  kind: ClusterRole
  name: cs-demo-k8s-connector
  apiGroup: rbac.authorization.k8s.io

    kubectl apply -f ns-example.yaml

  7. Create Image Pull Secrets in all Namespaces to access Docker Images from Docker Hub (https://cloud.docker.com/u/configseeder/):

    kubectl create secret docker-registry -n cs-demo-k8s-connector image-pull-secret \
        --docker-username=<username> \
        --docker-password=<password>
    kubectl create secret docker-registry -n cs-demo-application-test image-pull-secret \
        --docker-username=<username> \
        --docker-password=<password>
    kubectl create secret docker-registry -n cs-demo-application-prod image-pull-secret \
        --docker-username=<username> \
        --docker-password=<password>

  8. Create API Key used by the Kubernetes Connector to access ConfigSeeder:

    1. Create API Key of Type Kubernetes Connector with access to the relevant Configuration Groups and Environments:

      Create API Key in ConfigSeeder

    2. Store generated API Key in file apikey.txt

    3. Create Secret:

      kubectl create secret generic -n cs-demo-k8s-connector cs-demo-k8s-connector-apikey \
        --from-file=apikey=apikey.txt

  9. Store configuration used by Kubernetes Connector in a ConfigMap:

    cs-demo-k8s-connector-config.yaml
    apiVersion: v1
kind: ConfigMap
metadata:
  name: cs-demo-k8s-connector-config
  namespace: cs-demo-k8s-connector
data:
  CONNECTOR_APIKEY_LIFETIMEINDAYS: "2"
  CONNECTOR_APIKEY_RECREATEDAYSBEFOREEXPIRATION: "1"
  CONNECTOR_CONFIGSEEDER_SERVERURL: https://staging-postgres-config-seeder.oneo.cloud
  CONNECTOR_CONFIGSEEDER_TENANTKEY: configseeder
  CONNECTOR_CONFIGSEEDER_CONFIGURATIONGROUPKEYS: kubernetes-connector-job-demo
  CONNECTOR_CONFIGSEEDER_CONNECTIONTIMEOUT: "5000"
  CONNECTOR_CONFIGSEEDER_READTIMEOUT: "5000"
  CONNECTOR_LOG_LEVEL: "INFO"
  CONNECTOR_STARTUPRETRYWAITTIME: "10000"
  CONNECTOR_STATEMANAGER_NAMESPACE: "cs-demo-k8s-connector"

    Be sure to set the follwing options to the correct values:

    CONFIGSEEDER_SERVERURL
CONFIGSEEDER_TENANTKEY
CONFIGSEEDER_CONFIGURATIONGROUPKEYS

    kubectl apply -f cs-demo-k8s-connector-config.yaml

2.4. Execute Kubernetes Connector

If the Kubernetes Connector is started as a Job or CronJob inside Kubernetes, it is essential that the attributes controlling the retry behaviour are set to the correct values. Otherwise, a lot of jobs will be started. Please find more information here:

2.4.1. Periodically execute Kubernetes Connector with a CronJob

Starting Kubernetes Connector with a CronJob is usually done if exact timing of a configuration Change is not really relevant and configuration changes should be applied to kubernetes as soon as possible (e.g. for dev or test environments).

  1. Create CronJob Template

    cs-demo-k8s-connector-cronjob-test.yaml
    apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cs-demo-k8s-connector-cronjob-test
  namespace: cs-demo-k8s-connector
spec:
  schedule: "*/5 * * * *"
  # Don't execute Job more than once at the same time
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      # Don't do any retries
      backoffLimit: 0
      template:
        spec:
          # Don't restart if an error occures
          restartPolicy: Never
          containers:
            - name: kubernetes-connector
              image: configseeder/kubernetes-connector:0.9.0
              #image: registry.gitlab.com/configseeder/kubernetes-connector/kubernetes-connector:staging-alpine
              imagePullPolicy: Always
              envFrom:
                - configMapRef:
                    name: cs-demo-k8s-connector-config
              env:
                - name: CONNECTOR_CONFIGSEEDER_ENVIRONMENTKEY
                  value: "TEST"
                - name: CONNECTOR_STATEMANAGER_NAME
                  value: "cs-demo-k8s-connector-state-test"
                - name: CONNECTOR_CONFIGSEEDER_APIKEY
                  valueFrom:
                    secretKeyRef:
                      name: cs-demo-k8s-connector-apikey
                      key: apikey
          #imagePullSecrets:
          #  - name: image-pull-secret
          serviceAccountName: cs-demo-k8s-connector

  2. Start CronJob

    startCronJobTest.sh
    #!/bin/bash

kubectl apply -f cs-demo-k8s-connector-cronjob-test.yaml

    ./startCronJobTest.sh

  3. Stop / Delete CronJob

    stopCronJobTest.sh
    #!/bin/bash

kubectl delete -f cs-demo-k8s-connector-cronjob-test.yaml

    ./startCronJobTest.sh

2.4.1.1. Results

As long as no Value Assembly of Type Secret: API Key is created in Configuration Group kubernetes-connector-job-demo the Kubernetes Connector won’t do anything.

  1. No secret should have been created:

    kubectl get secret -n cs-demo-application-test

  2. No error should be logged:

    kubectl logs -n cs-demo-k8s-connector cs-demo-k8s-connector-cronjob-test-<…​>

  3. No Object should be managed by Kubernetes Connector:

    kubectl get cm -n cs-demo-k8s-connector cs-demo-k8s-connector-state-test -o yaml

If a Value Assembly of Type Secret: API Key has been created or changed, re-executing the shell script is not required. The CronJob will re-execute Kubernetes Connector after the defined time and the changes will be applied automatically.

2.4.2. Start Kubernetes Connector as a Job

Starting Kubernetes Connector with a CronJob is usually done if exact timing of a configuration Change is relevant (e.g. for production level or similar environments).

  1. Create Job Template

    cs-demo-k8s-connector-job-prod.yaml
    apiVersion: batch/v1
kind: Job
metadata:
  name: cs-demo-k8s-connector-job
  namespace: cs-demo-k8s-connector
spec:
  # Don't do any retries
  backoffLimit: 0
  # Automatically clean up after 7 days
  ttlSecondsAfterFinished: 604800
  template:
    spec:
      # Don't restart if an error occures
      restartPolicy: Never
      containers:
        - name: kubernetes-connector
          image: configseeder/kubernetes-connector:0.9.0
          #image: registry.gitlab.com/configseeder/kubernetes-connector/kubernetes-connector:staging-alpine
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: cs-demo-k8s-connector-config
          env:
            - name: CONNECTOR_CONFIGSEEDER_ENVIRONMENTKEY
              value: "PROD"
            - name: CONNECTOR_STATEMANAGER_NAME
              value: "k8s-connector-state-prod"
            - name: CONNECTOR_CONFIGSEEDER_APIKEY
              valueFrom:
                secretKeyRef:
                  name: cs-demo-k8s-connector-apikey
                  key: apikey
      #imagePullSecrets:
      #  - name: image-pull-secret
      serviceAccountName: cs-demo-k8s-connector

  2. Execute Job

    startJobProd.sh
    #!/bin/bash

timestamp=`date +%F-%H.%M.%S`
jobYaml=`sed -e "s/k8s-connector-job/k8s-connector-job-${timestamp}/g" cs-demo-k8s-connector-job-prod.yaml`

echo "$jobYaml" | kubectl apply -f -

    ./startJobProd.sh

2.4.2.1. Results

As long as no Value Assembly of Type Secret: API Key is created in Configuration Group kubernetes-connector-job-demo the Kubernetes Connector won’t do anything.

  1. No secret should have been created:

    kubectl get secret -n cs-demo-application-prod

  2. No error should be logged:

    kubectl logs -n cs-demo-k8s-connector cs-demo-k8s-connector-job-<…​>

  3. No Object should be managed by Kubernetes Connector:

    kubectl get cm -n cs-demo-k8s-connector cs-demo-k8s-connector-state-prod -o yaml

If startJobProd.sh is (re)executed after a Value Assembly of Type Secret: API Key has been created, the given commands should show details of the created secret.

2.5. Manage a Secret containing an API Key

Create a ValueAssembly of Type Secret: API Key in Configuration Group Kubernetes Job Demo:

Create API Key in ConfigSeeder

Reexecute Job or wait until CronJob has run again, check results.

2.6. Set a label or annotation

Modify previously created ValueAssembly:

Secret: API Key with label and annotation

Reexecute Job or wait until CronJob has run again, check results.

3. Kubernetes Connector Configuration

3.1. Ways to configure the Connector

This section describes every available configuration option available for Kubernetes Connector and it’s modules.

There are three ways to configure the connector:

  1. Configuration by file (lowest priority)

    Default filename is connector.yaml, the connector looks for the file in the directoy it is started from. The filename can be overwritten, see Configuration Options.

    Example:

    connector:
  log:
    level: DEBUG
  apiKey:
    lifetimeInDays: 2
    recreateDaysBeforeExpiration: 1
  configSeeder:
    tenantKey: "configseeder"
    environmentKey: "TEST"
    configurationGroupKeys: "kubernetes-connector-integrationtest"
    serverUrl: "https://staging-postgres-config-seeder.oneo.cloud"

  2. Configuration by environment variables

    Example:

    export CONNECTOR_LOG_LEVEL=DEBUG
export CONNECTOR_CONFIGSEEDER_APIKEY=xxxx
...
kubernetes-connector

  3. Configuration by startup parameter (highest priority)

    Example:

    kubernetes-connector -connector.log.level=DEBUG -connector.configseeder.apikey=xxx ...

It is possible to mix the different ways to configure the connector. You probably want to configure the connector by file and provide the API Key with an environment variable so it isn’t readable by anyone (You don’t want to provide the API Key as a startup parameter because the values of startup parameters can be retrieved by anyone using ps awx).

Priorities of ways to configure the connector

  • A parameter configured by file can be overwritten by environment variable or startup parameter

  • A parameter configured by environment variable can be overwritten by startup parameter

3.2. Configuration Options

The Kubernetes Connector can be configured by the following options:

All keys are prefixed with connector.. This prefix is neglected in the table to save space.
Key Mandatory Value

configfile

no

Location and name of the configfile containing the connector configuration.

Defaults to connector.yaml.

apiKey.disabled

no

Can be set to true if the Api Key Management - Module should be disabled

apiKey.lifetimeInDays

yes

Defines how many days an API Key created by Kubernetes Connector will be valid.

apiKey.recreateDaysBeforeExpiration

yes

Defines how many days before reaching the expiration date an API Key managed by Kubernetes Connector should be replaced. The replaced API Key won’t be invalidated.

This configuration option must be smaller than lifetimeInDays by at least 1

configfile

no

Name of the file configuring Kubernetes Connector (if configured by file).

Defaults to connector.yaml.

configseeder.apiKey

yes

API Key used by Connector to access ConfigSeeder.

The API Key must be of Type Kubernetes connector

configseeder.configurationGroupKeys

yes

Defines which configuration groups should be read by the Connector.

configseeder.context

no

Optional for filtering configuration values by context.

configseeder.connectionTimeout

no

Connection Timeout, defaults to 1000 (ms).

configseeder.dateTime

no

Optional for filtering configuration values by validFrom & validTo.

Defaults to now.

configseeder.environmentKey

yes

Environment for which the Connector is running.

configseeder.maxRetries

no

Defines, how many times the connector should try to connect to ConfigSeeder

Defaults to 3.

configseeder.readTimeout

no

Read Timeout, defaults to 2000 (ms).

configseeder.retryWaitTime

no

Time in Milliseconds between failed tries to connect to ConfigSeeder.

Defaults to 500 (ms).

configseeder.serverUrl

yes

URL under which ConfigSeeder can be reached.

configseeder.tenantKey

no

Defaults to Tenant default.

configseeder.version

no

Optional for filtering configuration values by versionFrom & versionTo

kubeconfig

no

Name (and path) of the kubernetes config file used by Kubernetes Connector to connecto to the cluster. Only used if the Kubernetes Connector runs outside of a Kubernetes Cluster.

log.filename

no

Name of the Logfile if logging to a file is configured.

Defaults to kubernetes-connector-<YYYY-MM-DD>.log.

log.level

no

ogLevel for the connector, valid levels are: trace, debug, info, warning, error, fatal, panic.

Defaults to 'info'

log.target

no

Target for the connector logging, valid targets are CONSOLE, CONSOLE_AND_FILE and FILE.

Defaults to 'CONSOLE'.

startupMaxWaitTime

no

Maximum duration in milliseconds which is waited if there is a problem connecting to the ConfigSeeder at startup time. Not setting startupMaxWaitTime means endless retries

Defaults to endless retries (not default set).

startupRetryWaitTime

no

Duration in milliseconds which is waited between retries if there is a problem connecting to the ConfigSeeder at startup time.

Defaults to 15000 (ms)

stateManager.name

no

Name of the ConfigMap containing the state

Defaults to kubernetes-connector-state

If more than one Kubernetes Connector is used, it must be ensured that every Connector has its own combination of namespace and name.

stateManager.namespace

no

Namespace in which the ConfigMap containing the state is created.

Defaults to configseeder.

If more than one Kubernetes Connector is used, it must be ensured that every Connector has its own combination of namespace and name.

The specified keys can be easily translated to environment variable names:

  • Take the specified key

  • Replace . with _

  • Only use UPPERCASE Characters

connector.configseeder.apiKey becomes CONNECTOR_CONFIGSEEDER_APIKEY

4. Modules

4.1. AKM (Api Key Management)

4.1.1. Introduction

The AKM Module is responsible for creating API Keys that are stored in Kubernetes Secrets.

The required API Keys/secrets are configured with Value Assemblies of type Secret: API Key within ConfigSeeder:

Secret: API Key in ConfigSeeder

The example above would create:

  • A secret named my-application-api-key in the namespace my-application-test containing an API Key which grants access on environment TEST and configuration group Release 2.7

  • A secret named my-application-api-key in the namespace my-application-prod containing an API Key which grants access on environment PROD and configuration group Release 2.7

4.1.2. Usage of Configuration Values within Value Assembly

Key Mandatory Value

apiKey.configurationGroups

yes

This value specifies to which configurationGroups the created API Key will have access.

meta.namespace

yes

Namespace in which the Secret will be created.

If more than one Environment is used, there could be

  • a namespace <application>-test with environment TEST and

  • a namespace <application>-prod with environment PROD

meta.secretname

yes

Name of the Secret which will be created because of this Value Assembly.

If more than one Environment is used, there could be

  • a secretname '<application-apikey>-test' with environment TEST and

  • a secretname '<application-apikey>-prod' with environment PROD

However, we recommend having a dedicated namespace for each application and environment.

meta.annotatation.xxx

no

Used to set one or more annotations (replace xxx with name of annotation)

meta.label.xxx

no

Used to set one or more label (replace xxx with name of label)

5. Status and Metadata

5.1. Status (stored per Kubernetes Connector)

The Kubernetes Connector stores its state in a ConfigMap, normally the name is configseeder/kubernetes-connector-state.

The state contains an overview of all objects managed by the connector, the last action and information if the last action was executed successfully or details to the failure if the action has failed.

Do not modify state-information unless you know what you are doing! There should be no reason to modify the state-information and there is a big chance of destroying the state if tempered with it.

If the Kubernetes Connector is unable to read the state or is otherwise unable to interpret it, the state can be deleted. This leads to a new beginning / recreating of configuration data in the Kubernetes Cluster.

5.2. Metadata (stored per managed object)

The Kubernetes Connector stores Metadata in every managed object. The following annotations are used:

configseeder.com/managed

Boolean flag which defines if an object is managed by Kubernetes Connector, meaning if Kubernetes Connector is allowed to modify or delete the given object. If a ConfigSeeder managed object shouldn’t be modified or deleted by the Kubernetes Connector, this flag can be deleted or set to false.

configseeder.com/module

String value which defines which module of the Kubernetes-Connector is responsible for managing the file.

Manually Read or Modify Metadata

Reading Metadata:

kubectl get secrets -n <namespace> <secret> -o yaml

kind: Secret
metadata:
  annotations:
    configseeder.com/managed: "true"
    configseeder.com/module: AKM
...

Writing Metadata:

Do not modify metadata unless you know what you are doing! There should be no reason to modify the metadata other than perhaps set annotation configseeder.com/managed to false if a file should be removed from ConfigSeeder control 
kubectl edit secrets -n <namespace> <secret>