ConfigSeeder OS Connector Documentation

1. Introduction

Almost every application needs configuration data to work as intended. This configuration data normally comes in form of property files, json- or yaml-formatted files, as environment variables set by a start-script or in some other form.

The OS Connector is a ConfigSeeder component that provides configuration data in form of files to an OS (Linux or Windows).

Two types of files can be provided to applications:

Files with static Content
Files with templating (e.g. content of file can differ depending on filter critera)

1.1. Use cases

Provide Configuration Data at installation time

During the install process, the OS Connector is executed and provides configuration files. These files are used by the installation process and copied to the right loation. At startup time, the OS Connector (and therefore the ConfigSeeder) isn’t required any more.

You can also execute the OS Connector with a tool like Ansible to setup a Linux VM (There will also be the possibility to retrieve configuration data directly with Ansible with an Ansible Plugin. Please contact us if you are interested in trying out our Ansible Integration).
Provide Configuration Data at startup time

Before an application is started, the OS Connector is executed and provides configuration data for the application.

The OS Connector is also available as docker container (configseeder/linux-connector:<version>) which can be used as init-container.

2. Quickstart

2.1. Introduction

2.1.1. Content of the quickstart manual

The OS Connector is able to provide configuration files to a linux or windows machine. This quickstart guide gives an introduction to the use of the OS Connector.

You will understand, how

static configuration files are defined in ConfigSeeder
configuration files with templating are defined in ConfigSeeder
the OS Connector provides configuration files to Linux or Windows

This quickstart guide also offers links to more detailed information.

Be advised that the OS Connector is a ConfigSeeder module under heavy development and currently available as a tech preview.. The structure of the Configuration Values / Assemblies used to configure files can be changed in the future.

2.1.2. Preconditions

ConfigSeeder up and running
Connectivity from the Linxu/Windows VM running the OS Connector to the ConfigSeeder
Licence which supports the use of OS Connector

2.2. Prepare configuration groups in ConfigSeeder

Create a Configuration Group os-connector-quickstart (the name of the Configuration Group can be chosen as desired)

When creating the Configuration Group, just enable the filter columns Environment and Context and leave the security options on their default values.
Create the two environments TEST and PROD (the name of the Environments can be chosen as desired)
Create two API Keys with permission to access the Configuration Group os-connector-quickstart One API Key should have permissions to access the Test-environment, the other one for Production.

Copy the generated API Key to a secure location. ConfigSeeder doesn’t store API Keys so you can’t retrieve it again.

2.3. Scenario 1: Static configuration

2.3.1. Prepare configuration values in ConfigSeeder

Open the Configuration Group os-connector-quickstart, switch to the Value Assemblies-Tab and create the following Assembly of Type File: static:

Remarks:
- In the example meta.filename is set to quickstart-static.txt. It can also be set to a absolute or relative path.
- With the value shown in the screenshot, the file will be created at the location where the OS Connector is executed.
- If you are trying out the OS Connector on Linux, pathes can be entered like <path>/<filename> where <path> can be absolute or relative
- If you are trying out the OS Connector on Windows, be sure to enter a valid windows path like c:\temp...

2.3.2. Setup OS Connector

Download und unpack the OS Connector to your system (see https://preview.configseeder.com/download/, depending on your os please use linux-connector or windows-connector)

Create the file connector.yaml with the follwing content:

connector:
  configSeeder:
    configurationGroupKeys: "os-connector-quickstart"
    tenantKey: "configseeder"                          (1)
    environmentKey: "TEST"                             (2)
    context: "example-static"                          (3)
    serverUrl: "http://url"                            (4)
    connectionTimeout: 2000
    readTimeout: 2000
    apiKey: "eyJraWQiO..."                             (5)

1	Tenant, in which the configuration is stored. The name of the default tenant is `default` (it is only required to specify the tenant if not the default tenant is used. If no tenant is specified `default` is used by convention)
2	Environment, for which the configuration should be read.
3	The context, for which the configuration should be retrieved. Is normally the name of the os (hostname) but can be any value or not used at all. If multiple os connectors are used, the combination of environment and context should be unique. In this quickstart guide, context is used to distinguish the different scenarios.
4	URL under which the ConfigSeeder is reachable.
5	Api Key with permissions to access the configured configuration groups and enviroment.

2.3.3. Execute os-connector

Start the OS Connector by executing ./linux-connector or executing windows-connector.exe

As a result, the OS Connector should log Created file /tmp/quickstart-static.txt for Application quickstart and the file /tmp/quckstart-static.txt should have been created with the configured content.

If the executable is run again, the OS Connector should log something like LastUpdate matches, no action required for file /tmp/quickstart-static.txt for Application quickstart and the file should be unchanged. If the content is changed and the command rerun, the OS Connector will change content of the file so it matches the configured content again.

2.4. Scenario 2: Configuration with templating

2.4.1. Prepare configuration values in ConfigSeeder

Open the Configuration Group os-connector-quickstart, switch to the Value Assemblies-Tab and create the following Assembly of Type File: template:

Remarks:
- In this example there is no environment set. The content of the file is controlled by the values which are used for the templating, see next step.
Open the Configuration Group os-connector-quickstart-values and create the following Configuration Values:

2.4.2. Setup OS Connector

Ensure that the OS Connector is available.

Change the connector.yaml to

connector:
  configSeeder:
    configurationGroupKeys: "os-connector-quickstart"
    tenantKey: "default"
    environmentKey: "TEST"
    context: "example-templating" (1)
    serverUrl: "http://url"
    connectionTimeout: 2000
    readTimeout: 2000
    apiKey: "eyJraWQiO..."

1	change context to `example-templating`

2.4.3. Execute OS Connector

Start the OS Connector by executing ./linux-connector or executing windows-connector.exe

As a result, the OS Connector should log Created file quickstart-templating.txt for Application quickstart and the file quckstart-template.txt should have been created with the content configured for test.

3. Configuration Options

3.1. Ways to configure the Connector

This section describes every available configuration option available for OS Connector and it’s modules.

There are three ways to configure the connector:

Configuration by file (lowest priority)

Default filename is connector.yaml, the connector looks for the file in the directoy it is started from. The filename can be overwritten, see Configuration Options.

Example:

connector:
  log:
    level: DEBUG
  file:
    directoryWhitelist:
    - /tmp
  configSeeder:
    tenantKey: "configseeder"
    environmentKey: "TEST"
    configurationGroupKeys: "os-connector-integrationtest"
    serverUrl: "https://staging-postgres-config-seeder.oneo.cloud"

Configuration by environment variables

Example:

export CONNECTOR_LOG_LEVEL=DEBUG
export CONNECTOR_CONFIGSEEDER_APIKEY=xxxx
...
linux-connector

Configuration by startup parameter (highest priority)

Example:

linux-connector -connector.log.level=DEBUG -connector.configseeder.apikey=xxx ...

It is possible to mix the different ways to configure the connector. You probably want to configure the connector by file and provide the API Key with an environment variable so it isn’t readable by anyone (You don’t want to provide the API Key as a startup parameter because the values of startup parameters can be retrieved by anyone using ps awx).

Priorities of ways to configure the connector

A parameter configured by file can be overwritten by environment variable or startup parameter
A parameter configured by environment variable can be overwritten by startup parameter

3.2. Configuration Options

The OS Connector can be configured by the following options:

All keys are prefixed with connector.. This prefix is neglected in the table to save space.

Key Mandatory Value

Key	Mandatory	Value
`configfile`	no	Location and name of the configfile containing the connector configuration. Defaults to `connector.yaml`.
`configseeder.apiKey`	yes	API Key used by Connector to access ConfigSeeder. The API Key must be of Type `OS connector`
`configseeder.configurationGroupKeys`	yes	Defines which configuration groups should be read by the Connector.
`configseeder.context`	no	Optional for filtering configuration values by `context`.
`configseeder.connectionTimeout`	no	Connection Timeout, defaults to `1000` (ms).
`configseeder.dateTime`	no	Optional for filtering configuration values by `validFrom` & `validTo`. Defaults to now.
`configseeder.environmentKey`	yes	Environment for which the Connector is running.
`configseeder.maxRetries`	no	Defines, how many times the connector should try to connect to ConfigSeeder Defaults to `3`.
`configseeder.readTimeout`	no	Read Timeout, defaults to `2000` (ms).
`configseeder.retryWaitTime`	no	Time in Milliseconds between failed tries to connect to ConfigSeeder. Defaults to `500` (ms).
`configseeder.serverUrl`	yes	URL under which ConfigSeeder can be reached.
`configseeder.tenantKey`	no	Defaults to Tenant `default`.
`configseeder.version`	no	Optional for filtering configuration values by `versionFrom` & `versionTo`
`file.allowExecutableMode`	no	Set to `true` if the OS Connector is allowed to create files which are executable (Only relevant under Linux) Defaults to `false`.
`file.directoryWhitelist`	no	If set, the OS Connector is only allowed to manage files in the specified directories.
`file.disabled`	no	Set to `true` to disable the file module.
`file.fileMode`	no	Default file mode, if no mode is specified. Defaults to `644`.
`file.pathPrefix`	no	Optional prefix for files with a relative path. If set, all relative files are prefixed with this value.
`file.replacingPostfix`	no	Postfix for keys which will be replaced by ConfigSeeder. Defaults to `}`.
`file.replacingPrefix`	no	Prefix for keys which will be replaced by ConfigSeeder. Defaults to `${`.
`log.filename`	no	Name of the Logfile if logging to a file is configured. Defaults to `os-connector-<YYYY-MM-DD>.log`.
`log.level`	no	ogLevel for the connector, valid levels are: trace, debug, info, warning, error, fatal, panic. Defaults to 'info'
`log.target`	no	Target for the connector logging, valid targets are CONSOLE, CONSOLE_AND_FILE and FILE. Defaults to 'CONSOLE'.
`startupMaxWaitTime`	no	Maximum duration in milliseconds which is waited if there is a problem connecting to the ConfigSeeder at startup time. Not setting `startupMaxWaitTime` means endless retries Defaults to endless retries (not default set).
`startupRetryWaitTime`	no	Duration in milliseconds which is waited between retries if there is a problem connecting to the ConfigSeeder at startup time. Defaults to `15000` (ms)
`useHostnameAsContext`	no	Defines if the hostname should be sent to ConfigSeeder as context when looking up configuration values. Defaults to `false`.

configfile

Location and name of the configfile containing the connector configuration.

Defaults to connector.yaml.

configseeder.apiKey

yes

API Key used by Connector to access ConfigSeeder.

The API Key must be of Type OS connector

configseeder.configurationGroupKeys

yes

Defines which configuration groups should be read by the Connector.

configseeder.context

Optional for filtering configuration values by context.

configseeder.connectionTimeout

Connection Timeout, defaults to 1000 (ms).

configseeder.dateTime

Optional for filtering configuration values by validFrom & validTo.

Defaults to now.

configseeder.environmentKey

yes

Environment for which the Connector is running.

configseeder.maxRetries

Defines, how many times the connector should try to connect to ConfigSeeder

Defaults to 3.

configseeder.readTimeout

Read Timeout, defaults to 2000 (ms).

configseeder.retryWaitTime

Time in Milliseconds between failed tries to connect to ConfigSeeder.

Defaults to 500 (ms).

configseeder.serverUrl

yes

URL under which ConfigSeeder can be reached.

configseeder.tenantKey

Defaults to Tenant default.

configseeder.version

Optional for filtering configuration values by versionFrom & versionTo

file.allowExecutableMode

Set to true if the OS Connector is allowed to create files which are executable (Only relevant under Linux)

Defaults to false.

file.directoryWhitelist

If set, the OS Connector is only allowed to manage files in the specified directories.

file.disabled

Set to true to disable the file module.

file.fileMode

Default file mode, if no mode is specified.

Defaults to 644.

file.pathPrefix

Optional prefix for files with a relative path. If set, all relative files are prefixed with this value.

file.replacingPostfix

Postfix for keys which will be replaced by ConfigSeeder.

Defaults to }.

file.replacingPrefix

Prefix for keys which will be replaced by ConfigSeeder.

Defaults to ${.

log.filename

Name of the Logfile if logging to a file is configured.

Defaults to os-connector-<YYYY-MM-DD>.log.

log.level

ogLevel for the connector, valid levels are: trace, debug, info, warning, error, fatal, panic.

Defaults to 'info'

log.target

Target for the connector logging, valid targets are CONSOLE, CONSOLE_AND_FILE and FILE.

Defaults to 'CONSOLE'.

startupMaxWaitTime

Maximum duration in milliseconds which is waited if there is a problem connecting to the ConfigSeeder at startup time. Not setting startupMaxWaitTime means endless retries

Defaults to endless retries (not default set).

startupRetryWaitTime

Duration in milliseconds which is waited between retries if there is a problem connecting to the ConfigSeeder at startup time.

Defaults to 15000 (ms)

useHostnameAsContext

Defines if the hostname should be sent to ConfigSeeder as context when looking up configuration values.

Defaults to false.

The specified keys can be easily translated to environment variable names:

Take the specified key
Replace . with _
Only use UPPERCASE Characters

connector.configseeder.apiKey becomes CONNECTOR_CONFIGSEEDER_APIKEY

4. Modules

4.1. File Module

4.1.1. Introduction

The File Module is responsible for creating different types of configuration files on a Linux or Windows VM.

The following Value Assemblies are used by the file module: * File: static: Manage files with static content * File: template Manage files which are created with a templating mechanism

4.1.2. Configuration Values within Value Assemblies

Key Mandatory Value

file

(yes)

Mandatory for Value Assemblies of type File: static

Content of the file.

If a different file content is required for different environments (or other filter criteria), there could be

a file with content Content for Test with environment TEST
a file with content Content for Production with environment PROD

meta.filename

yes

Name of the file

filename can have different kind of values:

filename.txt if only the filename is specified
./my/relative/path/filename.txt if a relative filename must be used
/absolute/path/filename.txt if the file should be managed with an absolut path

If relative or absolut pathes are configured, the correct path separators must be used:

Linux: …/…
Windows: c:\...

If a file should be provided to a Linux and a Windows VM, there could be the following configuration:

meta.filename = /tmp/filename with context Linux
meta.filename = c:\\temp\\filename.txt with context Windows

meta.mode

File mode which should be set:

Linux: All modes can be set. Owner should have at least read-access. Otherwise, managing the file isn’t possible
Windows: Only 0444 and 0666 are valid modes.

See Configuration Options for setting a default mode.

meta.user

Owner of the file, examples:

Linux: user
Windows: Setting the owner is not supported under Windows

Setting the user is only supported under linux.

When using owner, the OS Connector must be run as root

meta.group

Group of the file, examples:

Linux: group
Windows: Setting the group is not supported under Windows

Setting the group is only supported under linux.

When using group, the OS Connector must be run as root

template.file

(yes)

Mandatory for Value Assemblies of type File: template

Content/Template of the file. This content isn’t written directly to a VM, but first sent to the templating service of ConfigSeeder. Normally, templating is done with Prefix ${ and Postfix }. If you like to use an other Prefix and Postfix, please see Configuration Options.

template.configurationGroups

(yes)

Mandatory for Value Assemblies of type File: template

This value specifies which Configuration Groups are used for the templating. Defaults to the Configuration Group in which the Value Assembly is defined.

5. Status and Metadata

5.1. Metadata (Stored per File)

5.1.1. Introduction

The OS Connector stores Metadata for every managed file. Depending on the used Operating System, the Metadata is stored in a different way, see later sections.

The following Metadata is required by the OS Connector:

managedByConfigseeder

Boolean flag which defines if a file is managed by ConfigSeeder, meaning if ConfigSeeder OS Connector is allowed to modify or delete the given file. If a ConfigSeeder managed file shouldn’t be modified or deleted by the OS Connector, this flag can be deleted or set to false.

configseederModule

String value which defines which module of the OS Connector is responsible for managing the file.

contentHash Sha256 Hash of the content. Used by the OS Connector to verify if the content of a file has changed since the last run.

This attribute is set when a file is created and updated when the content of a file is updated by OS Connector.

lastUpdatedInMilliseconds

Timestamp which states at which time the file was last updated in ConfigSeeder. This Flag is used to determine if a File is still up to date or if it must be recreated. If the attribute isn’t available, the file will be recreated. This attribute is set or updated to lastUpdatedInMilliseconds of newest ConfigValue at creation and recreation time.

lastUpdatedInMilliseconds is set to the value of the attribute lastUpdateInMilliseconds of the newest ConfigValue which was used to create the file.

lastUpdatedInMilliseconds is not set to the time at which the file was modified on the os.

5.1.2. Linux

Under Linux, the metadata is stored in the extended file attributes, see https://en.wikipedia.org/wiki/Extended_file_attributes#Linux

Medatada Extended File Attribute

Medatada	Extended File Attribute
configseeder-Module	`user.configseeder-module`
contentHash	`user.configseeder-contentHash`
lastUpdatedInMilliseconds	`user.configseeder-lastUpdatedInMs`
managedBy-Configseeder	`user.configseeder-managed`

configseeder-Module

user.configseeder-module

contentHash

user.configseeder-contentHash

lastUpdatedInMilliseconds

user.configseeder-lastUpdatedInMs

managedBy-Configseeder

user.configseeder-managed

Manually Read or Modify Metadata

Reading Metadata:

$ getfattr -d /tmp/testfile.txt
getfattr: Removing leading '/' from absolute path names
# file: tmp/testfile.txt
user.configseeder-lastUpdatedInMs=0s4YOs3GcBAAA=
user.configseeder-managed="true"
user.configseeder-module="FILE"

Modifying Metadata:

Do not modify metadata unless you know what you are doing! There should be no reason to modify the metadata other than parhaps set user.configseeder-managed to false if a file should be removed from ConfigSeeder control

$ setfattr /tmp/testfile.txt -n user.configseeder-managed -v "false"
$ getfattr -d /tmp/testfile.txt
getfattr: Removing leading '/' from absolute path names
# file: tmp/testfile.txt
user.configseeder-contentHash="8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a"
user.configseeder-lastUpdatedInMs=0s4YOs3GcBAAA=
user.configseeder-managed="false"
user.configseeder-module="FILE"

5.1.3. Windows

Under Linux, the metadata is stored as json in an alternate data stream, see https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_(ADS)

Modifying and saving a file with an editor like Notepad++ or Ultraedit will remove the Alternate Data Stream and the Metadata.

Manually Read or Modify Metadata

Reading Metadata:

> more < testfile.txt:ConfigseederMetadata
{"ManagedByConfigseeder":true,"ConfigSeederModule":"FILE","LastUpdateInMilliseconds":1545587516835,"ContentHash":"8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a"}

Writing Metadata:

Do not modify metadata unless you know what you are doing! There should be no reason to modify the metadata other than perhaps set ManagedByConfigseeder to false if a file should be removed from ConfigSeeder control

Create a File metadata.txt with content

{"ManagedByConfigseeder":false,"ConfigSeederModule":"FILE","LastUpdateInMilliseconds":1545587516835,"ContentHash":"8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a"}

Update metadata

> type metadata.txt > testfile.txt:ConfigseederMetadata
> more < testfile.txt:ConfigseederMetadata
{"ManagedByConfigsee...