1. Introduction
Almost every application needs configuration data to work as intended. This configuration data normally comes in form of property files, json- or yaml-formatted files, as environment variables set by a start-script or in some other form.
The OS Connector is a ConfigSeeder component that provides configuration data in form of files to an OS (Linux or Windows).
The following types of files can be provided to applications:
-
Files with static content
-
Files with templating (e.g. content of file can differ depending on filter critera)
-
Files with binary content
1.1. Use cases
-
Provide Configuration Data at installation time
During the install process, the OS Connector is executed and provides configuration files. These files are used by the installation process and copied to the right loation. At startup time, the OS Connector (and therefore the ConfigSeeder) isn’t required any more.
You can also execute the OS Connector with a tool like Ansible to setup a Linux VM (There will also be the possibility to retrieve configuration data directly with Ansible with an Ansible Plugin. Please contact us if you are interested in trying out our Ansible Integration).
-
Provide Configuration Data at startup time
Before an application is started, the OS Connector is executed and provides configuration data for the application.
The OS Connector is also available as docker container (
configseeder/linux-connector:<version>
) which can be used as init-container.
2. Quickstart
2.1. Introduction
2.1.1. Content of the quickstart manual
The OS Connector is able to provide configuration files to a linux or windows machine. This quickstart guide gives an introduction to the use of the OS Connector.
You will understand, how
-
static configuration files are defined in ConfigSeeder
-
configuration files with templating are defined in ConfigSeeder
-
the OS Connector provides configuration files to Linux or Windows
This quickstart guide also offers links to more detailed information.
2.1.2. Preconditions
-
ConfigSeeder up and running
-
Connectivity from the Linxu/Windows VM running the OS Connector to the ConfigSeeder
-
Licence which supports the use of OS Connector
2.2. Prepare configuration groups in ConfigSeeder
-
Create a Configuration Group
os-connector-quickstart
(the name of the Configuration Group can be chosen as desired)When creating the Configuration Group, just enable the filter columns Environment and Context and leave the security options on their default values.
-
Create the two environments
TEST
andPROD
(the name of the Environments can be chosen as desired) -
Create two API Keys with permission to access the Configuration Group
os-connector-quickstart
One API Key should have permissions to access the Test-environment, the other one for Production.Copy the generated API Key to a secure location. ConfigSeeder doesn’t store API Keys so you can’t retrieve it again.
2.3. Scenario 1: Static configuration
2.3.1. Prepare configuration values in ConfigSeeder
-
Open the Configuration Group
os-connector-quickstart
, switch to the Value Assemblies-Tab and create the following Assembly of TypeFile: static
:Remarks:
-
In the example
file.name
is set toquickstart-static.txt
. It can also be set to a absolute or relative path. -
With the value shown in the screenshot, the file will be created at the location where the OS Connector is executed.
-
If you are trying out the OS Connector on Linux, paths can be entered like
<path>/<filename>
where<path>
can be absolute or relative -
If you are trying out the OS Connector on Windows, be sure to enter a valid windows path like
c:\temp...
-
2.3.2. Setup OS Connector
-
Download und unpack the OS Connector to your system (see https://preview.configseeder.com/download/, depending on your os please use
linux-connector
orwindows-connector
) -
Create the file
connector.yaml
with the follwing content:connector: configSeeder: configurationGroupKeys: "os-connector-quickstart" tenantKey: "userdocumentation" (1) environmentKey: "TEST" (2) context: "example-static" (3) serverUrl: "https://staging-postgresql-config-seeder.oneo.cloud" (4) connectionTimeout: 2000 readTimeout: 2000 apiKey: "eyJraWQiO..." (5)
1 Tenant, in which the configuration is stored. The name of the default tenant is default
(it is only required to specify the tenant if not the default tenant is used. If no tenant is specifieddefault
is used by convention)2 Environment, for which the configuration should be read. 3 The context, for which the configuration should be retrieved. Is normally the name of the os (hostname) but can be any value or not used at all. If multiple os connectors are used, the combination of environment and context should be unique. In this quickstart guide, context is used to distinguish the different scenarios. 4 URL under which the ConfigSeeder is reachable. 5 Api Key with permissions to access the configured configuration groups and enviroment.
2.3.3. Execute os-connector
Start the OS Connector by executing ./linux-connector
or executing windows-connector.exe
As a result, the OS Connector should log Created file /tmp/quickstart-static.txt for Application quickstart
and the
file /tmp/quckstart-static.txt
should have been created with the configured content.
If the executable is run again, the OS Connector should log something like LastUpdate matches, no action required for
file /tmp/quickstart-static.txt for Application quickstart
and the file should be unchanged. If the content is
changed and the command rerun, the OS Connector will change content of the file so it matches the configured content
again.
2.4. Scenario 2: Configuration with templating
2.4.1. Prepare configuration values in ConfigSeeder
-
Open the Configuration Group
os-connector-quickstart-values
and create the following Configuration Values: -
Open the Configuration Group
os-connector-quickstart
, switch to the Value Assemblies-Tab and create the following Assembly of TypeFile: template
: -
Click on the Value of
file.template
and enter the template:
Remarks:
-
In this example there is no environment set. The content of the file is controlled by the values which are used for the templating, see next step.
2.4.2. Setup OS Connector
-
Ensure that the OS Connector is available.
-
Change the
connector.yaml
toconnector: configSeeder: configurationGroupKeys: "os-connector-quickstart" tenantKey: "userdocumentation" environmentKey: "TEST" context: "example-templating" (1) serverUrl: "https://staging-postgresql-config-seeder.oneo.cloud" connectionTimeout: 2000 readTimeout: 2000 apiKey: "eyJraWQiO..."
1 change context to example-templating
2.4.3. Execute OS Connector
Start the OS Connector by executing ./linux-connector
or executing windows-connector.exe
As a result, the OS Connector should log Created file quickstart-templating.txt for Application quickstart
and
the file quckstart-template.txt
should have been created with the content configured for test.
3. Configuration Options
3.1. Ways to configure the Connector
This section describes every available configuration option available for OS Connector and it’s modules.
There are three ways to configure the connector:
-
Configuration by file (lowest priority)
Default filename is
connector.yaml
, the connector looks for the file in the directoy it is started from. The filename can be overwritten, see Configuration Options.Example:
connector: log: level: DEBUG fileProperties: directoryWhitelist: - /tmp configSeeder: tenantKey: "configseeder" environmentKey: "TEST" configurationGroupKeys: "os-connector-integrationtest" serverUrl: "https://staging-postgres-config-seeder.oneo.cloud"
-
Configuration by environment variables
Example:
export CONNECTOR_LOG_LEVEL=DEBUG export CONNECTOR_CONFIGSEEDER_APIKEY=xxxx ... linux-connector
-
Configuration by startup parameter (highest priority)
Example:
linux-connector -connector.log.level=DEBUG -connector.configseeder.apikey=xxx ...
It is possible to mix the different ways to configure the connector. You probably want to configure the connector by
file and provide the API Key with an environment variable so it isn’t readable by anyone (You don’t want to provide the
API Key as a startup parameter because the values of startup parameters can be retrieved by anyone using ps awx
).
Priorities of ways to configure the connector
|
3.2. Configuration Options
The OS Connector can be configured by the following options:
All keys are prefixed with |
Key | Mandatory | Value |
---|---|---|
|
no |
Location and name of the configfile containing the connector configuration. Defaults to |
|
no |
Set to |
|
yes |
API Key used by Connector to access ConfigSeeder. The API Key must be of Type |
|
no |
Allows to specify a file containing additional (PEM-formatted) certificates that are used for tls verification when connecting to configseeder. If this configuration option is not set, tls verification is done with the system certificate pool. |
|
yes |
Defines which configuration groups should be read by the Connector. |
|
no |
Optional for filtering configuration values by |
|
no |
Connection Timeout, defaults to |
|
no |
Optional for filtering configuration values by |
|
no |
Optional for filtering configuration values by Defaults to now. |
|
yes |
Environment for which the Connector is running. |
|
no |
Defines, how many times the connector should try to connect to ConfigSeeder Defaults to |
|
no |
Read Timeout, defaults to |
|
no |
Time in Milliseconds between failed tries to connect to ConfigSeeder. Defaults to |
|
yes |
URL under which ConfigSeeder can be reached. |
|
no |
Defaults to Tenant |
|
no |
Optional for filtering configuration values by |
|
no |
Set to Defaults to |
|
no |
If set, the OS Connector is only allowed to manage files in the specified directories. |
|
no |
Default file mode, if no mode is specified. Defaults to |
|
no |
Optional prefix for files with a relative path. If set, all relative files are prefixed with this value. |
|
no |
Set to |
|
no |
Name of the Logfile if logging to a file is configured. Defaults to |
|
no |
ogLevel for the connector, valid levels are: trace, debug, info, warning, error, fatal, panic. Defaults to 'info' |
|
no |
Target for the connector logging, valid targets are CONSOLE, CONSOLE_AND_FILE and FILE. Defaults to 'CONSOLE'. |
|
no |
Maximum duration in milliseconds which is waited if there is a problem
connecting to the ConfigSeeder at startup time. Not setting Defaults to endless retries (not default set). |
|
no |
Duration in milliseconds which is waited between retries if there is a problem connecting to the ConfigSeeder at startup time. Defaults to |
|
no |
Set to |
|
no |
Set to |
|
no |
Postfix for keys which will be replaced by ConfigSeeder. Defaults to |
|
no |
Prefix for keys which will be replaced by ConfigSeeder. Defaults to |
|
no |
Defines if the hostname should be sent to ConfigSeeder as context when looking up configuration values. Defaults to |
The specified keys can be easily translated to environment variable names:
|
4. Modules
4.1. Introduction
The different modules are responsible for managing different types of files. The following table gives an overview of the available modules.
Module | Assembly | Description |
---|---|---|
Binary File |
File: Binary |
Create and manage files which contain binary content |
Generated File |
File: Generated |
Create and manage files created out of existing configuration data |
Keystore File |
File: Keystore |
Create and manage Key-/Truststores based on PEM Certificates and Private Keys |
Static File |
File: Static |
Create and manage files which contain static content |
Template File |
File: Template |
Create and manage files which are created using a templating mechanism |
The modules can by configured with different options. The following table shows the configuration options which are valid for all the modules:
Key | Mandatory | Value | ||||
---|---|---|---|---|---|---|
|
yes |
With this flag, an assembly can be activated or deactivated for different scenarios. To enable an assembly for just environment
To disable an assembly for label
|
||||
|
yes |
Name of the file
If relative or absolut paths are configured, the correct path separators must be used:
If a file should be provided to a Linux and a Windows VM, there could be the following configuration:
|
||||
|
no |
File mode which should be set:
See Configuration Options for setting a default mode. |
||||
|
no |
Owner of the file, examples:
|
||||
|
no |
Group of the file, examples:
|
4.2. Module Binary File
4.2.1. Introduction
This module is responsible for creating configuration files with binary content on a Linux or Windows VM.
Binary Files are configured with Value Assemblies of type File: Binary
4.2.2. Configuration Values within Value Assemblies
Key | Mandatory | Value |
---|---|---|
|
yes |
Binary content of the file. If a different file content is required for different environments (or other filter criteria), there could be
|
4.3. Module Generated File
4.3.1. Introduction
This module is responsible for creating configuration files out of the configuration data stored in ConfigSeeder. The configuration values from one or more configuration groups are converted into a file of one of several predefined formats.
Generated Files are configured with Value Assemblies of type File: Generated
4.3.2. Configuration Values within Value Assemblies
Key | Mandatory | Value |
---|---|---|
|
yes |
Format in which the file should be generated. Can be one of:
|
|
yes |
This value specifies which Configuration Groups are used for the generation of the file. Defaults to the Configuration Group in which the Value Assembly is defined. |
4.3.3. Examples
-
Define the following values
-
Create the following assembly
-
Run the OS Connector with the different labels or just open the preview
4.3.3.1. INI
4.3.3.2. JSON
4.3.3.3. Properties
4.3.3.4. Yaml
4.4. Module Keystore File
4.4.1. Introduction
This module is responsible for creating PKCS12 files containing only Certificates (Truststore) or also Private Keys (Keystore).
Keystores are configured with Value Assemblies of type File: Keystore
4.4.2. Configuration Values within Value Assemblies
Key | Mandatory | Value |
---|---|---|
|
no |
Reference to one or multiple configuration groups containing
certificates. All Configuration Values with the type
|
|
no |
Without this attribute, all Certificates contained in the configuration
groups referenced by This attribute enables to filter these certificates. Examples:
|
|
yes |
Password of the Key-/Truststore Requirements:
|
|
no |
Type of the Keystore. The following types are supported:
|
|
The following configuration values have an |
|
|
no |
Password if the private key is stored encrypted in ConfigSeederĀ®. It the private key is not stored encrypted, this password will be ignored. |
|
no |
PEM encoded Private Key
|
|
no |
PEM encoded Certificate
|
|
no |
Password with which the private key is protected when stored in the key store. This password can only be set for Keystores with type |
4.4.3. Examples
-
Add a certificate (optional, keystores can be created without additional certificates)
-
Create the following assembly (adding private keys is optional, of course, and not required if a truststore should be created)
By hitting the preview Button, the keystore defined the assembly is created on the fly:
4.5. Module Static File
4.5.1. Introduction
This module is responsible for creating configuration files with static content on a Linux or Windows VM.
Static Files are configured with Value Assemblies of type File: Static
4.5.2. Configuration Values within Value Assemblies
Key | Mandatory | Value |
---|---|---|
|
yes |
Static content of the file. If a different file content is required for different environments (or other filter criteria), there could be
|
4.6. Module Template File
4.6.1. Introduction
This module is responsible for creating configuration files with a templating mechanism on a Linux or Windows VM.
Template Files are configured with Value Assemblies of type File: Template
4.6.2. Configuration Values within Value Assemblies
Key | Mandatory | Value |
---|---|---|
|
yes |
Content/Template of the file. This content isn’t written directly to a VM, but first sent to the templating service of ConfigSeeder. See Supported templating engines. |
|
yes |
This value specifies which Configuration Groups are used for the templating. Defaults to the Configuration Group in which the Value Assembly is defined. |
4.6.3. Supported templating engines
In the current version, the os connector supports the following template engines:
Template Engine | Description |
---|---|
HANDLEBARS |
Templating based on the handlebars templating engine, see https://handlebarsjs.com/ Supports simple variable replacement as well as more complex constructs like |
SIMPLE |
Simple template engine implementation that just replaces variables pre-/postfixed by |
5. Status and Metadata
5.1. Metadata (Stored per File)
5.1.1. Introduction
The OS Connector stores Metadata for every managed file. Depending on the used Operating System, the Metadata is stored in a different way, see later sections.
The following Metadata is required by the OS Connector:
managedByConfigseeder
Boolean flag which defines if a file is managed by ConfigSeeder, meaning if ConfigSeeder OS Connector is allowed to modify or delete the given file. If a ConfigSeeder managed file shouldn’t be modified or deleted by the OS Connector, this flag can be deleted or set to false.
configseederModule
String value which defines which module of the OS Connector is responsible for managing the file.
contentHash Sha256 Hash of the content. Used by the OS Connector to verify if the content of a file has changed since the last run.
This attribute is set when a file is created and updated when the content of a file is updated by OS Connector.
lastUpdatedInMilliseconds
Timestamp which states at which time the file was last updated in ConfigSeeder. This Flag is used to determine if a File is still up to date or if it must be recreated. If the attribute isn’t available, the file will be recreated. This attribute is set or updated to lastUpdatedInMilliseconds of newest ConfigValue at creation and recreation time.
lastUpdatedInMilliseconds is set to the value of the attribute lastUpdateInMilliseconds of the newest ConfigValue which was used to create the file. lastUpdatedInMilliseconds is not set to the time at which the file was modified on the os. |
5.1.2. Linux
Under Linux, the metadata is stored in the extended file attributes, see https://en.wikipedia.org/wiki/Extended_file_attributes#Linux
Medatada | Extended File Attribute |
---|---|
configseeder-Module |
|
contentHash |
|
lastUpdatedInMilliseconds |
|
managedBy-Configseeder |
|
Manually Read or Modify Metadata
Reading Metadata:
$ getfattr -d /tmp/testfile.txt getfattr: Removing leading '/' from absolute path names # file: tmp/testfile.txt user.configseeder-lastUpdatedInMs=0s4YOs3GcBAAA= user.configseeder-managed="true" user.configseeder-module="FILE"
Modifying Metadata:
Do not modify metadata unless you know what you are doing! There should be no reason to modify the metadata
other than parhaps set user.configseeder-managed to false if a file should be removed from ConfigSeeder control
|
$ setfattr /tmp/testfile.txt -n user.configseeder-managed -v "false" $ getfattr -d /tmp/testfile.txt getfattr: Removing leading '/' from absolute path names # file: tmp/testfile.txt user.configseeder-contentHash="8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a" user.configseeder-lastUpdatedInMs=0s4YOs3GcBAAA= user.configseeder-managed="false" user.configseeder-module="FILE"
5.1.3. Windows
Under Linux, the metadata is stored as json in an alternate data stream, see https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_(ADS)
Modifying and saving a file with an editor like Notepad++ or Ultraedit will remove the Alternate Data Stream and the Metadata. |
Manually Read or Modify Metadata
Reading Metadata:
> more < testfile.txt:ConfigseederMetadata {"ManagedByConfigseeder":true,"ConfigSeederModule":"FILE","LastUpdateInMilliseconds":1545587516835,"ContentHash":"8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a"}
Writing Metadata:
Do not modify metadata unless you know what you are doing! There should be no reason to modify the metadata
other than perhaps set ManagedByConfigseeder to false if a file should be removed from ConfigSeeder control
|
- Create a File
metadata.txt
with content
{"ManagedByConfigseeder":false,"ConfigSeederModule":"FILE","LastUpdateInMilliseconds":1545587516835,"ContentHash":"8a8f264c0f554ad07c2a6248990ebbaa166b3b00404e8a6dd297ca8715a7859a"}
- Update metadata
> type metadata.txt > testfile.txt:ConfigseederMetadata > more < testfile.txt:ConfigseederMetadata {"ManagedByConfigsee...
Appendix A: Running the OS Connector as an Init Container
A.1. Introduction
In some cases, an containerized application requires configuration data which can’t be provided by the Kubernetes Connector.
Reasons for this can be:
-
One of the (configuration) files required by the application is a binary file (license files, …)
-
The files are too large to be stored by the Kubernetes Connector in a ConfigMap
-
The files are to complex to be created by the Kubernetes Connector
-
The configuration must be reloaded whenever the application / pod restarts
Preconditions:
-
The application will be deployed to a Kubernetes or Openshift Cluster
-
The deployment descriptor of the application must be modifiable
A.2. Setup
A.2.1. Provide the API Key
First of all, a Kubernetes Secret containing the API Key (of type OS_Connector) must be available (in the same Namespace as the OS Connector will run).
This secret can be managed automatically by the Kubernetes Connector or it can be created manually. Either way, the Secret should look like this:
apiVersion: v1
kind: Secret
type: Opaqu
metadata:
name: os-connector-apikey
data:
apiKey: ...
If the application also accesses the ConfigSeeder (and therefore also requires an API Key) it is not necessary to have two separate API Keys. One API Key of the type OS_Connector is sufficient. The OS Connector and the application can access the ConfigSeeder with this key. Of course, the key must have permissions to access all relevant Configuration Groups.
A.2.2. Provide OS Connector Configuration
In this example, the OS Connector will be configured with a ConfigMap. This ConfigMap can either be provided automatically with the Kubernetes Connector or you can create it manually. The following key/values should be configured:
apiVersion: v1
kind: ConfigMap
metadata:
name: os-connector-configuration
data:
CONNECTOR_CONFIGSEEDER_CONFIGURATIONGROUPKEYS: my-configuration-group
CONNECTOR_CONFIGSEEDER_CONNECTIONTIMEOUT: "5000"
CONNECTOR_CONFIGSEEDER_READTIMEOUT: "5000"
CONNECTOR_CONFIGSEEDER_SERVERURL: https://demo.configseeder.com/
CONNECTOR_CONFIGSEEDER_TENANTKEY: default
CONNECTOR_CONFIGSEEDER_ENVIRONMENTKEY: TEST
CONNECTOR_FILE_PATHPREFIX: /config
The CONNECTOR_FILE_PATHPREFIX
ensures that all files with a relative path are written to the /config
-Folder.
Of course you don’t need a ConfigMap, it is also possible to manually define the environment variables directly in the deployment.
A.2.3. Add the OS Connector as Init Container
The OS Connector can be embedded in the following way in the deployment descriptor:
apiVersion: apps/v1
kind: Deployment
...
initContainers:
- name: os-connector
image: configseeder/linux-connector:<version>
imagePullPolicy: Always
envFrom:
- configMapRef:
name: os-connector-configuration
env:
- name: CONNECTOR_CONFIGSEEDER_APIKEY
valueFrom:
secretKeyRef:
name: os-connector-apikey
key: apiKey
volumeMounts:
- name: config-volume
mountPath: /config
...
volumes:
- name: config-volume
emptyDir: {}
As you can see, a volume of type emptyDir
is mounted to /config
. Together with the configuration option
CONNECTOR_FILE_PATHPREFIX
(see ConfigMap) this ensures, that the application will have a volume where all
configuration files will be available.