kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6142) Connect worker configurations and connector configurations should accept environment variables
Date Mon, 30 Oct 2017 16:25:00 GMT
Randall Hauch created KAFKA-6142:
------------------------------------

             Summary: Connect worker configurations and connector configurations should accept
environment variables
                 Key: KAFKA-6142
                 URL: https://issues.apache.org/jira/browse/KAFKA-6142
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
    Affects Versions: 1.0.0
            Reporter: Randall Hauch


Currently, when a worker or connector configuration is parsed, the values are used as-is without
any kind of pre-processing before the value is used. It should be possible to define configuration
properties such that string literal values or default values can use *_configuration variables_*
that reference environment variables and/or system properties, and that these configuration
variables are resolved/replaced before the configuration value is used.

I propose doing this enhancement in Kafka client's {{ConfigDef}} by adding a {{ConfigDef.Transformer}}
interface:

{code:java}
    /**
     * Transform the configuration value.
     */
    public interface Transformer {
        /**
         * Transform the configuration value.
         * @param name The name of the configuration
         * @param value The value of the configuration
         * @return the preprocessed value
         * @throws ConfigException if the value is invalid.
         */
        Object apply(String name, Object value);
    }
{code}

and then allowing {{Transformer}} implementations to be passed to {{ConfigDef.define(...)}}
such all existing signatures are maintained for backward compatibility. By default, the definition
would use an identity transform that simply returns the value. The transformers would be called
in {{ConfigDef.parseValue(...)}} before the {{parseType(...)}} method is called, and would
also be called on the default value if one is provided.

Then, a {{ConfigDef.ReplaceSystemVariables}} implementation would be provided to look in {{String}}
values for zero or more variables defined with this EBNF grammar:

{noformat}
'$' '{' varName { ',' varName } [ ':' defaultValue] '}'
{noformat}

where:
* {{varName}} is the name of a Java system property or {{env.}} followed by the name of an
environment variable, and 
* {{defaultValue}} specifies the replacement value used when no environment variable or system
property is found, and defaults to an empty string. 

The value of the first system property or environment variable resolved is then used to replace
the variable expression. This implementation would have trace or debug level logging to describe
what it is doing.

For example, the variable {{$\{env.KAFKA_HOME\}}} would replace the variable expression with
the value of the {{KAFKA_HOME}} environment variable or an blank string if that variable doesn't
exist. 

Likewise, the variable {{$\{foo.prop1,foo.prop2,env.MY_ENV_VAR:value\}}} would be replaced
with the value of the {{foo.prop1}} system property if it exists, or with the value of the
{{foo.prop2}} system property if it exists, or with the value of the {{MY_ENV_VAR}} environment
variable if it exists, or {{value}} if none of the system properties exist.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message