flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <decho...@gmail.com>
Subject Re: Short guide to custom configuration provider?
Date Tue, 02 Apr 2013 19:23:30 GMT
I did a short proof of concept for the YAML part here
https://github.com/BertrandDechoux/flume-ng-yaml

Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look
at what was done in that issue and see what can be done for a clean
selection of a custom configuration provider.

Bertrand


On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

>  I actually do like the idea of pluggable config provider. Like you
> mentioned, it should be possible to hack the Application class, and use a
> CLI arg to select the config provider. If no CLI arg is provided, just fall
> back to the current mechanism. If you submit a patch, I'd be glad to review
> it :-)
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
>
> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of the Application class, that's something which
> is already mostly done by
> https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
> from Zookeeper watcher
>
> It might be a good idea to split this improvement into :
> 1) Selection of an alternative configuration provider
> 2) A zookeeper based configuration provider
>
> If point 1 is done, anyone could use her/his custom configuration provider
> by dropping the implementation as a third party jar and then selecting it
> with the help of the command line.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <dechouxb@gmail.com>wrote:
>
> Thanks for the hints!
>
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
>
> Configuration Provider
>
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>
>
> I will look at the classes you pointed for my own education and I will see
> what I can do.
>
> Thanks a lot.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <brock@cloudera.com> wrote:
>
> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>
> Brock
>
>
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <dechouxb@gmail.com>wrote:
>
> Hi,
>
> Flume NG has a pluggable configuration system called the configuration
> provider. I cloned the source repository and I indeed found it but I am a
> bit lost. Is there some kind of short guide explaining how to write a
> custom configuration provider?
>
> I would like to start with something simple : using a YAML configuration
> file instead of a properties file.
> It should be almost trivial to change the following configuration
>
> *agent1.channels.ch1.type = memory
>
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
>
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = log-sink1*
>
> into
>
> agent1:
>
>     channels:
>         ch1:
>             type: memory
>
>     sources:
>         avroSource1:
>             channels: ch1
>             type: avro
>             bind: 0.0.0.0
>             port: 41414
>
>     sinks:
>         logSink1:
>             channel: ch1
>             type: logger
>
>
> At least, I know how to convert the latter into the former but I am not
> sure how this transformation can be included into flume-ng using a
> configuration provider.
> I have not been able to find much information about the customisation of
> the configuration provider. Did I miss any documentation?
>
>
> Bertrand
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
>
>
>
>
> --
> Bertrand Dechoux
>
>
>


-- 
Bertrand Dechoux

Mime
View raw message