flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Short guide to custom configuration provider?
Date Tue, 02 Apr 2013 20:34:55 GMT
Hi Bertrand, 

https://issues.apache.org/jira/browse/FLUME-1630 changed a lot of the config system, and made
it much easier to write a config provider. Once you write a provider you can update the application
class to use reflection to instantiate the provider. I dont remember how the patch on FLUME-1491
looks like, but if it is possible to integrate it into this, it might make sense.


Thanks
Hari

-- 
Hari Shreedharan


On Tuesday, April 2, 2013 at 12:23 PM, Bertrand Dechoux wrote:

> I did a short proof of concept for the YAML part here
> https://github.com/BertrandDechoux/flume-ng-yaml
> 
> Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look at what was
done in that issue and see what can be done for a clean selection of a custom configuration
provider.
> 
> Bertrand
> 
> 
> On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)>
wrote:
> > I actually do like the idea of pluggable config provider. Like you mentioned, it
should be possible to hack the Application class, and use a CLI arg to select the config provider.
If no CLI arg is provided, just fall back to the current mechanism. If you submit a patch,
I'd be glad to review it :-) 
> > 
> > 
> > Thanks
> > Hari
> > 
> > -- 
> > Hari Shreedharan
> > 
> > 
> > On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
> > 
> > > It seems quite easier than dealing with commons configuration.
> > > 
> > > I might take a shot at it during the week-end, if I find the time.
> > > 
> > > About the customisation of the Application class, that's something which is
already mostly done by
> > > https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from
Zookeeper watcher
> > > 
> > > It might be a good idea to split this improvement into :
> > > 1) Selection of an alternative configuration provider
> > > 2) A zookeeper based configuration provider
> > > 
> > > If point 1 is done, anyone could use her/his custom configuration provider
by dropping the implementation as a third party jar and then selecting it with the help of
the command line.
> > > 
> > > Bertrand
> > > 
> > > 
> > > On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <dechouxb@gmail.com (mailto:dechouxb@gmail.com)>
wrote:
> > > > Thanks for the hints!
> > > > 
> > > > The part that you quoted is in fact from the flume wiki and I forgot to
quote it myself. But the page is quite old so I wasn't sure about the state of the feature.
I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or
more exactly I lost myself in this dependency, which might be used by one of my current projects
somehow).
> > > > 
> > > > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> > > > 
> > > > > Configuration Provider
> > > > > 
> > > > > Flume NG has a pluggable configuration system called the configuration
> > > > > provider. By default, Flume NG ships with a Java property file based
> > > > > configuration system that is both simple and easy to generate
> > > > > programmatically. Flume OG has a centralized configuration system
with
> > > > > a master and ZooKeeper for coordination and we recognize this is
very
> > > > > appealing to some users where as others see it as overhead they simply
> > > > > don't want. We opted to make this a pluggable extension point and
ship
> > > > > a basic implementation that would let many users get started quickly
> > > > > and easily. There's almost certainly enough desire for a similar
> > > > > implementation to that of Flume OG, but it isn't yet implemented.
> > > > > Users may also implement arbitrary plugins to integrate with any
type
> > > > > of configuration system (JSON files, a shared RDBMS, a central system
> > > > > with ZooKeeper, and so forth). We see this as something more
> > > > > interesting to system integrators.
> > > > 
> > > > I will look at the classes you pointed for my own education and I will
see what I can do.
> > > > 
> > > > Thanks a lot.
> > > > 
> > > > Bertrand
> > > > 
> > > > 
> > > > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)>
wrote:
> > > > > Hi,
> > > > > 
> > > > > "Flume NG has a pluggable configuration system called the configuration
provider." 
> > > > > 
> > > > > There is an object hierarchy, but I don't consider it pluggable as
there is no place to "plugin" your own implementation. I consider these things "private" to
Flume. Meaning that there is no guaranteed API stability between releases. With that caveat,
you'd want to implement the following abstract class: 
> > > > > 
> > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
> > > > > 
> > > > > e.g https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java

> > > > > 
> > > > > and then'd you have to customize the Application class to "plugin"
your own implementation:
> > > > > 
> > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java

> > > > > 
> > > > > Brock
> > > > > 
> > > > > 
> > > > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <dechouxb@gmail.com
(mailto:dechouxb@gmail.com)> wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > Flume NG has a pluggable configuration system called the configuration
provider. I cloned the source repository and I indeed found it but I am a bit lost. Is there
some kind of short guide explaining how to write a custom configuration provider?
> > > > > > 
> > > > > > I would like to start with something simple : using a YAML configuration
file instead of a properties file.
> > > > > > It should be almost trivial to change the following configuration
> > > > > > 
> > > > > > agent1.channels.ch1.type = memory
> > > > > > 
> > > > > > agent1.sources.avro-source1.channels = ch1
> > > > > > agent1.sources.avro-source1.type = avro
> > > > > > agent1.sources.avro-source1.bind = 0.0.0.0
> > > > > > agent1.sources.avro-source1.port = 41414
> > > > > > 
> > > > > > agent1.sinks.log-sink1.channel = ch1
> > > > > > agent1.sinks.log-sink1.type = logger
> > > > > > 
> > > > > > agent1.channels = ch1
> > > > > > agent1.sources = avro-source1
> > > > > > agent1.sinks = log-sink1
> > > > > > 
> > > > > > into
> > > > > > 
> > > > > > agent1:
> > > > > > 
> > > > > >     channels:
> > > > > >         ch1:
> > > > > >             type: memory
> > > > > > 
> > > > > >     sources:
> > > > > >         avroSource1:
> > > > > >             channels: ch1
> > > > > >             type: avro
> > > > > >             bind: 0.0.0.0
> > > > > >             port: 41414
> > > > > > 
> > > > > >     sinks:
> > > > > >         logSink1:
> > > > > >             channel: ch1
> > > > > >             type: logger
> > > > > > 
> > > > > > 
> > > > > > At least, I know how to convert the latter into the former but
I am not sure how this transformation can be included into flume-ng using a configuration
provider.
> > > > > > I have not been able to find much information about the customisation
of the configuration provider. Did I miss any documentation?
> > > > > > 
> > > > > > 
> > > > > > Bertrand
> > > > > 
> > > > > 
> > > > > 
> > > > > -- 
> > > > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

> > > 
> > > 
> > > 
> > > -- 
> > > Bertrand Dechoux 
> > 
> 
> 
> 
> -- 
> Bertrand Dechoux 


Mime
View raw message