flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a2b2...@gmail.com
Subject Re: Plugin Support in Flume-NG
Date Sat, 05 May 2012 22:22:12 GMT
I recommend modifying the following flume plugin to work with flume-ng. We use it in production
on 0.9.x with good results. 


We opted to fanout incoming events to the Cassandra plugin for storage and then a thrift based
service which handles aggregation and Cassandra counter updates separately. 

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Eric Sammer <esammer@cloudera.com>
Date: Sat, 5 May 2012 15:11:10 
To: <flume-user@incubator.apache.org>
Reply-To: flume-user@incubator.apache.org
Subject: Re: Plugin Support in Flume-NG


Sources, sinks, channels, configuration providers, event serializers,
compression codecs, and channel processors are all plugin interfaces. We've
significantly expanded what can be extended, although some are far more
likely to be extended (the rest are for system implementors / integrators).

For a simple example of a sink, check out NullSink or the slightly more
interesting LoggerSink. Both live in the flume-ng-core module; shouldn't be
hard to find. If you want to see how to write a standalone Maven module
with your own sink, check out the flume-ng-sinks module which contains some
implementations that aren't part of flume-ng-core (that's the preferred way
to build your own plugins). The HDFSEventSink is a really good example of
building a sink that isn't part of Flume's core, proper, but still uses
many of the more complex features a sink can use.

Let us know if you hit any barriers. Also, we'd really love to see an open
source Cassandra sink plugin, hint, hint. :)

On Sat, May 5, 2012 at 3:00 PM, Tejinder Aulakh <tejinder@sharethis.com>wrote:

> Hi Mike,
> Basically, I need two plugins:
> 1) A custom sink to process raw logs and extract/prepare the relevant data
> which will be stored in Cassandra.
> 2) Another plugin (sink) to write the data to cassandra. This plugin also
> needs to perform cassandra increment operations.
> Thanks,
> TJ
> On Fri, May 4, 2012 at 6:35 PM, Mike Percy <mpercy@cloudera.com> wrote:
>> Hi TJ,
>> What are you trying to do w/ the plugins?
>> Mike
>> On May 4, 2012, at 5:27 PM, Tejinder Aulakh <tejinder@sharethis.com>
>> wrote:
>> Flume 0.9.x has a very good support for custom plugins. It comes with
>> HelloWorld plugins which can be easily modified for your needs.
>> https://github.com/cloudera/flume/tree/master/plugins
>> However, I'm not able to find any help/documentation on how to write
>> plugins for FlumeNG which has a totally different architecture than
>> Flume 0.9.x
>> Does anyone know how to write custom plugins for Flume NG.
>> TJ
> --
> Thanks,
> TJ Aulakh
> Senior Software Engineer, ShareThis
> tejinder@sharethis.com
> Cell: (510)708-2499
> 250 Cambridge Ave, Suite 103,
> Palo Alto CA 94306

Eric Sammer
twitter: esammer
data: www.cloudera.com

View raw message