flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Multiple Channels for a single source
Date Mon, 23 Apr 2012 06:14:47 GMT
Thanks Will. I need to try it. Well my scenario is something like this:

I have two agents namely agent1 and agent2.

Agent1 : Source is exec, which tails the output of the log file and Sink is
Avro Sink and channel used is memory.

Agent 2. Source is Avro source which picks up the events from the Avro sink
from agent1 and send it to Esper Sink.Here also channel is memory.
Basically the Esper sink is used to execute EPL (Event Processing Language)
query on the incoming events, basically filtering the events based on some

Now what i want to achieve is something like this:
Since the logs consist of 3 different types of events, so i will have 3
different channels so as not to overload the channel. And i will have 3
different Esper Sinks, which will accumulate the events (a second level
filtering will be employed)

As you have mentioned that, based on the event headers we can direct the
events to different channels, but in my case headers are empty. So how can
i achieve this, everything is coming as body. Or since i am using exec
source,it takes everything as body?

Som Shekhar

On Fri, Apr 20, 2012 at 9:04 PM, Will McQueen <will@cloudera.com> wrote:

> Here's a sample config, based on the user guide example
> //flume.conf
> ===============
> # a = agent
> # c = channel
> # r = source
> # k = sink
> # g = sink group
> agent.channels = c1 c2 c3
> agent.sources = r1
> agent.sinks = k1 k2 k3
> #agent.sinkgroups =
> agent.channels.c1.type = MEMORY
> agent.channels.c2.type = MEMORY
> agent.channels.c3.type = MEMORY
> agent.sources.r1.type = org.example.MySource
> agent.sources.r1.channels = c1 c2 c3
> agent.sources.r1.bind =
> agent.sources.r1.port = 41414
> agent.sources.r1.selector.type = multiplexing
> agent.sources.r1.selector.header = state
> agent.sources.r1.selector.mapping.CA = c1
> agent.sources.r1.selector.mapping.TX = c2
> agent.sources.r1.selector.mapping.HI = c3
> agent.sources.r1.selector.default = c1
> agent.sinks.k1.type = LOGGER
> agent.sinks.k1.channel = c1
> agent.sinks.k2.type = NULL
> agent.sinks.k2.channel = c2
> agent.sinks.k3.type = org.example.MySink
> agent.sinks.k3.channel = c3
> agent.sinks.k3.mysinkprop = mysinkvalue
> ===============
> Cheers,
> Will
> On Fri, Apr 20, 2012 at 7:52 AM, Will McQueen <will@cloudera.com> wrote:
>> Hi Som,
>> It sounds like you want to use a multiplexing channel selector and custom
>> source. Please see the Flume NG User's Guide for discussion of multiplexing
>> channel selector, under "Fan out flow".
>> I'm looking at latest Apache trunk (commit
>> 37f3597e6fdcbbc7dd43493d0327076868733a09). After I compile the code with
>> maven, I point my web browser to open:
>>      flume-ng-doc/xhtml/FlumeUserGuide.xhtml
>> (there's also FlumeDeveloperGuide.xhtml)
>> One scenario is that your source component can process each incoming
>> event and assign a kvp (header and header value) to each one. To assign,
>> you could use something like:
>>      channelProcessor.processEvent(EventBuilder.withBody(body, headers));
>> ...within your source component, and the header you specify would need to
>> be specified in the config file. Then based on the value of that header,
>> the event would be directed (based on the header value) to the channel
>> specified in the config file.
>> Another scenario is that the incoming event already contains the header
>> that your source component needs (eg, when you're using an avro source), in
>> which case your agent would just propagate the event to the desired
>> channel, again according to the flume config file.
>> Cheers,
>> Will
>> On Fri, Apr 20, 2012 at 4:30 AM, shekhar sharma <shekhar2581@gmail.com>wrote:
>>> Hello,
>>> Is it possible to assign  multiple channels from a source and send it to
>>> different sinks in Flume-NG. My use case is something like this:
>>>  (I am currently using trunk)
>>> (A) I have three types of events coming from the web logs lets say
>>> E1,E2,E3 and ofcourse they can come in any order. Lets say i am using exec
>>> source which will tail the logs of the log file to the flume source
>>> (B) I will filter the incoming events to the source based on certain
>>> parameters and send it to designated channel, let's say Ch1, Ch2 and Ch3
>>> for E1, E2 and E3. Since i have already done the integration of Esper as
>>> flume's custom sink, so i think/believe can fire EPL query on the incoming
>>> events and depending on the type of event E1, E2 or E3, i will send to
>>> channels (let it be memory channel) Ch1, Ch2, or Ch3.
>>> (3)Then  i will have 3 different sinks S1, S2, S3 which will take the
>>> events from Ch1, Ch2 and Ch3 respectively.
>>> How Can i achieve this using flume-ng? There was one discussion
>>> http://www.mail-archive.com/flume-user@incubator.apache.org/msg00617.htmland
>>> https://issues.apache.org/jira/browse/FLUME-930.
>>> Regards,
>>> Som Shekhar Sharma

View raw message