flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Bihani <jagadish.bih...@pubmatic.com>
Subject Re: "single source - multi channel" scenario and applying interceptor while writing to only one channel and not on others...possible approaches
Date Tue, 23 Apr 2013 09:02:16 GMT

Thanks Jeff and Connor. That was helpful.

@Connor :

  We have  a scenario where we receive logs where each record contains 
fields (around 100). We want to send every field of every record as it 
is to hdfs and send only
"subset" of fields of "every"record to another sink (say Storm).
    So in this we don't filter records, we are filtering some fields of 
every record in
  one channel and none in other.

Hence, apparently the solution of channel selector will not work here 
and custom event serializer might need to be written.
As mentioned by you if there are any directions related to that are 
available it will be of great help.


On 04/23/2013 12:22 PM, Connor Woodson wrote:
> Some more thoughts on this:
> The way Interceptors are currently set to work is that they apply to 
> an event as it is received. There are good uses for this - for 
> instances, it allows easily configuring a single Timestamp interceptor 
> that gives all events a source receives a timestamp, so even if you 
> have multiple sinks/channels responding to an event, you only have 
> that one interceptor. Interceptors in this sense serve to add data to 
> event headers, and as such it makes sense to have them applied only 
> once by the source instead of letting the channels change header data.
> If you wish to use an interceptor in the above way, to modify header 
> data, and still want that interceptor to apply for a single channel, 
> then if you don't mind could you elaborate on what you are trying to 
> do? I haven't been able to come up with a situation like that. The 
> solution here would be to do as Jeff suggested and use a serializer; 
> if you want more in-depth instructions on how to build it, please ask; 
> I have a set of directions lying around somewhere that I'll find for you.
> However, the way Interceptors work I have myself faced a situation 
> where I would like the interceptors to be channel only. This use case 
> is when I want to use an Interceptor to filter events; I want to send 
> an event to some subset of channels based on the contents of its data. 
> Here is how you can do this in the current setup (where Interceptors 
> are applied at the source instead of per-channel):
> Using the Multiplexing Channel Selector you are able to choose which 
> channels an event is written to based off of the value of a specified 
> header (documentation in that link). There are some more features to 
> the selector that aren't documented, called Optional Channels or 
> something, but I don't know very much about them - just figured I 
> would point out that they exist; digging through the source should 
> provide some more insight.
> So here is how you want to set your system up. Create an Interceptor 
> that will define a certain header value based off of the event's 
> contents. For instance, if you want all events containing exactly 1 
> character to be sent to a channel, you could create an Interceptor 
> that counts the characters in the event. Then that Interceptor will 
> set a certain header value to "SINGLE" if there is just one character, 
> or "MULTIPLE" if there are more.
> Then you can create your channel selector like this (modified from the 
> documentation example):
> a1.sources = r1
> a1.channels = all_events single_events multiple_events
> a1.sources.r1.interceptors = your_interceptor
> a1.sources.r1.interceptors.your_interceptor.header = header
> a1.sources.r1.selector.type = multiplexing
> a1.sources.r1.selector.header = header
> a1.sources.r1.selector.mapping.SINGLE = all_events single_events
> a1.sources.r1.selector.mapping.MULTIPLE = all_events multiple_events
> a1.sources.r1.selector.default = all_events
> The result is that now you have created a way to filter which channels 
> a certain event is sent to. Note that a channel can appear more than 
> once - for instance, all_events will get all events. And so the trick 
> is to just define the right interceptor (which are much simpler to 
> code than a serializer (which itself is fairly easy)).
> Hopefully that was clear. Feel free to ask more questions,
> - Connor
> On Fri, Apr 19, 2013 at 11:14 AM, Jeff Lord <jlord@cloudera.com 
> <mailto:jlord@cloudera.com>> wrote:
>     Jagadish,
>     Here is an example of how to write a custom serializer.
>     https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MyCustomSerializer.java
>     -Jeff
>     On Fri, Apr 19, 2013 at 9:34 AM, Jeff Lord <jlord@cloudera.com
>     <mailto:jlord@cloudera.com>> wrote:
>         Hi Jagadish,
>         Have you considered using a custom event serializer to modify
>         your event?
>         Its possible to replicate your flow using two channels and
>         then have one sink that implements a custom serializer to
>         modify the event.
>         -Jeff
>         On Tue, Apr 16, 2013 at 11:12 PM, Jagadish Bihani
>         <jagadish.bihani@pubmatic.com
>         <mailto:jagadish.bihani@pubmatic.com>> wrote:
>             Hi
>             If anybody has any inputs on this that will surely help.
>             Regards,
>             Jagadish
>             On 04/16/2013 12:06 PM, Jagadish Bihani wrote:
>                 Hi
>                 We have a use case in which
>                 1. spooling source reads data.
>                 2. It needs to write events into multiple channels. It
>                 should apply
>                 interceptor only when putting into one channel and
>                 should put
>                 the event as it is while putting into another channel.
>                 Possible approach we have thought:
>                 1. Create  2 different sources and then apply
>                 interceptor on one and dont
>                 apply on other. But that duplicates reads and
>                 increases IO.
>                 Is there any better way of achieving this use case?
>                 Regards,
>                 Jagadish

View raw message