flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Ortiz <konstt2...@gmail.com>
Subject Re: Interceptor, multiplexing Events to two differents path in HDFS dynamically,
Date Wed, 30 Jul 2014 16:06:57 GMT
I think that it's exactly what I need, thanks.


2014-07-30 17:46 GMT+02:00 Matt Tenenbaum <matt.tenenbaum@rockyou.com>:

> Hi Guillermo --
>
> If I understand correctly, you want to create two paths. Events that pass
> the validating interceptor get directed to one path in HDFS and events that
> fail get directed to another. If so, you could define your interceptor to
> set a 'valid' header to 'yes' or 'no' depending on its internal logic. Then
> you would be able to use the ordinary 'selector.mapping' configuration
> behavior to route to separate channels and attach your separate
> destinations as sinks to each of those channels. I'm omitting all but the
> relevant configurations:
>
>
> multi.sources = s1
> multi.sinks = valid invalid
> multi.channels = cpos cneg
>
> multi.sources.s1.selector.type = multiplexing
> # Route based on the value of the 'valid' header set by the interceptor
> multi.sources.s1.selector.header = valid
> # Send events with valid == yes to 'cpos' channel
> multi.sources.s1.selector.mapping.yes = cpos
> # Send events with valid == no to 'cneg' channel
> multi.sources.s1.selector.mapping.no = cneg
>
> multi.sinks.valid.type = hdfs
> multi.sinks.valid.channel = cpos
> multi.sinks.valid.hdfs.path = hdfs://nn/path/for/valid/data
> # Other configurations for valid data...
>
> multi.sinks.invalid.type = hdfs
> multi.sinks.invalid.channel = cneg
> multi.sinks.invalid.hdfs.path = hdfs://nn/path/for/invalid/data
> # Other configuration for data that fails validation...
>
> Would that do what you want?
>
> Cheers
> -mt
>
>
> On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <konstt2000@gmail.com>
> wrote:
>
>> Hello,
>>
>>
>> I want to create an interceptor to validate and multiplexing the Events,
>> if they don't get validated to send them a different path in HDFS.
>>
>> Is it possible to do it?
>> I thought.. to have others interceptor and the HDFS path is built on base
>> of these static interceptors, so I could modify them on my validator
>> interceptor. Is it fine?
>>
>
>

Mime
View raw message