flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Tenenbaum <matt.tenenb...@rockyou.com>
Subject Re: Interceptor, multiplexing Events to two differents path in HDFS dynamically,
Date Wed, 30 Jul 2014 15:46:55 GMT
Hi Guillermo --

If I understand correctly, you want to create two paths. Events that pass
the validating interceptor get directed to one path in HDFS and events that
fail get directed to another. If so, you could define your interceptor to
set a 'valid' header to 'yes' or 'no' depending on its internal logic. Then
you would be able to use the ordinary 'selector.mapping' configuration
behavior to route to separate channels and attach your separate
destinations as sinks to each of those channels. I'm omitting all but the
relevant configurations:

multi.sources = s1
multi.sinks = valid invalid
multi.channels = cpos cneg

multi.sources.s1.selector.type = multiplexing
# Route based on the value of the 'valid' header set by the interceptor
multi.sources.s1.selector.header = valid
# Send events with valid == yes to 'cpos' channel
multi.sources.s1.selector.mapping.yes = cpos
# Send events with valid == no to 'cneg' channel
multi.sources.s1.selector.mapping.no = cneg

multi.sinks.valid.type = hdfs
multi.sinks.valid.channel = cpos
multi.sinks.valid.hdfs.path = hdfs://nn/path/for/valid/data
# Other configurations for valid data...

multi.sinks.invalid.type = hdfs
multi.sinks.invalid.channel = cneg
multi.sinks.invalid.hdfs.path = hdfs://nn/path/for/invalid/data
# Other configuration for data that fails validation...

Would that do what you want?


On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <konstt2000@gmail.com>

> Hello,
> I want to create an interceptor to validate and multiplexing the Events,
> if they don't get validated to send them a different path in HDFS.
> Is it possible to do it?
> I thought.. to have others interceptor and the HDFS path is built on base
> of these static interceptors, so I could modify them on my validator
> interceptor. Is it fine?

View raw message