flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@apache.org>
Subject Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
Date Tue, 22 Jan 2013 18:43:50 GMT
I'd recommend using the FileChannel instead of the JDBC Channel. The
FileChannel would give you better performance and the same reliability
guarantees as the JDBC Channel.

To answer your question - you can insert custom headers in an agent using
Interceptors. You can write an interceptor which inserts the headers based
on the content of the event etc. Interceptors are available in version 1.2+.


Hari


On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <aladdin_1973@hotmail.com> wrote:

>
> Hi,
> I'd like to use the multiplexing feature of Flume-NG, routing events from
> one source to 2 jdbc-channels.
> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
> Moreover, the multiplexing feature can base its routing on headers. But no
> information is given on how to setup custom headers (the example in the
> documentation uses a "State" header).
> I'm using a Log4jAppender to write to an AvroSource.
>
> # list the sources, sinks and channels in the agent
> agent_foo.sources = avro-AppSrv-source1
> agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
> agent_foo.channels = jdbc-channel-1 jdbc-channel-2
>
> # set channels for source
> agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1
> jdbc-channel-2
>
> # set channel for sinks
> agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
> agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2
>
> # channel selector configuration
> agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
> agent_foo.sources.avro-AppSrv-source1.selector.header = State
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1
> jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1
>
> Thanks for your help,
> Best regards

Mime
View raw message