flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terrey shih <terreys...@gmail.com>
Subject Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Fri, 08 Aug 2014 20:20:28 GMT
Hi,

One more thing.  I would like to know if one serializer can be used for two
table.

thanks,
-Gary

server-agent.sources = mySrc
server-agent.sinks = hbase-sink1 hbase-sink2
server-agent.channels = C1 C2

# Describe/configure the source
server-agent.sources.mySrc.type = avro
server-agent.sources.mySrc.bind = 127.0.0.1
server-agent.sources.mySrc.port = 5000

# Use a channel which buffers events in memory
server-agent.channels.C1.type = memory
server-agent.channels.C1.capacity = 1000
server-agent.channels.C1.transactionCapacity = 100
server-agent.sinks.hbase-sink1.type = asynchbase
server-agent.sinks.hbase-sink1.table = table1
server-agent.sinks.hbase-sink1.columnFamily = fam1
server-agent.sinks.hbase-sink1.batchSize = 1000
server-agent.sinks.hbase-sink1.serializer =
com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink1.channel = C1
server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3

server-agent.channels.C2.type = memory
server-agent.channels.C2.capacity = 1000
server-agent.channels.C2.transactionCapacity = 100
server-agent.sinks.hbase-sink2.type = asynchbase
server-agent.sinks.hbase-sink2.table = table2
server-agent.sinks.hbase-sink2.columnFamily = fam2
server-agent.sinks.hbase-sink2.batchSize = 1000
server-agent.sinks.hbase-sink2.serializer =
com.test.flume.server.HBaseSinkSerializer
server-agent.sinks.hbase-sink2.channel = C2
server-agent.sinks.hbase-sink2.serializer.columns = table2Col

# Bind the source and sink to the channel
server-agent.sources.mySrc.channels = C1 C2
server-agent.sources.avor-Src.selector.type = multiplexing
server-agent.sources.avor-Src.selector.header = DataSrc
server-agent.sources.avor-Src.selector.mapping.ABC = C1
server-agent.sources.avor-Src.selector.mapping.BCD = C2



On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <terreyshih@gmail.com> wrote:

> HI,
>
> Here is my sample config.
>
>
>
>
> On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> Can you please send your config? That would make it easier to understand.
>>
>>
>> terrey shih wrote:
>>
>>
>> I have a fanning out operation where I have one source and and based
>> on the event headers (headers are added from the source input), I
>> would like to channel the event to different Hbase tables.  I am using
>> the same serializer for the hbase tables.
>>
>>
>

Mime
View raw message