flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terrey shih <terreys...@gmail.com>
Subject Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Fri, 08 Aug 2014 22:09:28 GMT
Hi,

Yes.  The key and values are correct because I printed them out in my
serializer class..  Can I use the same serializer for for two different
channels for two hbase tables ? I should be able to right ?

thanks,
-Gary


On Fri, Aug 8, 2014 at 1:51 PM, Hari Shreedharan <hshreedharan@cloudera.com>
wrote:

> Do you see anything in the logs? If you are configuring your serializer
> correctly it should work fine. Are you sure your events come in with the
> header key DataSrc and values either ABC or BCD?
>
> terrey shih wrote:
>
>
> Hi,
>
> One more thing.  I would like to know if one serializer can be used
> for two table.
>
> thanks,
> -Gary
>
> server-agent.sources = mySrc
> server-agent.sinks = hbase-sink1 hbase-sink2
> server-agent.channels = C1 C2
>
> # Describe/configure the source
> server-agent.sources.mySrc.type = avro
> server-agent.sources.mySrc.bind = 127.0.0.1
> server-agent.sources.mySrc.port = 5000
>
> # Use a channel which buffers events in memory
> server-agent.channels.C1.type = memory
> server-agent.channels.C1.capacity = 1000
> server-agent.channels.C1.transactionCapacity = 100
> server-agent.sinks.hbase-sink1.type = asynchbase
> server-agent.sinks.hbase-sink1.table = table1
> server-agent.sinks.hbase-sink1.columnFamily = fam1
> server-agent.sinks.hbase-sink1.batchSize = 1000
> server-agent.sinks.hbase-sink1.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink1.channel = C1
> server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3
>
> server-agent.channels.C2.type = memory
> server-agent.channels.C2.capacity = 1000
> server-agent.channels.C2.transactionCapacity = 100
> server-agent.sinks.hbase-sink2.type = asynchbase
> server-agent.sinks.hbase-sink2.table = table2
> server-agent.sinks.hbase-sink2.columnFamily = fam2
> server-agent.sinks.hbase-sink2.batchSize = 1000
> server-agent.sinks.hbase-sink2.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink2.channel = C2
> server-agent.sinks.hbase-sink2.serializer.columns = table2Col
>
> # Bind the source and sink to the channel
> server-agent.sources.mySrc.channels = C1 C2
> server-agent.sources.avor-Src.selector.type = multiplexing
> server-agent.sources.avor-Src.selector.header = DataSrc
> server-agent.sources.avor-Src.selector.mapping.ABC = C1
> server-agent.sources.avor-Src.selector.mapping.BCD = C2
>
>
>
> On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <terreyshih@gmail.com
> <mailto:terreyshih@gmail.com>> wrote:
>
>     HI,
>
>     Here is my sample config.
>
>
>
>
>     On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan
>     <hshreedharan@cloudera.com <mailto:hshreedharan@cloudera.com>> wrote:
>
>         Can you please send your config? That would make it easier to
>         understand.
>
>
>         terrey shih wrote:
>
>
>
>         I have a fanning out operation where I have one source and
>         and based
>         on the event headers (headers are added from the source
>         input), I
>         would like to channel the event to different Hbase tables.  I
>         am using
>         the same serializer for the hbase tables.
>
>
>
>
>

Mime
View raw message