flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: question using multiplexing and the the same serializer for multiple sinks to multiple hbase tables
Date Sat, 09 Aug 2014 03:39:42 GMT
Nice catch! Yep, that sounds like an issue.

On Friday, August 8, 2014, Paul Chavez <pchavez@ntent.com> wrote:

>  There is a configuration error in your multiplexing channel selector
> section. You are referencing ‘server-agent.sources.avor-Src.’ and it should
> be ‘server-agent.sources.mySrc.’. Otherwise, the configuration looks good
> and should satisfy your requirements.
>
>
>
> *From:* terrey shih [mailto:terreyshih@gmail.com
> <javascript:_e(%7B%7D,'cvml','terreyshih@gmail.com');>]
> *Sent:* Friday, August 08, 2014 1:20 PM
> *To:* user@flume.apache.org
> <javascript:_e(%7B%7D,'cvml','user@flume.apache.org');>
> *Subject:* Re: question using multiplexing and the the same serializer
> for multiple sinks to multiple hbase tables
>
>
>
> Hi,
>
> One more thing.  I would like to know if one serializer can be used for
> two table.
>
> thanks,
>
> -Gary
>
> server-agent.sources = mySrc
> server-agent.sinks = hbase-sink1 hbase-sink2
> server-agent.channels = C1 C2
>
> # Describe/configure the source
> server-agent.sources.mySrc.type = avro
> server-agent.sources.mySrc.bind = 127.0.0.1
> server-agent.sources.mySrc.port = 5000
>
> # Use a channel which buffers events in memory
> server-agent.channels.C1.type = memory
> server-agent.channels.C1.capacity = 1000
> server-agent.channels.C1.transactionCapacity = 100
> server-agent.sinks.hbase-sink1.type = asynchbase
> server-agent.sinks.hbase-sink1.table = table1
> server-agent.sinks.hbase-sink1.columnFamily = fam1
> server-agent.sinks.hbase-sink1.batchSize = 1000
> server-agent.sinks.hbase-sink1.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink1.channel = C1
> server-agent.sinks.hbase-sink1.serializer.columns = col1,col2,col3
>
> server-agent.channels.C2.type = memory
> server-agent.channels.C2.capacity = 1000
> server-agent.channels.C2.transactionCapacity = 100
> server-agent.sinks.hbase-sink2.type = asynchbase
> server-agent.sinks.hbase-sink2.table = table2
> server-agent.sinks.hbase-sink2.columnFamily = fam2
> server-agent.sinks.hbase-sink2.batchSize = 1000
> server-agent.sinks.hbase-sink2.serializer =
> com.test.flume.server.HBaseSinkSerializer
> server-agent.sinks.hbase-sink2.channel = C2
> server-agent.sinks.hbase-sink2.serializer.columns = table2Col
>
> # Bind the source and sink to the channel
> server-agent.sources.mySrc.channels = C1 C2
> server-agent.sources.avor-Src.selector.type = multiplexing
> server-agent.sources.avor-Src.selector.header = DataSrc
> server-agent.sources.avor-Src.selector.mapping.ABC = C1
> server-agent.sources.avor-Src.selector.mapping.BCD = C2
>
>
>
> On Fri, Aug 8, 2014 at 11:19 AM, terrey shih <terreyshih@gmail.com
> <javascript:_e(%7B%7D,'cvml','terreyshih@gmail.com');>> wrote:
>
>  HI,
>
>
>
> Here is my sample config.
>
>
>
>
>
>
>
> On Fri, Aug 8, 2014 at 10:52 AM, Hari Shreedharan <
> hshreedharan@cloudera.com
> <javascript:_e(%7B%7D,'cvml','hshreedharan@cloudera.com');>> wrote:
>
>  Can you please send your config? That would make it easier to understand.
>
>
>
> terrey shih wrote:
>
>
> I have a fanning out operation where I have one source and and based
> on the event headers (headers are added from the source input), I
> would like to channel the event to different Hbase tables.  I am using
> the same serializer for the hbase tables.
>
>
>
>
>

Mime
View raw message