flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang, Yongkun | Yongkun | BDD" <yongkun.w...@mail.rakuten.com>
Subject Re: Transactional Multiplex (fan out) Sink
Date Wed, 01 Aug 2012 08:36:04 GMT
Hi Brock, 

Thanks for commenting on this post.

If the contents of each channel is different, using multiple channel makes
But if the source simply replicating the event to several channels,
maintaining a bunch of identical channels on the same host may not be an
optimized solution. So I want to try the proposed fan out sink.


On 12/07/31 21:43, "Brock Noland" <brock@cloudera.com> wrote:

>I would create a JIRA for this. FWIW, many people run multiple
>channels per host so I don't necessarily agree with this:
>"But in our case, the fan out is replicating in most cases. If using
>the current "Replicating to Channels" configuration, we will get
>several identical channels on the same host, which may consume a large
>amount resources (memory, disk, etc.). The performance may possibly
>On Tue, Jul 31, 2012 at 5:09 AM, Wang, Yongkun | Yongkun | BDD
><yongkun.wang@mail.rakuten.com> wrote:
>> Any suggestions on this proposal? It won't break current design.
>> Regards,
>> Yongkun Wang
>> From: <Wang>, "Wang, Yongkun | Yongkun | BDD"
>> Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>"
>> Date: 2012年7月25日水曜日 19:32
>> To: "dev@flume.apache.org<mailto:dev@flume.apache.org>"
>> Cc: "user@flume.apache.org<mailto:user@flume.apache.org>"
>> Subject: Transactional Multiplex (fan out) Sink
>> Hi,
>> In our system, we need to fan out the aggregated flow to several
>>destinations. Usually the flow to each destination is identical.
>> There is a nice feature of NG, the "multiplexing flow", which can
>>satisfy our requirements. It is implemented by using separated channels,
>>which is easy to do transaction control.
>> But in our case, the fan out is replicating in most cases. If using the
>>current "Replicating to Channels" configuration, we will get several
>>identical channels on the same host, which may consume a large amount
>>resources (memory, disk, etc.). The performance may possibly drop. And
>>the events to each destination may not be synchronized.
>> I read NG source, I think I could move the multiplex from Channel to
>>Sink, that is, using single Channel, fan out to different Sinks, which
>>may solve the problems (resource usage, performance, event
>>synchronization) of multiple Channels.
>> I see that there are "LoadBalancingSinkProcessor" and "SinkSelector"
>>classes, but they cannot be used to achieve the target of replicating
>>events from one Channel to different Sinks.
>> The following is an optional implementation of the Transactional
>>Multiplex (or fan out) Sink:
>>   1.  Add a Transactional Multiplex Sink Processor, which will group
>>the operations of all fan out Sinks into one transaction, and use a
>>certain policy to commit the transaction.
>>   2.  Add MultiplexSink, which simply processes the Events and report
>>status, no transaction.
>>   3.  Add "peek()" and "remove()" to Channel and Transaction.
>> The policy of committing a transaction can be defined as follow
>>(suppose we have N Sinks) :
>>   1.  When M(0=<M<=N) Sinks succeed;
>>   2.  When specified M(0<M<=N) Sinks (important sinks) succeed.
>> A selector can also be used by Transactional Multiplex Sink Processor
>>to filter the events for some Sinks (Optional).
>> And this can be combined with the existing Multiplex Channel Flow:
>>Multiplex events into different Channels, each Channel can replicate to
>>different Sinks.
>> Would like to hear your suggestions firstly.
>> If it is reasonable, I will create a ticket in JIRA and provide the
>>patch for review.
>> Cheers,
>> Yongkun Wang (Kun)
>Apache MRUnit - Unit testing MapReduce -

View raw message