flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Transactional Multiplex (fan out) Sink
Date Tue, 31 Jul 2012 12:43:30 GMT
I would create a JIRA for this. FWIW, many people run multiple
channels per host so I don't necessarily agree with this:

"But in our case, the fan out is replicating in most cases. If using
the current "Replicating to Channels" configuration, we will get
several identical channels on the same host, which may consume a large
amount resources (memory, disk, etc.). The performance may possibly
drop."

Brock

On Tue, Jul 31, 2012 at 5:09 AM, Wang, Yongkun | Yongkun | BDD
<yongkun.wang@mail.rakuten.com> wrote:
> Any suggestions on this proposal? It won't break current design.
>
> Regards,
> Yongkun Wang
>
> From: <Wang>, "Wang, Yongkun | Yongkun | BDD" <yongkun.wang@mail.rakuten.com<mailto:yongkun.wang@mail.rakuten.com>>
> Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
> Date: 2012年7月25日水曜日 19:32
> To: "dev@flume.apache.org<mailto:dev@flume.apache.org>" <dev@flume.apache.org<mailto:dev@flume.apache.org>>
> Cc: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
> Subject: Transactional Multiplex (fan out) Sink
>
> Hi,
>
> In our system, we need to fan out the aggregated flow to several destinations. Usually
the flow to each destination is identical.
>
> There is a nice feature of NG, the "multiplexing flow", which can satisfy our requirements.
It is implemented by using separated channels, which is easy to do transaction control.
>
> But in our case, the fan out is replicating in most cases. If using the current "Replicating
to Channels" configuration, we will get several identical channels on the same host, which
may consume a large amount resources (memory, disk, etc.). The performance may possibly drop.
And the events to each destination may not be synchronized.
>
> I read NG source, I think I could move the multiplex from Channel to Sink, that is, using
single Channel, fan out to different Sinks, which may solve the problems (resource usage,
performance, event synchronization) of multiple Channels.
>
> I see that there are "LoadBalancingSinkProcessor" and "SinkSelector" classes, but they
cannot be used to achieve the target of replicating events from one Channel to different Sinks.
>
> The following is an optional implementation of the Transactional Multiplex (or fan out)
Sink:
>
>   1.  Add a Transactional Multiplex Sink Processor, which will group the operations of
all fan out Sinks into one transaction, and use a certain policy to commit the transaction.
>   2.  Add MultiplexSink, which simply processes the Events and report status, no transaction.
>   3.  Add "peek()" and "remove()" to Channel and Transaction.
>
> The policy of committing a transaction can be defined as follow (suppose we have N Sinks)
:
>
>   1.  When M(0=<M<=N) Sinks succeed;
>   2.  When specified M(0<M<=N) Sinks (important sinks) succeed.
>
> A selector can also be used by Transactional Multiplex Sink Processor to filter the events
for some Sinks (Optional).
>
> And this can be combined with the existing Multiplex Channel Flow: Multiplex events into
different Channels, each Channel can replicate to different Sinks.
>
> Would like to hear your suggestions firstly.
> If it is reasonable, I will create a ticket in JIRA and provide the patch for review.
>
> Cheers,
> Yongkun Wang (Kun)



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Mime
View raw message