flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Channel locks
Date Tue, 19 May 2015 15:30:22 GMT
Each sink does not hold a lock on the channel. Channels can handle multiple
sinks reading from them and in fact, that is encouraged.

On Tuesday, May 19, 2015, Sverre Bakke <sverre.bakke@gmail.com> wrote:

> Hi,
>
> When using a single source, channel and sink setup with Syslog, memory
> channel and KafkaSink (from
> https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing
> performance issues.
>
> After looking at the source code of the sink, it seems that the sink
> will begin a transaction, then do everything (e.g. compress, send over
> the network, wait for ack), and eventually close/commit the
> transaction after everything is done.
>
> Is there any way to increase the performance of this setup?
>
> Earlier I have seen people proposing adding more sinks to get higher
> throughput, but if every sink holds a lock on the channel until done
> with processing, then I would assume that this only causes additional
> performance issues? Are my understanding correct on how channel
> locking works?
>
> Regards,
> Sverre
>


-- 

Thanks,
Hari

Mime
View raw message