flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: Channel locks
Date Tue, 19 May 2015 16:24:24 GMT
distributed between the sinks.

(Is that VSRE enough? ;)

On Tue, May 19, 2015 at 7:18 PM, Needham, Guy <Guy.Needham@virginmedia.co.uk
> wrote:

>  With multiple sinks reading from one channel, will each sink read each
> event, or will the events be distributed between the sinks?
>
>
>
> Regards,
>
> Guy Needham | Data Discovery
> Virgin Media   | Technology and Transformation | Data
> Bartley Wood Business Park, Hook, Hampshire RG27 9UP
> D 01256 75 3362
> I welcome VSRE emails. Learn more at http://vsre.info/
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> *Sent:* 19 May 2015 16:30
> *To:* user@flume.apache.org
> *Subject:* Re: Channel locks
>
>
>
> Each sink does not hold a lock on the channel. Channels can handle
> multiple sinks reading from them and in fact, that is encouraged.
>
> On Tuesday, May 19, 2015, Sverre Bakke <sverre.bakke@gmail.com> wrote:
>
> Hi,
>
> When using a single source, channel and sink setup with Syslog, memory
> channel and KafkaSink (from
> https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing
> performance issues.
>
> After looking at the source code of the sink, it seems that the sink
> will begin a transaction, then do everything (e.g. compress, send over
> the network, wait for ack), and eventually close/commit the
> transaction after everything is done.
>
> Is there any way to increase the performance of this setup?
>
> Earlier I have seen people proposing adding more sinks to get higher
> throughput, but if every sink holds a lock on the channel until done
> with processing, then I would assume that this only causes additional
> performance issues? Are my understanding correct on how channel
> locking works?
>
> Regards,
> Sverre
>
>
>
> --
>
>
>
> Thanks,
>
> Hari
>
>
>
>
> --------------------------------------------------------------------
> Save Paper - Do you really need to print this e-mail?
>
> Visit www.virginmedia.com for more information, and more fun.
>
> This email and any attachments are or may be confidential and legally
> privileged
> and are sent solely for the attention of the addressee(s). If you have
> received this
> email in error, please delete it from your system: its use, disclosure or
> copying is
> unauthorised. Statements and opinions expressed in this email may not
> represent
> those of Virgin Media. Any representations or commitments in this email are
> subject to contract.
>
> Registered office: Media House, Bartley Wood Business Park, Hook,
> Hampshire, RG27 9UP
> Registered in England and Wales with number 2591237
>

Mime
View raw message