flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: [KafkaChannel] About kafka producer type "sync" or "aync"
Date Mon, 13 Apr 2015 17:26:42 GMT
We implemented the KafkaChannel with the "sync" producer, because we
want to guarantee the transaction success and handle errors
appropriately.
We think it is worth the performance loss. Even with the sync
producer, we found that the Kafka Channel can handle 20K events per
second on my (pretty simple) setup. By tuning the batch size you can
have some control over the latency and throughput.

I also started testing the new Kafka producer (0.8.2.0 and up) - the
way we use the producer in the channel (batches), there was almost no
performance improvement.

Gwen

On Mon, Apr 13, 2015 at 9:41 AM, Tao Li <litao.buptsse@gmail.com> wrote:
> Hi all:
>
> JIRA: https://issues.apache.org/jira/browse/FLUME-2500
>
> I have a question about this issue.
>
> Flume support transaction between source and channel. So for kafka channel:
> 1. If we use "sync" kafka producer, it surely can guarantee the transaction,
> but sync send has low performance.
> 2. If we use "async" kafka producer, the performance will be better, but it
> can't guarantee the transaction.
>
> How do you think on it?

Mime
View raw message