kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: Cross-site Kafka installation
Date Mon, 21 Jan 2013 23:26:15 GMT
We do mirroring across data-centers (but in the same continent). You should
basically set a high fetch size and socket buffer size in such scenarios.

In general, you should set a high value for the socket buffer size on the
consumer configuration (socket.buffersize) and the source cluster's broker
configuration (socket.send.buffer).

Assuming you are using the high-level consumer, the fetch size (fetch.size)
should be higher than the consumer's socket buffer size. Note that the
socket buffer size configurations are a hint to the underlying platform's
networking code. If you enable trace logging, you can check the actual
receive buffer size and determine whether the setting in the OS networking
layer also needs to be adjusted. Likewise, you will need to use higher
connection/session timeouts for zookeeper and set your offset commit
intervals to be fairly large.

Thanks,

Joel


On Mon, Jan 21, 2013 at 2:04 PM, Pablo Barrera González <
pablo.barrera@gmail.com> wrote:

> Hello
>
> In my enterprise we are deploying an cross-site installation of Kakfa. One
> of the Kafka cluster is located in USA and one consumer is in Europe. Does
> anybody have experience in such an environment? Any comments on the
> configuration and best practices?
>
> Thanks in advance
>
> Pablo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message