kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Transferring events across data centers
Date Mon, 04 Feb 2013 05:25:44 GMT
Apoorva,

Kafka replication in 0.8 is designed for a Kafka cluster within the same
DC. The following wiki describes cross DC mirroring using the tool
MirrorMaker and how to optimize the throughput for long network latency.

https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html

Thanks,

Jun

On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <apoorvagaurav@gmail.com>wrote:

> Hello All,
>
> We are working on Kafka based event collection system. This needs to gather
> events from across data centers. Lets say all the events will be produced
> in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> between DC1 and DC2 can be around ~80 ms. Number of events should be around
> ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> peak being ~10MB a day. What is the best way to do it.
>
> --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
> and DC2.
> --- Will this problem be solved easily in version .8.0 through broker
> replication by keeping brokers in both DC1 and DC2.
>
> --
> Thanks & Regards,
> Apoorvave
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message