Mailing-List: contact users-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@kafka.apache.org
Received-SPF: pass (nike.apache.org: domain of otto@wikimedia.org designates
 209.85.128.43 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Kafka Mirroring setup
From: Andrew Otto <otto@wikimedia.org>
In-Reply-To: 
 <CAOeJiJjnFUPJ=25C7dMY_xhM0sJzbtbPm3wsg2JUP0BEXxqZ0A@mail.gmail.com>
Date: Tue, 20 Aug 2013 13:56:27 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <2A115330-EEE1-4463-B56B-8FA30629C910@wikimedia.org>
References: <183A2B11-3C40-4993-B98F-3E2D4BFBC5E6@gmail.com>
 <CAOeJiJhQrRTmJrjKyOYZb0xHoTdgZe+w518mLGJboL4vScxTxg@mail.gmail.com>
 <177597C7-93B8-4461-A303-E225B7336F78@wikimedia.org>
 <CAOeJiJjnFUPJ=25C7dMY_xhM0sJzbtbPm3wsg2JUP0BEXxqZ0A@mail.gmail.com>
To: users@kafka.apache.org

In our case, our aggregator/analytics cluster is in our main datacenter, =
so there's no risk of the main producers becoming disconnected from it.  =
It seems nicer to have a dedicated aggregator cluster, that only only =
gets its data via MirrorMaker (Option A), but in our case this isn't =
necessary.

The aggregator cluster could use MirrorMaker to consume from remote =
datacenters, but still have regular local producers send it data =
directly (Option B).


On Aug 20, 2013, at 1:47 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> We do something like A (though I'm not sure I understand B):
> http://kafka.apache.org/documentation.html#datacenters
>=20
> Essentially what we wanted was that each datacenter stood alone so =
that we
> would not lose data if the datacenters became disconnected. Network
> partitions within our data centers are extremely rare but between
> datacenters relatively common.
>=20
> -Jay
>=20
>=20
>=20
>=20
> On Tue, Aug 20, 2013 at 10:35 AM, Andrew Otto <otto@wikimedia.org> =
wrote:
>=20
>> Hi all!
>>=20
>> Wikimedia is investigating how best to set up Broker clusters in =
multiple
>> data centers.  Our main analytics Broker cluster is currently in our =
main
>> datacenter.  It is possible for all of the main DC's frontend =
producers to
>> produce directly to our analytics cluster, but we're not sure if this =
is a
>> best practice.  So!  What does LinkedIn recommend?
>>=20
>> Option A: N + 1 clusters.
>> - N production Broker Clusters (1 for each DC).
>> - +1 aggregator/analytics Broker cluster that mirrors all of the
>> production clusters.
>>=20
>> - Option B: N total Broker clusters.
>> - Frontend producers in the main cluster produce directly to the
>> aggregator/analytics cluster.
>> - Other DC's clusters are mirrored to the aggregator/analytics =
cluster.
>>=20
>> Thanks!
>> -Andrew