kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: [DISCUSS] KIP-382: MirrorMaker 2.0
Date Tue, 16 Oct 2018 13:14:13 GMT
Ohhhh ok apologies. Interesting!

On Tue, Oct 16, 2018 at 9:06 AM Jan Filipiak <Jan.Filipiak@trivago.com>
wrote:

> Hi Andrew,
>
> thanks for your message, you missed my point.
>
> Mirrormaker collocation with target is for sure correct.
> But then group coordination happens across WAN which is unnecessary.
> And I request to be thought about again.
> I made a PR back then for zk Consumer to allow having 2 zookeeper
> connects. One for group coordination one for broker and topic discovery.
>
> I am requesting this to be added to the kip so that the target cluster
> can become the group coordinator.
>
>
>
> On 16.10.2018 15:04, Andrew Otto wrote:
> >> I would generally say a LAN is better than a WAN for doing group
> >> coordinaton
> >
> > For sure, but a LAN is better than a WAN for producing messages too.  If
> > there is network congestion during network production, messages will be
> > dropped.  With MirrorMaker currently, you can either skip these dropped
> > messages, or have the MirrorMaker processes themselves die on produce
> > failure, which will also cause (a series) of MirrorMaker consumer
> > rebalances.
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 16, 2018 at 7:58 AM Jan Filipiak <Jan.Filipiak@trivago.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Currently MirrorMaker is usually run collocated with the target cluster.
> >> This is all nice and good. But one big obstacle in this was
> >> always that group coordination happened on the source cluster. So when
> >> then network was congested, you sometimes loose group membership and
> >> have to rebalance and all this.
> >>
> >> So one big request from we would be the support of having coordination
> >> cluster != source cluster.
> >>
> >> I would generally say a LAN is better than a WAN for doing group
> >> coordinaton and there is no reason we couldn't have a group consuming
> >> topics from a different cluster and committing offsets to another one
> >> right?
> >>
> >> Other than that. It feels like the KIP has too much features where many
> >> of them are not really wanted and counter productive but I will just
> >> wait and see how the discussion goes.
> >>
> >> Best Jan
> >>
> >>
> >> On 15.10.2018 18:16, Ryanne Dolan wrote:
> >>> Hey y'all!
> >>>
> >>> Please take a look at KIP-382:
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
> >>>
> >>> Thanks for your feedback and support.
> >>>
> >>> Ryanne
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message