kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Filipiak <Jan.Filip...@trivago.com>
Subject Re: [DISCUSS] KIP-382: MirrorMaker 2.0
Date Wed, 17 Oct 2018 12:09:10 GMT
This is not a performance optimisation. Its a fundamental design choice.

I never really took a look how streams does exactly once. (its a trap 
anyways and you usually can deal with at least once donwstream pretty 
easy). But I am very certain its not gonna get somewhere if offset 
commit and record produce cluster are not the same.

Pretty sure without this _design choice_ you can skip on that exactly 
once already

Best Jan

On 16.10.2018 18:16, Ryanne Dolan wrote:
>  >  But one big obstacle in this was
> always that group coordination happened on the source cluster.
> Jan, thank you for bringing up this issue with legacy MirrorMaker. I
> totally agree with you. This is one of several problems with MirrorMaker
> I intend to solve in MM2, and I already have a design and prototype that
> solves this and related issues. But as you pointed out, this KIP is
> already rather complex, and I want to focus on the core feature set
> rather than performance optimizations for now. If we can agree on what
> MM2 looks like, it will be very easy to agree to improve its performance
> and reliability.
> That said, I look forward to your support on a subsequent KIP that
> addresses consumer coordination and rebalance issues. Stay tuned!
> Ryanne
> On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak <Jan.Filipiak@trivago.com
> <mailto:Jan.Filipiak@trivago.com>> wrote:
>     Hi,
>     Currently MirrorMaker is usually run collocated with the target
>     cluster.
>     This is all nice and good. But one big obstacle in this was
>     always that group coordination happened on the source cluster. So when
>     then network was congested, you sometimes loose group membership and
>     have to rebalance and all this.
>     So one big request from we would be the support of having coordination
>     cluster != source cluster.
>     I would generally say a LAN is better than a WAN for doing group
>     coordinaton and there is no reason we couldn't have a group consuming
>     topics from a different cluster and committing offsets to another
>     one right?
>     Other than that. It feels like the KIP has too much features where many
>     of them are not really wanted and counter productive but I will just
>     wait and see how the discussion goes.
>     Best Jan
>     On 15.10.2018 18:16, Ryanne Dolan wrote:
>      > Hey y'all!
>      >
>      > Please take a look at KIP-382:
>      >
>      >
>     https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
>      >
>      > Thanks for your feedback and support.
>      >
>      > Ryanne
>      >
View raw message