kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Becket Qin <becket....@gmail.com>
Subject Re: [DISCUSS] KIP-382: MirrorMaker 2.0
Date Thu, 13 Dec 2018 06:07:11 GMT
Hi Ryanne,

Thanks for the KIP. In general, I think the proposal makes a lot of sense
to me. Unifying Kafka clusters in different locations has been a very
important scenario. Right now open source community does not have a
standard solution to that.

A few comments/questions are following:

1. Only relying on the topic naming seems a little fragile. It seems
difficult to prevent conflict names. One workaround might be adding a
config to the broker, preventing topics with character such as "_" or "."
from being created, unless it is from some specific client such as MM2
admin clients.

2. For ACLs, is it possible that the local topic and remote topic have
different ACLs? Will the connector do the sanity check?

3. For the checkpoint. Because the consumer may commit at arbitrary offset.
Does the connector need to keep a mapping between each source offset to
destination offset? If so how would that be done?

Thanks,

Jiangjie (Becket) Qin

On Thu, Dec 13, 2018 at 8:23 AM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> > Consuming system restarts and restates from compacted topics, using
> *.account_state
>
> Michael, I think I understand your concern. With MM2's remote topic
> concept, it's the consumer's responsibility to aggregate records across
> .*account_state. MM2 will not merge these for you (by design). This
> complicates your consumer that expects a single topic to be compacted
> across all clusters, since log-compaction does not span topics.
>
> Notice that MM2 isn't really breaking anything here. The problem is that
> you are using MirrorMaker to aggregate records from multiple clusters into
> a single topic, which MM2 will not do by default.
>
> It's easy to accomplish the same thing with MM2 though. You have a couple
> options:
>
> 1) turn off topic renaming or use "legacy mode". MM2 won't get in your way,
> but it won't help much for DR, failover/failback etc. You're on your own :)
>
> 2) Aggregate your topic downstream of MM2. Turn .*account_state into
> account_state.aggregate, which itself is log-compacted. KTables make this
> trivial!
>
> With (2) you still get the nice DR semantics. The KTable will represent the
> latest account states aggregated across all clusters. If your producers
> need to migrate/failover/failback, the KTable in each region will be
> eventually consistent, and downstream consumers won't notice. I'd recommend
> blacklisting .*aggregate in MM2 so these don't get replicated themselves.
>
> TBH, I thought about including this as a feature in MM2, s.t. you would get
> a foo.aggregate topic automatically. Personally, I think Streams is better
> suited than Connect for this, but I won't rule it out for a future KIP.
>
> Thanks again!
> Ryanne
>
>
>
> On Wed, Dec 12, 2018 at 2:49 PM Michael Pearce <Michael.Pearce@ig.com>
> wrote:
>
> > Ok so you have your transaction system running master master.
> >
> > You have three dcs, regions where-ever.
> >
> > Accounts are distributed across area based on an number of factors
> >
> > Account A + B are in Ireland
> > Account C + D are in Germany
> > Account E are in UK
> >
> > Lets prefix call kafka cluster in Ireland Z, Germany Y and UK X
> >
> > So now I have topics:
> >
> > Z.account_state (holding state for accounts A  + B)
> > Y.account_state (holding state for accounts C + D)
> > X.account_state (holding state for account E)
> >
> > To get a full view of all accounts a consumer would consume
> *.account_state
> > Great!
> >
> > Now image we take a planned or unplanned outage in the UK.
> > We move/shift the transactional processing of account E to Ireland.
> >
> > Now we end up with
> > Z.account_state (holding state for accounts A  + B + current state for
> > account E)
> > Y.account_state (holding state for accounts C + D)
> > X.account_state (holding outdated state for account E)
> >
> >
> > Consuming system restarts and restates from compacted topics, using
> > *.account_state
> >
> > It will get both current state and also outdated state for E, which state
> > would it use? And this gets worse as it scales and you move the
> processing
> > of accounts around over time.
> >
> >
> > Likewise the issue is the same without compacted state.
> >
> > Imagine order of state change is important by account globally, how do
> you
> > re-order the state changes? As order guarantee can be given within a
> single
> > topic partition only. You only care for order by the account (not for
> > offets being same, we simply care that updates are in order and latest
> > state is at head on restart we ignore offsets). So it doesn’t matter if
> in
> > Z cluster you see update A1, A2, B1, A3, B2, E1, E2 and in cluster Y you
> > see B1,B2, A1,E1, A2, A3, E2 as the ordering by of the updates account is
> > preserved.
> >
> > With the topic solution your suggesting we would have no way true way of
> > replaying and re-constituting the order between X.account_state and
> > Z.account_state topics in the case of E, as messages will be in different
> > topics and partitions.
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Ryanne Dolan <ryannedolan@gmail.com>
> > Sent: Wednesday, December 12, 2018 4:37 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-382: MirrorMaker 2.0
> >
> > > You haven’t described how one would handle the ordering issues and also
> > the compaction issues where transactional processing is master-master in
> > regions, where the processing is sticky to region but of failure or
> > planned, processing of certain accounts move regions.
> >
> > Michael, a couple points:
> >
> > - With "remote topics", the order of events is consistent between
> clusters.
> > Cluster A's "topic1" is the same records in the same order as cluster B's
> > "A.topic1". The plan is to enable exactly-once semantics within MM2 so
> > there aren't additional dupes either (though I believe this will require
> > support within Connect and a separate KIP).
> >
> > - A consumer that is only interested in events produced in a particular
> > region can migrate to a cluster in a different region by updating it's
> > subscription accordingly. For example, a consumer in us-west processing
> > events local to us-west would consume topics like "topic1" (a normal
> > topic). If you migrate this consumer to us-east, it would need to
> subscribe
> > to "us-west.topic1" instead. It's clear from the naming convention that
> > "us-west.topic1" is a replicated topic with records originating from a
> > remote cluster.
> >
> > I'm not sure I understand your concern w.r.t compacted topics and state.
> > Can you elaborate?
> >
> > Ryanne
> >
> >
> > On Wed, Dec 12, 2018 at 8:41 AM Michael Pearce <Michael.Pearce@ig.com>
> > wrote:
> >
> > > Ryanne,
> > >
> > > You haven’t described how one would handle the ordering issues and also
> > > the compaction issues where transactional processing is master-master
> in
> > > regions, where the processing is sticky to region but of failure or
> > > planned, processing of certain accounts move regions.
> > >
> > > Also I ask that you keep compatibility of the handler api interface in
> MM
> > > into MM2.
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Ryanne Dolan <ryannedolan@gmail.com>
> > > Sent: Wednesday, December 12, 2018 6:41 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-382: MirrorMaker 2.0
> > >
> > > > One based on hops using headers, and another based on topic naming.
> > >
> > > Michael, this was also suggested by Alex Mironov. Other replication
> > engines
> > > use headers as you describe, but there are several issues with this
> > > approach:
> > >
> > > - The Connect framework provides Transformations that could be used for
> > > this purpose, so MM2 doesn't necessarily need to address this feature
> > > directly. For example, a Transformation could tag each record with its
> > > source cluster alias or could decrement a hop/TTL value if you like.
> > >
> > > - We want MM2 to benefit from "shallow iteration", meaning that e.g.
> > > compressed message sets should pass through MM2 without being
> > decompressed
> > > and decomposed. In many cases, this can result in significant
> performance
> > > gains. But since these message sets are not decomposed into individual
> > > messages and so on, we cannot reliably tag each message that passes
> > > through.
> > >
> > > - Fundamentally, I believe it is a bad idea to treat topics on
> different
> > > clusters as if they are the same topic. If two clusters have topics of
> > the
> > > same name, they are still inherently different topics, as they exist on
> > > different clusters, with potentially different records and certainly
> > > different offsets. Moreover, there is no robust way to keep such topics
> > > consistent and in sync.
> > >
> > > - Some applications are interested in events from a particular data
> > > center/region/cluster, while other applications will want to process
> > events
> > > regardless of where they are produced. Indeed, a common use-case for
> > > building out multiple clusters in the first place is to support this
> sort
> > > of geolocation-aware processing and aggregation. It sounds like your
> org
> > > attempts to make topics the same everywhere, which is undesirable in
> many
> > > cases.
> > >
> > > - Advanced operations such as automatic failover and failback rely on
> the
> > > ability to segregate records based on their cluster of origin, while
> > > preserving order within each topic-partition. This is extremely
> difficult
> > > if your app's producers and MM's producers "cross streams" by writing
> > into
> > > the same topic. (Your mention of "ensure inflight processing at the old
> > > region ceased" etc is hand-holding symptomatic of this problem.)
> > >
> > > - With a consistent naming convention in place (along with
> checkpointing,
> > > heartbeats and the other "best practices" mentioned in the KIP), we can
> > > build tooling that understands multi-cluster environments. For example,
> > the
> > > KIP describes a utility for translating consumer offsets between
> > clusters.
> > > This will enable operators to migrate consumer groups between clusters
> > > without knowing anything about the topics involved.
> > >
> > > That all said, I am sure some organizations will want to apply their
> own
> > > conventions, and I don't believe MM2 should get in the way of that.
> > >
> > > Thanks again!
> > > Ryanne
> > >
> > > On Tue, Dec 11, 2018 at 7:20 PM michael.andre.pearce
> > > <michael.andre.pearce@me.com.invalid> wrote:
> > >
> > > > Another benefit of using hops vs topic naming (also how we currently
> do
> > > > master master in my org)
> > > > You have a transactional processing app that's multi regioned. So for
> > > sake
> > > > of discussion all A and B accounts process normally in ireland region
> > > all C
> > > > and D in germany region and all E in Uk region. Increase regions to
> > > across
> > > > the globe almost one per financial centre where close regions might
> > mesh
> > > > together one to one and continental regions form some backbones
> trunks.
> > > > If uk region goes down planned or unplanned we move the transactional
> > > > routing and processing to germany region. During this flip over we
> > ensure
> > > > inflight processing at the old region ceased and restate account
> states
> > > > from topics before further processing thus ensuring no out of order
> > > message
> > > > production by account.
> > > > The with using region broker or some other topic prefixes this will
> > mean
> > > > now i have a topic for uk region with some E account data and after
> > > process
> > > > movement will end up with E accounts in the topic for germany region.
> > Now
> > > > ordering is lost. Worst still if compacted topic i will have two
> states
> > > as
> > > > state would be in two topics.
> > > > With using hops and single named topic (as we do already in our own
> > > custom
> > > > handlers which do hop logic with todays mirrormakers) we can avoid
> this
> > > > issue entirely as ordering by account is preserved as is all in one
> > topic
> > > > still and also when using compacted we have only one state.
> > > >
> > > > Before you say why not name topics a. b. I use that to simplify the
> > case
> > > > to be able describe it. Accounts are autogenrated e.g. A16E45T1 and
> > > C43F4SA
> > > > could be processing in germany region currently and C43F2SA could be
> in
> > > uk
> > > > region currently.
> > > >
> > > >
> > > >
> > > > Sent from my Samsung Galaxy smartphone.
> > > > -------- Original message --------From: Andrew Otto <
> > otto@wikimedia.org>
> > > > Date: 11/12/2018  14:28  (GMT+00:00) To: dev@kafka.apache.org
> Subject:
> > > > Re: [DISCUSS] KIP-382: MirrorMaker 2.0
> > > > Wikimedia currently implements 'master <-> master' replication by
> > > manually
> > > > prefixing topics with datacenter names, and then configuring
> > MirrorMaker
> > > to
> > > > only replicate topics that begin with a DC name to another.
> > > >
> > > > While having topics named with topological details is manageable, I
> > > > wouldn't say it is desirable.  It pushes knowledge of the replication
> > > > topology up to clients.  Even if MirrorMaker was the one doing the
> > topic
> > > > prefixing, downstream consumers of a group of replicated topics are
> > still
> > > > going to have to know to subscribe to the correctly prefixed topics.
> > > >
> > > > If possible I'd much prefer header + hops based replication rather
> than
> > > > lots of renamed topics.  But either way, this KIP would be
> tremendously
> > > > useful to us so I support it all the way! :)
> > > >
> > > > On Tue, Dec 11, 2018 at 5:32 AM Michael Pearce <
> Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > So this is indeed what using headers with hops avoids is creating
> > lots
> > > > and
> > > > > lots of topics __, so you can have more complex topology setups.
> > > > >
> > > > > I ask why not support having two ways of setting up and closing the
> > > door?
> > > > >
> > > > > One based on hops using headers, and another based on topic naming.
> > > After
> > > > > all flexibility is what we want its for end users how to use right?
> > > > >
> > > > >
> > > > >
> > > > > On 12/7/18, 8:19 PM, "Ryanne Dolan" <ryannedolan@gmail.com>
> wrote:
> > > > >
> > > > >     Michael, thanks for the comments!
> > > > >
> > > > >     >  would like to see support for this to be done by hops, as
> well
> > > > [...]
> > > > >     This then allows ring (hops = number of brokers in the ring),
> > mesh
> > > > > (every
> > > > >     cluster interconnected so hop=1), or even a tree (more fine
> > grained
> > > > > setup)
> > > > >     cluster topology.
> > > > >
> > > > >     That's a good idea, though we can do this at the topic level
> > > without
> > > > >     tagging individual records. A max.hop of 1 would mean
> "A.topic1"
> > is
> > > > >     allowed, but not "B.A.topic1". I think the default behavior
> would
> > > > need
> > > > > to
> > > > >     be max.hops = 1 to avoid unexpectedly creating a bunch of
> > > D.C.B.A...
> > > > > topics
> > > > >     when you create a fully-connected mesh topology.
> > > > >
> > > > >     Looking ahead a bit, I can imagine an external tool computing
> the
> > > > > spanning
> > > > >     tree of topics among a set of clusters based on inter-cluster
> > > > > replication
> > > > >     lag, and setting up MM2 accordingly. But that's probably
> outside
> > > the
> > > > > scope
> > > > >     of this KIP :)
> > > > >
> > > > >     >  ...standalone MirrorMaker connector...
> > > > >     >     ./bin/kafka-mirror-maker-2.sh --consumer
> > consumer.properties
> > > > >     --producer producer.properties
> > > > >
> > > > >     Eventually, I'd like MM2 to completely replace legacy MM,
> > including
> > > > the
> > > > >     ./bin/kafka-mirror-maker.sh script. In the meantime, it's a
> good
> > > idea
> > > > > to
> > > > >     include a standalone driver. Something like
> > > > >     ./bin/connect-mirror-maker-standalone.sh with the same
> high-level
> > > > >     configuration file. I'll do that, thanks.
> > > > >
> > > > >     > I see no section on providing support for mirror maker
> > Handlers,
> > > > > today
> > > > >     people can add handlers to have a little extra custom logic if
> > > > needed,
> > > > > and
> > > > >     the handler api is public today so should be supported going
> > > forwards
> > > > > so
> > > > >     people are not on mass re-writing these.
> > > > >
> > > > >     Great point. Connect offers single-message transformations and
> > > > > converters
> > > > >     for this purpose, but I agree that we should honor the existing
> > API
> > > > if
> > > > >     possible. This might be as easy as providing an adapter class
> > > between
> > > > >     connect's Transformation and mirror-maker's Handler. Maybe
> file a
> > > > Jira
> > > > >     ticket to track this?
> > > > >
> > > > >     Really appreciate your feedback!
> > > > >
> > > > >     Ryanne
> > > > >
> > > > >
> > > > >     On Thu, Dec 6, 2018 at 7:03 PM Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >     > Re hops to stop the cycle and to allow a range of multi
> cluster
> > > > >     > topologies, see
> > > https://www.rabbitmq.com/federated-exchanges.html
> > > > > where
> > > > >     > very similar was done in rabbit.
> > > > >     >
> > > > >     >
> > > > >     >
> > > > >     > On 12/7/18, 12:47 AM, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >     >
> > > > >     >     Nice proposal.
> > > > >     >
> > > > >     >     Some comments.
> > > > >     >
> > > > >     >
> > > > >     >     On the section around cycle detection.
> > > > >     >
> > > > >     >     I would like to see support for this to be done by hops,
> as
> > > > well
> > > > > e.g.
> > > > >     > using approach is to use a header for the number of hops, as
> > the
> > > > mm2
> > > > >     > replicates it increases the hop count and you can make the
> mm2
> > > > > configurable
> > > > >     > to only produce messages onwards where hops are less than x.
> > > > >     >     This then allows ring (hops = number of brokers in the
> > ring),
> > > > > mesh
> > > > >     > (every cluster interconnected so hop=1), or even a tree (more
> > > fine
> > > > > grained
> > > > >     > setup) cluster topology.
> > > > >     >     FYI we do this currently with the current mirror maker,
> > > using a
> > > > > custom
> > > > >     > handler.
> > > > >     >
> > > > >     >
> > > > >     >     On the section around running a standalone MirrorMaker
> > > > connector
> > > > >     >
> > > > >     >     I would suggest making this as easy to run as the
> > > mirrormakers
> > > > > are
> > > > >     > today, with a simple single sh script.
> > > > >     >     I assume this is what is proposed in section "Running
> > > > > MirrorMaker in
> > > > >     > legacy mode" but I would even do this before MM would be
> > removed,
> > > > > with a -2
> > > > >     > varient.
> > > > >     >     e.g.
> > > > >     >     ./bin/kafka-mirror-maker-2.sh --consumer
> > consumer.properties
> > > > >     > --producer producer.properties
> > > > >     >
> > > > >     >     Lastly
> > > > >     >
> > > > >     >     I see no section on providing support for mirror maker
> > > > Handlers,
> > > > > today
> > > > >     > people can add handlers to have a little extra custom logic
> if
> > > > > needed, and
> > > > >     > the handler api is public today so should be supported going
> > > > > forwards so
> > > > >     > people are not on mass re-writing these.
> > > > >     >
> > > > >     >     On 12/5/18, 5:36 PM, "Ryanne Dolan" <
> ryannedolan@gmail.com
> > >
> > > > > wrote:
> > > > >     >
> > > > >     >         Sönke,
> > > > >     >
> > > > >     >         > The only thing that I could come up with is the
> > > > limitation
> > > > > to a
> > > > >     > single
> > > > >     >         offset commit interval
> > > > >     >
> > > > >     >         Yes, and other internal properties, e.g. those used
> by
> > > the
> > > > > internal
> > > > >     >         consumers and producers, which, granted, probably are
> > not
> > > > > often
> > > > >     > changed
> > > > >     >         from their defaults, but that apply to Connectors
> > across
> > > > the
> > > > >     > entire cluster.
> > > > >     >
> > > > >     >         Ryanne
> > > > >     >
> > > > >     >         On Wed, Dec 5, 2018 at 3:21 AM Sönke Liebau
> > > > >     >         <soenke.liebau@opencore.com.invalid> wrote:
> > > > >     >
> > > > >     >         > Hi Ryanne,
> > > > >     >         >
> > > > >     >         > when you say "Currently worker configs apply across
> > the
> > > > > entire
> > > > >     > cluster,
> > > > >     >         > which is limiting even for use-cases involving a
> > single
> > > > > Kafka
> > > > >     > cluster.",
> > > > >     >         > may I ask you to elaborate on those limitations a
> > > little?
> > > > >     >         > The only thing that I could come up with is the
> > > > limitation
> > > > > to a
> > > > >     > single
> > > > >     >         > offset commit interval value for all running
> > > connectors.
> > > > >     >         > Maybe also the limitation to shared config
> > providers..
> > > > >     >         >
> > > > >     >         > But you sound like you had painful experiences with
> > > this
> > > > > before,
> > > > >     > maybe
> > > > >     >         > you'd like to share the burden :)
> > > > >     >         >
> > > > >     >         > Best regards,
> > > > >     >         > Sönke
> > > > >     >         >
> > > > >     >         > On Wed, Dec 5, 2018 at 5:15 AM Ryanne Dolan <
> > > > >     > ryannedolan@gmail.com> wrote:
> > > > >     >         >
> > > > >     >         > > Sönke,
> > > > >     >         > >
> > > > >     >         > > I think so long as we can keep the differences
> at a
> > > > very
> > > > > high
> > > > >     > level (i.e.
> > > > >     >         > > the "control plane"), there is little downside to
> > MM2
> > > > and
> > > > >     > Connect
> > > > >     >         > > coexisting. I do expect them to converge to some
> > > > extent,
> > > > > with
> > > > >     > features
> > > > >     >         > from
> > > > >     >         > > MM2 being pulled into Connect whenever this is
> > > possible
> > > > >     > without breaking
> > > > >     >         > > things.
> > > > >     >         > >
> > > > >     >         > > I could definitely see your idea re hierarchies
> or
> > > > > groups of
> > > > >     > connectors
> > > > >     >         > > being useful outside MM2. Currently "worker
> > configs"
> > > > > apply
> > > > >     > across the
> > > > >     >         > > entire cluster, which is limiting even for
> > use-cases
> > > > > involving
> > > > >     > a single
> > > > >     >         > > Kafka cluster. If Connect supported multiple
> > workers
> > > in
> > > > > the
> > > > >     > same cluster,
> > > > >     >         > > it would start to look a lot like a MM2 cluster.
> > > > >     >         > >
> > > > >     >         > > Ryanne
> > > > >     >         > >
> > > > >     >         > > On Tue, Dec 4, 2018 at 3:26 PM Sönke Liebau
> > > > >     >         > > <soenke.liebau@opencore.com.invalid> wrote:
> > > > >     >         > >
> > > > >     >         > > > Hi Ryanne,
> > > > >     >         > > >
> > > > >     >         > > > thanks for your response!
> > > > >     >         > > >
> > > > >     >         > > > It seems like you have already done a lot of
> > > > > investigation
> > > > >     > into the
> > > > >     >         > > > existing code and the solution design and all
> of
> > > what
> > > > > you
> > > > >     > write makes
> > > > >     >         > > sense
> > > > >     >         > > > to me. Would it potentially be worth adding
> this
> > to
> > > > > the KIP,
> > > > >     > now that
> > > > >     >         > you
> > > > >     >         > > > had to write it up because of me anyway?
> > > > >     >         > > >
> > > > >     >         > > > However, I am afraid that I am still not
> entirely
> > > > > convinced
> > > > >     > of the
> > > > >     >         > > > fundamental benefit this provides over an
> > extended
> > > > > Connect
> > > > >     > that has the
> > > > >     >         > > > following functionality:
> > > > >     >         > > > - allow for organizing connectors into a
> > > hierarchical
> > > > >     > structure -
> > > > >     >         > > > "clusters/us-west/..."
> > > > >     >         > > > - allow defining external Kafka clusters to be
> > used
> > > > by
> > > > >     > Source and Sink
> > > > >     >         > > > connectors instead of the local cluster
> > > > >     >         > > >
> > > > >     >         > > > Personally I think both of these features are
> > > useful
> > > > >     > additions to
> > > > >     >         > > Connect,
> > > > >     >         > > > I'll address both separately below.
> > > > >     >         > > >
> > > > >     >         > > > Allowing to structure connectors in a hierarchy
> > > > >     >         > > > Organizing running connectors will grow more
> > > > important
> > > > > as
> > > > >     > corporate
> > > > >     >         > > > customers adapt Connect and installations grow
> in
> > > > size.
> > > > >     > Additionally
> > > > >     >         > this
> > > > >     >         > > > could be useful for ACLs in case they are ever
> > > added
> > > > to
> > > > >     > Connect, as you
> > > > >     >         > > > could allow specific users access only to
> > specific
> > > > >     > namespaces (and
> > > > >     >         > until
> > > > >     >         > > > ACLs are added it would facilitate using a
> > reverse
> > > > > proxy for
> > > > >     > the same
> > > > >     >         > > > effect).
> > > > >     >         > > >
> > > > >     >         > > > Allow accessing multiple external clusters
> > > > >     >         > > > The reasoning for this feature is pretty much
> the
> > > > same
> > > > > as
> > > > >     > for a central
> > > > >     >         > > > Mirror Maker cluster, if a company has multiple
> > > > > clusters for
> > > > >     > whatever
> > > > >     >         > > > reason but wants to have ingest centralized in
> > one
> > > > > system
> > > > >     > aka one
> > > > >     >         > Connect
> > > > >     >         > > > cluster they would need the ability to read
> from
> > > and
> > > > > write
> > > > >     > to an
> > > > >     >         > > arbitrary
> > > > >     >         > > > number of Kafka clusters.
> > > > >     >         > > > I haven't really looked at the code, just poked
> > > > around
> > > > > a
> > > > >     > couple of
> > > > >     >         > > minutes,
> > > > >     >         > > > but it appears like this could be done with
> > fairly
> > > > low
> > > > >     > effort. My
> > > > >     >         > general
> > > > >     >         > > > idea would be to leave the existing
> configuration
> > > > > options
> > > > >     > untouched -
> > > > >     >         > > > Connect will always need a "primary" cluster
> that
> > > is
> > > > > used
> > > > >     > for storage
> > > > >     >         > of
> > > > >     >         > > > internal data (config, offsets, status) there
> is
> > no
> > > > > need to
> > > > >     > break
> > > > >     >         > > existing
> > > > >     >         > > > configs. But additionally allow adding named
> > extra
> > > > > clusters
> > > > >     > by
> > > > >     >         > specifying
> > > > >     >         > > > options like
> > > > >     >         > > >   external.sales_cluster.bootstrap_servers=...
> > > > >     >         > > >
> >  external.sales_cluster.ssl.keystore.location=...
> > > > >     >         > > >
> >  external.marketing_cluster.bootstrap_servers=...
> > > > >     >         > > >
> > > > >     >         > > > The code for status, offset and config storage
> is
> > > > > mostly
> > > > >     > isolated in
> > > > >     >         > the
> > > > >     >         > > > Kafka[Offset|Status|Config]BackingStore classes
> > and
> > > > > could
> > > > >     > remain pretty
> > > > >     >         > > > much unchanged.
> > > > >     >         > > >
> > > > >     >         > > > Producer and consumer creation for Tasks is
> done
> > in
> > > > the
> > > > >     > Worker as of
> > > > >     >         > > > KAFKA-7551 and is isolated in two functions. We
> > > could
> > > > > add a
> > > > >     > two more
> > > > >     >         > > > functions with an extra argument for the
> external
> > > > > cluster
> > > > >     > name to be
> > > > >     >         > used
> > > > >     >         > > > and return fitting consumers/producers.
> > > > >     >         > > > The source and sink config would then simply
> gain
> > > an
> > > > >     > optional setting
> > > > >     >         > to
> > > > >     >         > > > specify the cluster name.
> > > > >     >         > > >
> > > > >     >         > > > I am very sure that I am missing a few large
> > issues
> > > > > with
> > > > >     > these ideas,
> > > > >     >         > I'm
> > > > >     >         > > > mostly back-of-the-napkin designing here, but
> it
> > > > might
> > > > > be
> > > > >     > worth a
> > > > >     >         > second
> > > > >     >         > > > look.
> > > > >     >         > > >
> > > > >     >         > > > Once we decide to diverge into two clusters:
> > > > > MirrorMaker and
> > > > >     > Connect, I
> > > > >     >         > > > think realistically the chance of those two
> ever
> > > > being
> > > > >     > merged again
> > > > >     >         > > because
> > > > >     >         > > > they grow back together is practically zero -
> > hence
> > > > my
> > > > >     > hesitation.
> > > > >     >         > > >
> > > > >     >         > > > ----
> > > > >     >         > > >
> > > > >     >         > > > All of that being said, I am absolutely happy
> to
> > > > agree
> > > > > to
> > > > >     > disagree, I
> > > > >     >         > > think
> > > > >     >         > > > to a certain extent this is down to a question
> of
> > > > > personal
> > > > >     >         > > > style/preference. And as this is your baby and
> > you
> > > > > have put
> > > > >     > a lot more
> > > > >     >         > > > effort and thought into it than I ever will
> I'll
> > > shut
> > > > > up now
> > > > >     > :)
> > > > >     >         > > >
> > > > >     >         > > > Again, thanks for all your good work!
> > > > >     >         > > >
> > > > >     >         > > > Best regards,
> > > > >     >         > > > Sönke
> > > > >     >         > > >
> > > > >     >         > > > On Fri, Nov 30, 2018 at 9:00 PM Ryanne Dolan <
> > > > >     > ryannedolan@gmail.com>
> > > > >     >         > > > wrote:
> > > > >     >         > > >
> > > > >     >         > > > > Thanks Sönke.
> > > > >     >         > > > >
> > > > >     >         > > > > > it just feels to me like an awful lot of
> > > Connect
> > > > >     > functionality
> > > > >     >         > would
> > > > >     >         > > > need
> > > > >     >         > > > > to be reimplemented or at least wrapped
> > > > >     >         > > > >
> > > > >     >         > > > > Connect currently has two drivers,
> > > > > ConnectDistributed and
> > > > >     >         > > > > ConnectStandalone. Both set up a Herder,
> which
> > > > > manages
> > > > >     > Workers. I've
> > > > >     >         > > > > implemented a third driver which sets up
> > multiple
> > > > > Herders,
> > > > >     > one for
> > > > >     >         > each
> > > > >     >         > > > > Kafka cluster as specified in a config file.
> > From
> > > > the
> > > > >     > Herder level
> > > > >     >         > > down,
> > > > >     >         > > > > nothing is changed or duplicated -- it's just
> > > > > Connect.
> > > > >     >         > > > >
> > > > >     >         > > > > For the REST API, Connect wraps a Herder in a
> > > > > RestServer
> > > > >     > class, which
> > > > >     >         > > > > creates a Jetty server with a few JAX-RS
> > > resources.
> > > > > One of
> > > > >     > these
> > > > >     >         > > > resources
> > > > >     >         > > > > is ConnectorsResource, which is the real meat
> > of
> > > > the
> > > > > REST
> > > > >     > API,
> > > > >     >         > enabling
> > > > >     >         > > > > start, stop, creation, deletion, and
> > > configuration
> > > > of
> > > > >     > Connectors.
> > > > >     >         > > > >
> > > > >     >         > > > > I've added MirrorRestServer, which wraps a
> set
> > of
> > > > > Herders
> > > > >     > instead of
> > > > >     >         > > one.
> > > > >     >         > > > > The server exposes a single resource,
> > > > > ClustersResource,
> > > > >     > which is
> > > > >     >         > only a
> > > > >     >         > > > few
> > > > >     >         > > > > lines of code:
> > > > >     >         > > > >
> > > > >     >         > > > > @GET
> > > > >     >         > > > > @Path("/")
> > > > >     >         > > > > public Collection<String> listClusters() {
> > > > >     >         > > > >   return clusters.keySet();
> > > > >     >         > > > > }
> > > > >     >         > > > >
> > > > >     >         > > > > @Path("/{cluster}")
> > > > >     >         > > > > public ConnectorsResource
> > > > >     >         > getConnectorsForCluster(@PathParam("cluster")
> > > > >     >         > > > > cluster) {
> > > > >     >         > > > >   return new
> > > > > ConnectorsResource(clusters.get(cluster));
> > > > >     >         > > > > }
> > > > >     >         > > > >
> > > > >     >         > > > > (simplified a bit and subject to change)
> > > > >     >         > > > >
> > > > >     >         > > > > The ClustersResource defers to the existing
> > > > >     > ConnectorsResource, which
> > > > >     >         > > > again
> > > > >     >         > > > > is most of the Connect API. With this in
> > place, I
> > > > > can make
> > > > >     > requests
> > > > >     >         > > like:
> > > > >     >         > > > >
> > > > >     >         > > > > GET /clusters
> > > > >     >         > > > >
> > > > >     >         > > > > GET /clusters/us-west/connectors
> > > > >     >         > > > >
> > > > >     >         > > > > PUT
> /clusters/us-west/connectors/us-east/config
> > > > >     >         > > > > { "topics" : "topic1" }
> > > > >     >         > > > >
> > > > >     >         > > > > etc.
> > > > >     >         > > > >
> > > > >     >         > > > > So on the whole, very little code is involved
> > in
> > > > >     > implementing
> > > > >     >         > > > "MirrorMaker
> > > > >     >         > > > > clusters". I won't rule out adding additional
> > > > > features on
> > > > >     > top of this
> > > > >     >         > > > basic
> > > > >     >         > > > > API, but nothing should require
> re-implementing
> > > > what
> > > > > is
> > > > >     > already in
> > > > >     >         > > > Connect.
> > > > >     >         > > > >
> > > > >     >         > > > > > Wouldn't it be a viable alternative to look
> > > into
> > > > >     > extending Connect
> > > > >     >         > > > itself
> > > > >     >         > > > >
> > > > >     >         > > > > Maybe Connect will evolve to the point where
> > > > Connect
> > > > >     > clusters and
> > > > >     >         > > > > MirrorMaker clusters are indistinguishable,
> > but I
> > > > > think
> > > > >     > this is
> > > > >     >         > > unlikely,
> > > > >     >         > > > > since really no use-case outside replication
> > > would
> > > > > benefit
> > > > >     > from the
> > > > >     >         > > added
> > > > >     >         > > > > complexity. Moreover, I think support for
> > > multiple
> > > > > Kafka
> > > > >     > clusters
> > > > >     >         > would
> > > > >     >         > > > be
> > > > >     >         > > > > hard to add without significant changes to
> the
> > > > > existing
> > > > >     > APIs and
> > > > >     >         > > configs,
> > > > >     >         > > > > which all assume a single Kafka cluster. I
> > think
> > > > >     > Connect-as-a-Service
> > > > >     >         > > and
> > > > >     >         > > > > Replication-as-a-Service are sufficiently
> > > different
> > > > >     > use-cases that we
> > > > >     >         > > > > should expect the APIs and configuration
> files
> > to
> > > > be
> > > > > at
> > > > >     > least
> > > > >     >         > slightly
> > > > >     >         > > > > different, even if both use the same
> framework
> > > > > underneath.
> > > > >     > That
> > > > >     >         > said, I
> > > > >     >         > > > do
> > > > >     >         > > > > plan to contribute a few improvements to the
> > > > Connect
> > > > >     > framework in
> > > > >     >         > > support
> > > > >     >         > > > > of MM2 -- just nothing within the scope of
> the
> > > > > current KIP.
> > > > >     >         > > > >
> > > > >     >         > > > > Thanks again!
> > > > >     >         > > > > Ryanne
> > > > >     >         > > > >
> > > > >     >         > > > >
> > > > >     >         > > > > On Fri, Nov 30, 2018 at 3:47 AM Sönke Liebau
> > > > >     >         > > > > <soenke.liebau@opencore.com.invalid> wrote:
> > > > >     >         > > > >
> > > > >     >         > > > > > Hi Ryanne,
> > > > >     >         > > > > >
> > > > >     >         > > > > > thanks. I missed the remote to remote
> > > replication
> > > > >     > scenario in my
> > > > >     >         > > train
> > > > >     >         > > > of
> > > > >     >         > > > > > thought, you are right.
> > > > >     >         > > > > >
> > > > >     >         > > > > > That being said I have to admit that I am
> not
> > > yet
> > > > > fully
> > > > >     > on board
> > > > >     >         > with
> > > > >     >         > > > the
> > > > >     >         > > > > > concept, sorry. But I might just be
> > > > > misunderstanding
> > > > >     > what your
> > > > >     >         > > > intention
> > > > >     >         > > > > > is. Let me try and explain what I think it
> is
> > > you
> > > > > are
> > > > >     > trying to do
> > > > >     >         > > and
> > > > >     >         > > > > why
> > > > >     >         > > > > > I am on the fence about that and take it
> from
> > > > > there.
> > > > >     >         > > > > >
> > > > >     >         > > > > > You want to create an extra mirrormaker
> > driver
> > > > > class
> > > > >     > which will
> > > > >     >         > take
> > > > >     >         > > > > > multiple clusters as configuration options.
> > > Based
> > > > > on
> > > > >     > these clusters
> > > > >     >         > > it
> > > > >     >         > > > > will
> > > > >     >         > > > > > then reuse the connect workers and create
> as
> > > many
> > > > > as
> > > > >     > necessary to
> > > > >     >         > be
> > > > >     >         > > > able
> > > > >     >         > > > > > to replicate to/from each of those
> configured
> > > > > clusters.
> > > > >     > It will
> > > > >     >         > then
> > > > >     >         > > > > > expose a rest api (since you stated subset
> of
> > > > > Connect
> > > > >     > rest api I
> > > > >     >         > > assume
> > > > >     >         > > > > it
> > > > >     >         > > > > > will be a new / own one?)  that allows
> users
> > to
> > > > > send
> > > > >     > requests like
> > > > >     >         > > > > > "replicate topic a from cluster 1 to
> cluster
> > 1"
> > > > and
> > > > >     > start a
> > > > >     >         > connector
> > > > >     >         > > > on
> > > > >     >         > > > > > the relevant worker that can offer this
> > > "route".
> > > > >     >         > > > > > This can be extended to a cluster by
> starting
> > > > > mirror
> > > > >     > maker drivers
> > > > >     >         > on
> > > > >     >         > > > > other
> > > > >     >         > > > > > nodes with the same config and it would
> offer
> > > all
> > > > > the
> > > > >     > connect
> > > > >     >         > > features
> > > > >     >         > > > of
> > > > >     >         > > > > > balancing restarting in case of failure
> etc.
> > > > >     >         > > > > >
> > > > >     >         > > > > > If this understanding is correct then it
> just
> > > > > feels to
> > > > >     > me like an
> > > > >     >         > > awful
> > > > >     >         > > > > lot
> > > > >     >         > > > > > of Connect functionality would need to be
> > > > > reimplemented
> > > > >     > or at least
> > > > >     >         > > > > > wrapped, which potentially could mean
> > > additional
> > > > > effort
> > > > >     > for
> > > > >     >         > > maintaining
> > > > >     >         > > > > and
> > > > >     >         > > > > > extending Connect down the line. Wouldn't
> it
> > > be a
> > > > > viable
> > > > >     >         > alternative
> > > > >     >         > > to
> > > > >     >         > > > > > look into extending Connect itself to allow
> > > > > defining
> > > > >     > "remote
> > > > >     >         > > clusters"
> > > > >     >         > > > > > which can then be specified in the
> connector
> > > > > config to
> > > > >     > be used
> > > > >     >         > > instead
> > > > >     >         > > > of
> > > > >     >         > > > > > the local cluster? I imagine that change
> > itself
> > > > > would
> > > > >     > not be too
> > > > >     >         > > > > extensive,
> > > > >     >         > > > > > the main effort would probably be in coming
> > up
> > > > > with a
> > > > >     > sensible
> > > > >     >         > config
> > > > >     >         > > > > > structure and ensuring backwards
> > compatibility
> > > > with
> > > > >     > existing
> > > > >     >         > > connector
> > > > >     >         > > > > > configs.
> > > > >     >         > > > > > This would still allow to use a regular
> > Connect
> > > > > cluster
> > > > >     > for an
> > > > >     >         > > > arbitrary
> > > > >     >         > > > > > number of clusters, thus still having a
> > > dedicated
> > > > >     > MirrorMaker
> > > > >     >         > cluster
> > > > >     >         > > > by
> > > > >     >         > > > > > running only MirrorMaker Connectors in
> there
> > if
> > > > > you want
> > > > >     > the
> > > > >     >         > > > isolation. I
> > > > >     >         > > > > > agree that it would not offer the level of
> > > > > abstraction
> > > > >     > around
> > > > >     >         > > > replication
> > > > >     >         > > > > > that your concept would enable to
> implement,
> > > but
> > > > I
> > > > > think
> > > > >     > if would
> > > > >     >         > be
> > > > >     >         > > > far
> > > > >     >         > > > > > less implementation and maintenance effort.
> > > > >     >         > > > > >
> > > > >     >         > > > > > But again, all of that is based on my,
> > > > potentially
> > > > >     > flawed,
> > > > >     >         > > > understanding
> > > > >     >         > > > > of
> > > > >     >         > > > > > your proposal, please feel free to correct
> me
> > > :)
> > > > >     >         > > > > >
> > > > >     >         > > > > > Best regards,
> > > > >     >         > > > > > Sönke
> > > > >     >         > > > > >
> > > > >     >         > > > > > On Fri, Nov 30, 2018 at 1:39 AM Ryanne
> Dolan
> > <
> > > > >     >         > ryannedolan@gmail.com>
> > > > >     >         > > > > > wrote:
> > > > >     >         > > > > >
> > > > >     >         > > > > > > Sönke, thanks for the feedback!
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > >  the renaming policy [...] can be
> > disabled
> > > > > [...] The
> > > > >     > KIP itself
> > > > >     >         > > > does
> > > > >     >         > > > > > not
> > > > >     >         > > > > > > mention this
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > Good catch. I've updated the KIP to call
> > this
> > > > > out.
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > > "MirrorMaker clusters" I am not sure I
> > > fully
> > > > >     > understand the
> > > > >     >         > issue
> > > > >     >         > > > you
> > > > >     >         > > > > > > are trying to solve
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > MirrorMaker today is not scalable from an
> > > > > operational
> > > > >     >         > perspective.
> > > > >     >         > > > > Celia
> > > > >     >         > > > > > > Kung at LinkedIn does a great job of
> > > explaining
> > > > > this
> > > > >     > problem [1],
> > > > >     >         > > > which
> > > > >     >         > > > > > has
> > > > >     >         > > > > > > caused LinkedIn to drop MirrorMaker in
> > favor
> > > of
> > > > >     > Brooklin. With
> > > > >     >         > > > > Brooklin,
> > > > >     >         > > > > > a
> > > > >     >         > > > > > > single cluster, single API, and single UI
> > > > > controls
> > > > >     > replication
> > > > >     >         > > flows
> > > > >     >         > > > > for
> > > > >     >         > > > > > an
> > > > >     >         > > > > > > entire data center. With MirrorMaker 2.0,
> > the
> > > > > vision
> > > > >     > is much the
> > > > >     >         > > > same.
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > If your data center consists of a small
> > > number
> > > > of
> > > > >     > Kafka clusters
> > > > >     >         > > and
> > > > >     >         > > > an
> > > > >     >         > > > > > > existing Connect cluster, it might make
> > more
> > > > > sense to
> > > > >     > re-use the
> > > > >     >         > > > > Connect
> > > > >     >         > > > > > > cluster with MirrorSource/SinkConnectors.
> > > > There's
> > > > >     > nothing wrong
> > > > >     >         > > with
> > > > >     >         > > > > this
> > > > >     >         > > > > > > approach for small deployments, but this
> > > model
> > > > > also
> > > > >     > doesn't
> > > > >     >         > scale.
> > > > >     >         > > > This
> > > > >     >         > > > > > is
> > > > >     >         > > > > > > because Connect clusters are built
> around a
> > > > > single
> > > > >     > Kafka cluster
> > > > >     >         > --
> > > > >     >         > > > > what
> > > > >     >         > > > > > I
> > > > >     >         > > > > > > call the "primary" cluster -- and all
> > > > Connectors
> > > > > in
> > > > >     > the cluster
> > > > >     >         > > must
> > > > >     >         > > > > > either
> > > > >     >         > > > > > > consume from or produce to this single
> > > cluster.
> > > > > If you
> > > > >     > have more
> > > > >     >         > > than
> > > > >     >         > > > > one
> > > > >     >         > > > > > > "active" Kafka cluster in each data
> center,
> > > > > you'll end
> > > > >     > up needing
> > > > >     >         > > > > > multiple
> > > > >     >         > > > > > > Connect clusters there as well.
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > The problem with Connect clusters for
> > > > > replication is
> > > > >     > way less
> > > > >     >         > > severe
> > > > >     >         > > > > > > compared to legacy MirrorMaker. Generally
> > you
> > > > > need one
> > > > >     > Connect
> > > > >     >         > > > cluster
> > > > >     >         > > > > > per
> > > > >     >         > > > > > > active Kafka cluster. As you point out,
> > MM2's
> > > > >     > SinkConnector means
> > > > >     >         > > you
> > > > >     >         > > > > can
> > > > >     >         > > > > > > get away with a single Connect cluster
> for
> > > > > topologies
> > > > >     > that center
> > > > >     >         > > > > around
> > > > >     >         > > > > > a
> > > > >     >         > > > > > > single primary cluster. But each
> Connector
> > > > > within each
> > > > >     > Connect
> > > > >     >         > > > cluster
> > > > >     >         > > > > > must
> > > > >     >         > > > > > > be configured independently, with no
> > > high-level
> > > > > view
> > > > >     > of your
> > > > >     >         > > > > replication
> > > > >     >         > > > > > > flows within and between data centers.
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > With MirrorMaker 2.0, a single
> MirrorMaker
> > > > > cluster
> > > > >     > manages
> > > > >     >         > > > replication
> > > > >     >         > > > > > > across any number of Kafka clusters. Much
> > > like
> > > > >     > Brooklin, MM2 does
> > > > >     >         > > the
> > > > >     >         > > > > > work
> > > > >     >         > > > > > > of setting up connectors between clusters
> > as
> > > > > needed.
> > > > >     > This
> > > > >     >         > > > > > > Replication-as-a-Service is a huge win
> for
> > > > larger
> > > > >     > deployments, as
> > > > >     >         > > > well
> > > > >     >         > > > > as
> > > > >     >         > > > > > > for organizations that haven't adopted
> > > Connect.
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > [1]
> > > > >     >         > > > > > >
> > > > >     >         > > > > >
> > > > >     >         > > > >
> > > > >     >         > > >
> > > > >     >         > >
> > > > >     >         >
> > > > >     >
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/ConfluentInc/more-data-more-problems-scaling-kafkamirroring-pipelines-at-linkedin
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > Keep the questions coming! Thanks.
> > > > >     >         > > > > > > Ryanne
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > On Thu, Nov 29, 2018 at 3:30 AM Sönke
> > Liebau
> > > <
> > > > >     >         > > > > soenke.liebau@opencore.com
> > > > >     >         > > > > > >
> > > > >     >         > > > > > > wrote:
> > > > >     >         > > > > > >
> > > > >     >         > > > > > >> Hi Ryanne,
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> first of all, thanks for the KIP, great
> > work
> > > > > overall
> > > > >     > and much
> > > > >     >         > > > needed I
> > > > >     >         > > > > > >> think!
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> I have a small comment on the renaming
> > > policy,
> > > > > in one
> > > > >     > of the
> > > > >     >         > mails
> > > > >     >         > > > on
> > > > >     >         > > > > > >> this thread you mention that this can be
> > > > > disabled (to
> > > > >     > replicate
> > > > >     >         > > > topic1
> > > > >     >         > > > > > in
> > > > >     >         > > > > > >> cluster A as topic1 on cluster B I
> > assume).
> > > > The
> > > > > KIP
> > > > >     > itself does
> > > > >     >         > > not
> > > > >     >         > > > > > mention
> > > > >     >         > > > > > >> this, from reading just the KIP one
> might
> > > get
> > > > > the
> > > > >     > assumption
> > > > >     >         > that
> > > > >     >         > > > > > renaming
> > > > >     >         > > > > > >> is mandatory. It might be useful to add
> a
> > > > > sentence or
> > > > >     > two around
> > > > >     >         > > > > > renaming
> > > > >     >         > > > > > >> policies and what is possible here. I
> > assume
> > > > you
> > > > >     > intend to make
> > > > >     >         > > > these
> > > > >     >         > > > > > >> pluggable?
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> Regarding the latest addition of
> > > "MirrorMaker
> > > > >     > clusters" I am not
> > > > >     >         > > > sure
> > > > >     >         > > > > I
> > > > >     >         > > > > > >> fully understand the issue you are
> trying
> > to
> > > > > solve
> > > > >     > and what
> > > > >     >         > > exactly
> > > > >     >         > > > > > these
> > > > >     >         > > > > > >> scripts will do - but that may just me
> > being
> > > > > dense
> > > > >     > about it :)
> > > > >     >         > > > > > >> I understand the limitation to a single
> > > source
> > > > > and
> > > > >     > target
> > > > >     >         > cluster
> > > > >     >         > > > that
> > > > >     >         > > > > > >> Connect imposes, but isn't this worked
> > > around
> > > > > by the
> > > > >     > fact that
> > > > >     >         > you
> > > > >     >         > > > > have
> > > > >     >         > > > > > >> MirrorSource- and MirrorSinkConnectors
> and
> > > one
> > > > > part
> > > > >     > of the
> > > > >     >         > > equation
> > > > >     >         > > > > will
> > > > >     >         > > > > > >> always be under your control?
> > > > >     >         > > > > > >> The way I understood your intention was
> > that
> > > > > there is
> > > > >     > a
> > > > >     >         > (regular,
> > > > >     >         > > > not
> > > > >     >         > > > > > MM)
> > > > >     >         > > > > > >> Connect Cluster somewhere next to a
> Kafka
> > > > > Cluster A
> > > > >     > and if you
> > > > >     >         > > > deploy
> > > > >     >         > > > > a
> > > > >     >         > > > > > >> MirrorSourceTask to that it will read
> > > messages
> > > > > from a
> > > > >     > remote
> > > > >     >         > > > cluster B
> > > > >     >         > > > > > and
> > > > >     >         > > > > > >> replicate them into the local cluster A.
> > If
> > > > you
> > > > >     > deploy a
> > > > >     >         > > > > MirrorSinkTask
> > > > >     >         > > > > > it
> > > > >     >         > > > > > >> will read from local cluster A and
> > replicate
> > > > > into
> > > > >     > cluster B.
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> Since in both causes the configuration
> for
> > > > > cluster B
> > > > >     > will be
> > > > >     >         > > passed
> > > > >     >         > > > > into
> > > > >     >         > > > > > >> the connector in the ConnectorConfig
> > > contained
> > > > > in the
> > > > >     > rest
> > > > >     >         > > request,
> > > > >     >         > > > > > what's
> > > > >     >         > > > > > >> to stop us from starting a third
> connector
> > > > with
> > > > > a
> > > > >     >         > MirrorSourceTask
> > > > >     >         > > > > > reading
> > > > >     >         > > > > > >> from cluster C?
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> I am a bit hesitant about the entire
> > concept
> > > > of
> > > > >     > having extra
> > > > >     >         > > scripts
> > > > >     >         > > > > to
> > > > >     >         > > > > > >> run an entire separate Connect cluster -
> > I'd
> > > > > much
> > > > >     > prefer an
> > > > >     >         > option
> > > > >     >         > > > to
> > > > >     >         > > > > > use a
> > > > >     >         > > > > > >> regular connect cluster from an ops
> point
> > of
> > > > > view. Is
> > > > >     > it maybe
> > > > >     >         > > worth
> > > > >     >         > > > > > >> spending some time investigating whether
> > we
> > > > can
> > > > > come
> > > > >     > up with a
> > > > >     >         > > > change
> > > > >     >         > > > > to
> > > > >     >         > > > > > >> connect that enables what MM would need?
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> Best regards,
> > > > >     >         > > > > > >> Sönke
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> On Tue, Nov 27, 2018 at 10:02 PM Ryanne
> > > Dolan
> > > > <
> > > > >     >         > > > ryannedolan@gmail.com>
> > > > >     >         > > > > > >> wrote:
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >>> Hey y'all, I'd like you draw your
> > attention
> > > > to
> > > > > a new
> > > > >     > section in
> > > > >     >         > > > > KIP-382
> > > > >     >         > > > > > >>> re
> > > > >     >         > > > > > >>> MirrorMaker Clusters:
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > >
> > > > >     >         > > > >
> > > > >     >         > > >
> > > > >     >         > >
> > > > >     >         >
> > > > >     >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382:+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-MirrorMakerClusters
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> A common concern I hear about using
> > Connect
> > > > for
> > > > >     > replication is
> > > > >     >         > > that
> > > > >     >         > > > > all
> > > > >     >         > > > > > >>> SourceConnectors in a Connect cluster
> > must
> > > > use
> > > > > the
> > > > >     > same target
> > > > >     >         > > > Kafka
> > > > >     >         > > > > > >>> cluster, and likewise all
> SinkConnectors
> > > must
> > > > > use
> > > > >     > the same
> > > > >     >         > source
> > > > >     >         > > > > Kafka
> > > > >     >         > > > > > >>> cluster. In order to use multiple Kafka
> > > > > clusters
> > > > >     > from Connect,
> > > > >     >         > > > there
> > > > >     >         > > > > > are
> > > > >     >         > > > > > >>> two possible approaches:
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> 1) use an intermediate Kafka cluster,
> K.
> > > > >     > SourceConnectors (A,
> > > > >     >         > B,
> > > > >     >         > > C)
> > > > >     >         > > > > > write
> > > > >     >         > > > > > >>> to K and SinkConnectors (X, Y, Z) read
> > from
> > > > K.
> > > > > This
> > > > >     > enables
> > > > >     >         > flows
> > > > >     >         > > > > like
> > > > >     >         > > > > > A
> > > > >     >         > > > > > >>> ->
> > > > >     >         > > > > > >>> K - > X but means that some topologies
> > > > require
> > > > >     > extraneous hops,
> > > > >     >         > > and
> > > > >     >         > > > > > means
> > > > >     >         > > > > > >>> that K must be scaled to handle records
> > > from
> > > > > all
> > > > >     > sources and
> > > > >     >         > > sinks.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> 2) use multiple Connect clusters, one
> for
> > > > each
> > > > >     > target cluster.
> > > > >     >         > > Each
> > > > >     >         > > > > > >>> cluster
> > > > >     >         > > > > > >>> has multiple SourceConnectors, one for
> > each
> > > > > source
> > > > >     > cluster.
> > > > >     >         > This
> > > > >     >         > > > > > enables
> > > > >     >         > > > > > >>> direct replication of A -> X but means
> > > there
> > > > > is a
> > > > >     > proliferation
> > > > >     >         > > of
> > > > >     >         > > > > > >>> Connect
> > > > >     >         > > > > > >>> clusters, each of which must be managed
> > > > > separately.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> Both options are viable for small
> > > deployments
> > > > >     > involving a small
> > > > >     >         > > > > number
> > > > >     >         > > > > > of
> > > > >     >         > > > > > >>> Kafka clusters in a small number of
> data
> > > > > centers.
> > > > >     > However,
> > > > >     >         > > neither
> > > > >     >         > > > is
> > > > >     >         > > > > > >>> scalable, especially from an
> operational
> > > > > standpoint.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> KIP-382 now introduces "MirrorMaker
> > > > clusters",
> > > > > which
> > > > >     > are
> > > > >     >         > distinct
> > > > >     >         > > > > from
> > > > >     >         > > > > > >>> Connect clusters. A single MirrorMaker
> > > > cluster
> > > > >     > provides
> > > > >     >         > > > > > >>> "Replication-as-a-Service" among any
> > number
> > > > of
> > > > > Kafka
> > > > >     > clusters
> > > > >     >         > > via a
> > > > >     >         > > > > > >>> high-level REST API based on the
> Connect
> > > API.
> > > > > Under
> > > > >     > the hood,
> > > > >     >         > > > > > MirrorMaker
> > > > >     >         > > > > > >>> sets up Connectors between each pair of
> > > Kafka
> > > > >     > clusters. The
> > > > >     >         > REST
> > > > >     >         > > > API
> > > > >     >         > > > > > >>> enables on-the-fly reconfiguration of
> > each
> > > > >     > Connector, including
> > > > >     >         > > > > updates
> > > > >     >         > > > > > >>> to
> > > > >     >         > > > > > >>> topic whitelists/blacklists.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> To configure MirrorMaker 2.0, you need
> a
> > > > >     > configuration file
> > > > >     >         > that
> > > > >     >         > > > > lists
> > > > >     >         > > > > > >>> connection information for each Kafka
> > > cluster
> > > > >     > (broker lists,
> > > > >     >         > SSL
> > > > >     >         > > > > > settings
> > > > >     >         > > > > > >>> etc). At a minimum, this looks like:
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> clusters=us-west, us-east
> > > > >     >         > > > > > >>>
> > > > > cluster.us-west.broker.list=us-west-kafka-server:9092
> > > > >     >         > > > > > >>>
> > > > > cluster.us-east.broker.list=us-east-kafka-server:9092
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> You can specify topic whitelists and
> > other
> > > > >     > connector-level
> > > > >     >         > > settings
> > > > >     >         > > > > > here
> > > > >     >         > > > > > >>> too, or you can use the REST API to
> > > > > remote-control a
> > > > >     > running
> > > > >     >         > > > cluster.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> I've also updated the KIP with minor
> > > changes
> > > > to
> > > > >     > bring it in
> > > > >     >         > line
> > > > >     >         > > > with
> > > > >     >         > > > > > the
> > > > >     >         > > > > > >>> current implementation.
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> Looking forward to your feedback,
> thanks!
> > > > >     >         > > > > > >>> Ryanne
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> On Mon, Nov 19, 2018 at 10:26 PM Ryanne
> > > > Dolan <
> > > > >     >         > > > ryannedolan@gmail.com
> > > > >     >         > > > > >
> > > > >     >         > > > > > >>> wrote:
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>> > Dan, you've got it right. ACL sync
> will
> > > be
> > > > > done by
> > > > >     > MM2
> > > > >     >         > > > > automatically
> > > > >     >         > > > > > >>> > (unless disabled) according to simple
> > > > rules:
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > - If a principal has READ access on a
> > > topic
> > > > > in a
> > > > >     > source
> > > > >     >         > > cluster,
> > > > >     >         > > > > the
> > > > >     >         > > > > > >>> same
> > > > >     >         > > > > > >>> > principal should have READ access on
> > > > > downstream
> > > > >     > replicated
> > > > >     >         > > topics
> > > > >     >         > > > > > >>> ("remote
> > > > >     >         > > > > > >>> > topics").
> > > > >     >         > > > > > >>> > - Only MM2 has WRITE access on
> "remote
> > > > > topics".
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > This covers sync from upstream topics
> > > like
> > > > >     > "topic1" to
> > > > >     >         > > downstream
> > > > >     >         > > > > > >>> remote
> > > > >     >         > > > > > >>> > topics like "us-west.topic1". What's
> > > > missing
> > > > > from
> > > > >     > the KIP, as
> > > > >     >         > > you
> > > > >     >         > > > > > point
> > > > >     >         > > > > > >>> > out, is ACL sync between normal
> topics
> > > > >     > (non-remote). If a
> > > > >     >         > > > consumer
> > > > >     >         > > > > > has
> > > > >     >         > > > > > >>> READ
> > > > >     >         > > > > > >>> > access to topic1 in an upstream
> > cluster,
> > > > > should it
> > > > >     > have READ
> > > > >     >         > > > access
> > > > >     >         > > > > > in
> > > > >     >         > > > > > >>> > topic1 in a downstream cluster?
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > I think the answer generally is no,
> you
> > > > > don't want
> > > > >     > to give
> > > > >     >         > > > > principals
> > > > >     >         > > > > > >>> > blanket permissions across all DCs
> > > > > automatically.
> > > > >     > For
> > > > >     >         > example,
> > > > >     >         > > > I've
> > > > >     >         > > > > > >>> seen
> > > > >     >         > > > > > >>> > scenarios where certain topics are
> > > > replicated
> > > > >     > between an
> > > > >     >         > > internal
> > > > >     >         > > > > and
> > > > >     >         > > > > > >>> > external Kafka cluster. You don't
> want
> > to
> > > > >     > accidentally push
> > > > >     >         > ACL
> > > > >     >         > > > > > changes
> > > > >     >         > > > > > >>> > across this boundary.
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > Moreover, it's clear that MM2 "owns"
> > > > > downstream
> > > > >     > remote topics
> > > > >     >         > > > like
> > > > >     >         > > > > > >>> > "us-west.topic1" -- MM2 is the only
> > > > producer
> > > > > and
> > > > >     > the only
> > > > >     >         > admin
> > > > >     >         > > > of
> > > > >     >         > > > > > >>> these
> > > > >     >         > > > > > >>> > topics -- so it's natural to have MM2
> > set
> > > > > the ACL
> > > > >     > for these
> > > > >     >         > > > topics.
> > > > >     >         > > > > > >>> But I
> > > > >     >         > > > > > >>> > think it would be surprising if MM2
> > tried
> > > > to
> > > > >     > manipulate
> > > > >     >         > topics
> > > > >     >         > > it
> > > > >     >         > > > > > >>> doesn't
> > > > >     >         > > > > > >>> > own. So I think granting permissions
> > > across
> > > > > DCs is
> > > > >     > probably
> > > > >     >         > > > outside
> > > > >     >         > > > > > >>> MM2's
> > > > >     >         > > > > > >>> > purview, but I agree it'd be nice to
> > have
> > > > > tooling
> > > > >     > to help
> > > > >     >         > with
> > > > >     >         > > > > this.
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > Thanks.
> > > > >     >         > > > > > >>> > Ryanne
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > --
> > > > >     >         > > > > > >>> > www.ryannedolan.info
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> > On Mon, Nov 19, 2018 at 3:58 PM
> > > > >     > daniel.locious@gmail.com <
> > > > >     >         > > > > > >>> > daniel.locious@gmail.com> wrote:
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> >> Hi guys,
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> This is an exciting topic. could I
> > have
> > > a
> > > > > word
> > > > >     > here?
> > > > >     >         > > > > > >>> >> I understand there are many
> scenarios
> > > that
> > > > > we can
> > > > >     > apply
> > > > >     >         > > > > mirrormaker.
> > > > >     >         > > > > > >>> I am
> > > > >     >         > > > > > >>> >> at the moment working on
> active/active
> > > DC
> > > > >     > solution using
> > > > >     >         > > > > > MirrorMaker;
> > > > >     >         > > > > > >>> our
> > > > >     >         > > > > > >>> >> goal is to allow  the clients to
> > > failover
> > > > to
> > > > >     > connect the
> > > > >     >         > other
> > > > >     >         > > > > kafka
> > > > >     >         > > > > > >>> >> cluster (on the other DC) when an
> > > incident
> > > > >     > happens.
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> To do this, I need:
> > > > >     >         > > > > > >>> >> 1 MirrorMaker to replicate the
> > > partitioned
> > > > >     > messages in a
> > > > >     >         > > > > sequential
> > > > >     >         > > > > > >>> order
> > > > >     >         > > > > > >>> >> (in timely fashion) to the same
> > > partition
> > > > > on the
> > > > >     > other
> > > > >     >         > cluster
> > > > >     >         > > > > (also
> > > > >     >         > > > > > >>> need
> > > > >     >         > > > > > >>> >> keep the promise that both clusters
> > > > creates
> > > > > the
> > > > >     > same number
> > > > >     >         > of
> > > > >     >         > > > > > >>> partitions
> > > > >     >         > > > > > >>> >> for a topic) – so that a consumer
> can
> > > pick
> > > > > up the
> > > > >     > right
> > > > >     >         > order
> > > > >     >         > > of
> > > > >     >         > > > > the
> > > > >     >         > > > > > >>> latest
> > > > >     >         > > > > > >>> >> messages
> > > > >     >         > > > > > >>> >> 2 MirrorMaker to replicate the local
> > > > > consumer
> > > > >     > offset to the
> > > > >     >         > > > other
> > > > >     >         > > > > > >>> side –
> > > > >     >         > > > > > >>> >> so that the consumer knows where is
> > the
> > > > > offset/
> > > > >     > latest
> > > > >     >         > > messages
> > > > >     >         > > > > > >>> >> 3 MirrorMaker to provide cycle
> > detection
> > > > for
> > > > >     > messages across
> > > > >     >         > > the
> > > > >     >         > > > > > DCs.
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> I can see the possibility for Remote
> > > Topic
> > > > > to
> > > > >     > solve all
> > > > >     >         > these
> > > > >     >         > > > > > >>> problems,
> > > > >     >         > > > > > >>> >> as long as the consumer can see the
> > > remote
> > > > > topic
> > > > >     > equally as
> > > > >     >         > > the
> > > > >     >         > > > > > local
> > > > >     >         > > > > > >>> >> topic, i.e. For a consumer which
> has a
> > > > > permission
> > > > >     > to consume
> > > > >     >         > > > > topic1,
> > > > >     >         > > > > > >>> on
> > > > >     >         > > > > > >>> >> subscribe event it can automatically
> > > > > subscribe
> > > > >     > both
> > > > >     >         > > > remote.topic1
> > > > >     >         > > > > > and
> > > > >     >         > > > > > >>> >> local.topic1. First we need to find
> a
> > > way
> > > > > for
> > > > >     > topic ACL
> > > > >     >         > > granting
> > > > >     >         > > > > for
> > > > >     >         > > > > > >>> the
> > > > >     >         > > > > > >>> >> consumer across the DCs. Secondly
> the
> > > > > consumer
> > > > >     > need to be
> > > > >     >         > able
> > > > >     >         > > > to
> > > > >     >         > > > > > >>> subscribe
> > > > >     >         > > > > > >>> >> topics with wildcard or suffix. Last
> > but
> > > > > not the
> > > > >     > least, the
> > > > >     >         > > > > consumer
> > > > >     >         > > > > > >>> has to
> > > > >     >         > > > > > >>> >> deal with the timely ordering of the
> > > > > messages
> > > > >     > from the 2
> > > > >     >         > > topics.
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> My understanding is, all of these
> > should
> > > > be
> > > > >     > configurable to
> > > > >     >         > be
> > > > >     >         > > > > > turned
> > > > >     >         > > > > > >>> on
> > > > >     >         > > > > > >>> >> or off, to fit for different use
> > cases.
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> Interesting I was going to support
> > topic
> > > > > messages
> > > > >     > with extra
> > > > >     >         > > > > headers
> > > > >     >         > > > > > >>> of
> > > > >     >         > > > > > >>> >> source DC info, for cycle
> detection…..
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> Looking forward your reply.
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> Regards,
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >> Dan
> > > > >     >         > > > > > >>> >> On 2018/10/23 19:56:02, Ryanne
> Dolan <
> > > > >     > ryannedolan@gmail.com
> > > > >     >         > >
> > > > >     >         > > > > wrote:
> > > > >     >         > > > > > >>> >> > Alex, thanks for the feedback.
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > > Would it be possible to utilize
> > the
> > > > >     >         > > > > > >>> >> > > Message Headers feature to
> prevent
> > > > > infinite
> > > > >     > recursion
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > This isn't necessary due to the
> > topic
> > > > > renaming
> > > > >     > feature
> > > > >     >         > which
> > > > >     >         > > > > > already
> > > > >     >         > > > > > >>> >> > prevents infinite recursion.
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > If you turn off topic renaming you
> > > lose
> > > > > cycle
> > > > >     > detection,
> > > > >     >         > so
> > > > >     >         > > > > maybe
> > > > >     >         > > > > > we
> > > > >     >         > > > > > >>> >> could
> > > > >     >         > > > > > >>> >> > provide message headers as an
> > optional
> > > > > second
> > > > >     > mechanism.
> > > > >     >         > I'm
> > > > >     >         > > > not
> > > > >     >         > > > > > >>> >> opposed to
> > > > >     >         > > > > > >>> >> > that idea, but there are ways to
> > > improve
> > > > >     > efficiency if we
> > > > >     >         > > > don't
> > > > >     >         > > > > > >>> need to
> > > > >     >         > > > > > >>> >> > modify or inspect individual
> > records.
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > Ryanne
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > On Tue, Oct 23, 2018 at 6:06 AM
> Alex
> > > > > Mironov <
> > > > >     >         > > > > > alexandrfox@gmail.com
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>> >> wrote:
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >> > > Hey Ryanne,
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> > > Awesome KIP, exited to see
> > > > improvements
> > > > > in
> > > > >     > MirrorMaker
> > > > >     >         > > > land, I
> > > > >     >         > > > > > >>> >> particularly
> > > > >     >         > > > > > >>> >> > > like the reuse of Connect
> > framework!
> > > > > Would it
> > > > >     > be
> > > > >     >         > possible
> > > > >     >         > > to
> > > > >     >         > > > > > >>> utilize
> > > > >     >         > > > > > >>> >> the
> > > > >     >         > > > > > >>> >> > > Message Headers feature to
> prevent
> > > > > infinite
> > > > >     > recursion?
> > > > >     >         > For
> > > > >     >         > > > > > >>> example,
> > > > >     >         > > > > > >>> >> MM2
> > > > >     >         > > > > > >>> >> > > could stamp every message with a
> > > > special
> > > > >     > header payload
> > > > >     >         > > > (e.g.
> > > > >     >         > > > > > >>> >> > > MM2="cluster-name-foo") so in
> case
> > > > > another
> > > > >     > MM2 instance
> > > > >     >         > > sees
> > > > >     >         > > > > > this
> > > > >     >         > > > > > >>> >> message
> > > > >     >         > > > > > >>> >> > > and it is configured to
> replicate
> > > data
> > > > > into
> > > > >     >         > > > "cluster-name-foo"
> > > > >     >         > > > > > it
> > > > >     >         > > > > > >>> >> would
> > > > >     >         > > > > > >>> >> > > just skip it instead of
> > replicating
> > > it
> > > > > back.
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> > > On Sat, Oct 20, 2018 at 5:48 AM
> > > Ryanne
> > > > > Dolan <
> > > > >     >         > > > > > >>> ryannedolan@gmail.com>
> > > > >     >         > > > > > >>> >> > > wrote:
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> > > > Thanks Harsha. Done.
> > > > >     >         > > > > > >>> >> > > >
> > > > >     >         > > > > > >>> >> > > > On Fri, Oct 19, 2018 at 1:03
> AM
> > > > Harsha
> > > > >     > Chintalapani <
> > > > >     >         > > > > > >>> >> kafka@harsha.io>
> > > > >     >         > > > > > >>> >> > > > wrote:
> > > > >     >         > > > > > >>> >> > > >
> > > > >     >         > > > > > >>> >> > > > > Ryanne,
> > > > >     >         > > > > > >>> >> > > > >        Makes sense. Can you
> > > please
> > > > > add
> > > > >     > this under
> > > > >     >         > > > rejected
> > > > >     >         > > > > > >>> >> alternatives
> > > > >     >         > > > > > >>> >> > > > so
> > > > >     >         > > > > > >>> >> > > > > that everyone has context on
> > why
> > > > it
> > > > >     > wasn’t picked.
> > > > >     >         > > > > > >>> >> > > > >
> > > > >     >         > > > > > >>> >> > > > > Thanks,
> > > > >     >         > > > > > >>> >> > > > > Harsha
> > > > >     >         > > > > > >>> >> > > > > On Oct 18, 2018, 8:02 AM
> > -0700,
> > > > > Ryanne
> > > > >     > Dolan <
> > > > >     >         > > > > > >>> >> ryannedolan@gmail.com>,
> > > > >     >         > > > > > >>> >> > > > > wrote:
> > > > >     >         > > > > > >>> >> > > > >
> > > > >     >         > > > > > >>> >> > > > > Harsha, concerning
> uReplicator
> > > > >     > specifically, the
> > > > >     >         > > project
> > > > >     >         > > > > is
> > > > >     >         > > > > > a
> > > > >     >         > > > > > >>> >> major
> > > > >     >         > > > > > >>> >> > > > > inspiration for MM2, but I
> > don't
> > > > > think it
> > > > >     > is a good
> > > > >     >         > > > > > >>> foundation for
> > > > >     >         > > > > > >>> >> > > > anything
> > > > >     >         > > > > > >>> >> > > > > included in Apache Kafka.
> > > > > uReplicator
> > > > >     > uses Helix to
> > > > >     >         > > > solve
> > > > >     >         > > > > > >>> >> problems that
> > > > >     >         > > > > > >>> >> > > > > Connect also solves, e.g.
> REST
> > > > API,
> > > > > live
> > > > >     >         > configuration
> > > > >     >         > > > > > >>> changes,
> > > > >     >         > > > > > >>> >> cluster
> > > > >     >         > > > > > >>> >> > > > > management, coordination
> etc.
> > > This
> > > > > also
> > > > >     > means that
> > > > >     >         > > > > existing
> > > > >     >         > > > > > >>> >> tooling,
> > > > >     >         > > > > > >>> >> > > > > dashboards etc that work
> with
> > > > > Connectors
> > > > >     > do not work
> > > > >     >         > > > with
> > > > >     >         > > > > > >>> >> uReplicator,
> > > > >     >         > > > > > >>> >> > > > and
> > > > >     >         > > > > > >>> >> > > > > any future tooling would
> need
> > to
> > > > > treat
> > > > >     > uReplicator
> > > > >     >         > as
> > > > >     >         > > a
> > > > >     >         > > > > > >>> special
> > > > >     >         > > > > > >>> >> case.
> > > > >     >         > > > > > >>> >> > > > >
> > > > >     >         > > > > > >>> >> > > > > Ryanne
> > > > >     >         > > > > > >>> >> > > > >
> > > > >     >         > > > > > >>> >> > > > > On Wed, Oct 17, 2018 at
> 12:30
> > PM
> > > > > Ryanne
> > > > >     > Dolan <
> > > > >     >         > > > > > >>> >> ryannedolan@gmail.com>
> > > > >     >         > > > > > >>> >> > > > > wrote:
> > > > >     >         > > > > > >>> >> > > > >
> > > > >     >         > > > > > >>> >> > > > >> Harsha, yes I can do that.
> > I'll
> > > > > update
> > > > >     > the KIP
> > > > >     >         > > > > accordingly,
> > > > >     >         > > > > > >>> >> thanks.
> > > > >     >         > > > > > >>> >> > > > >>
> > > > >     >         > > > > > >>> >> > > > >> Ryanne
> > > > >     >         > > > > > >>> >> > > > >>
> > > > >     >         > > > > > >>> >> > > > >> On Wed, Oct 17, 2018 at
> 12:18
> > > PM
> > > > > Harsha <
> > > > >     >         > > > kafka@harsha.io
> > > > >     >         > > > > >
> > > > >     >         > > > > > >>> wrote:
> > > > >     >         > > > > > >>> >> > > > >>
> > > > >     >         > > > > > >>> >> > > > >>> Hi Ryanne,
> > > > >     >         > > > > > >>> >> > > > >>>                Thanks for
> > the
> > > > > KIP. I am
> > > > >     > also
> > > > >     >         > curious
> > > > >     >         > > > > about
> > > > >     >         > > > > > >>> why
> > > > >     >         > > > > > >>> >> not
> > > > >     >         > > > > > >>> >> > > use
> > > > >     >         > > > > > >>> >> > > > >>> the uReplicator design as
> > the
> > > > >     > foundation given it
> > > > >     >         > > > > alreadys
> > > > >     >         > > > > > >>> >> resolves
> > > > >     >         > > > > > >>> >> > > > some of
> > > > >     >         > > > > > >>> >> > > > >>> the fundamental issues in
> > > > current
> > > > >     > MIrrorMaker,
> > > > >     >         > > > updating
> > > > >     >         > > > > > the
> > > > >     >         > > > > > >>> >> confifgs
> > > > >     >         > > > > > >>> >> > > > on the
> > > > >     >         > > > > > >>> >> > > > >>> fly and running the mirror
> > > maker
> > > > > agents
> > > > >     > in a
> > > > >     >         > worker
> > > > >     >         > > > > model
> > > > >     >         > > > > > >>> which
> > > > >     >         > > > > > >>> >> can
> > > > >     >         > > > > > >>> >> > > > >>> deployed in mesos or
> > container
> > > > >     > orchestrations.  If
> > > > >     >         > > > > > possible
> > > > >     >         > > > > > >>> can
> > > > >     >         > > > > > >>> >> you
> > > > >     >         > > > > > >>> >> > > > >>> document in the rejected
> > > > > alternatives
> > > > >     > what are
> > > > >     >         > > missing
> > > > >     >         > > > > > parts
> > > > >     >         > > > > > >>> >> that
> > > > >     >         > > > > > >>> >> > > made
> > > > >     >         > > > > > >>> >> > > > you
> > > > >     >         > > > > > >>> >> > > > >>> to consider a new design
> > from
> > > > > ground up.
> > > > >     >         > > > > > >>> >> > > > >>>
> > > > >     >         > > > > > >>> >> > > > >>> Thanks,
> > > > >     >         > > > > > >>> >> > > > >>> Harsha
> > > > >     >         > > > > > >>> >> > > > >>>
> > > > >     >         > > > > > >>> >> > > > >>> On Wed, Oct 17, 2018, at
> > 8:34
> > > > AM,
> > > > >     > Ryanne Dolan
> > > > >     >         > > wrote:
> > > > >     >         > > > > > >>> >> > > > >>> > Jan, these are two
> > separate
> > > > > issues.
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > 1) consumer coordination
> > > > should
> > > > > not,
> > > > >     > ideally,
> > > > >     >         > > > involve
> > > > >     >         > > > > > >>> >> unreliable or
> > > > >     >         > > > > > >>> >> > > > >>> slow
> > > > >     >         > > > > > >>> >> > > > >>> > connections. Naively, a
> > > > >     > KafkaSourceConnector
> > > > >     >         > would
> > > > >     >         > > > > > >>> coordinate
> > > > >     >         > > > > > >>> >> via
> > > > >     >         > > > > > >>> >> > > the
> > > > >     >         > > > > > >>> >> > > > >>> > source cluster. We can
> do
> > > > > better than
> > > > >     > this, but
> > > > >     >         > > I'm
> > > > >     >         > > > > > >>> deferring
> > > > >     >         > > > > > >>> >> this
> > > > >     >         > > > > > >>> >> > > > >>> > optimization for now.
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > 2) exactly-once between
> > two
> > > > > clusters
> > > > >     > is
> > > > >     >         > > > mind-bending.
> > > > >     >         > > > > > But
> > > > >     >         > > > > > >>> >> keep in
> > > > >     >         > > > > > >>> >> > > > mind
> > > > >     >         > > > > > >>> >> > > > >>> that
> > > > >     >         > > > > > >>> >> > > > >>> > transactions are managed
> > by
> > > > the
> > > > >     > producer, not
> > > > >     >         > the
> > > > >     >         > > > > > >>> consumer. In
> > > > >     >         > > > > > >>> >> > > fact,
> > > > >     >         > > > > > >>> >> > > > >>> it's
> > > > >     >         > > > > > >>> >> > > > >>> > the producer that
> requests
> > > > that
> > > > >     > offsets be
> > > > >     >         > > committed
> > > > >     >         > > > > for
> > > > >     >         > > > > > >>> the
> > > > >     >         > > > > > >>> >> > > current
> > > > >     >         > > > > > >>> >> > > > >>> > transaction. Obviously,
> > > these
> > > > > offsets
> > > > >     > are
> > > > >     >         > > committed
> > > > >     >         > > > in
> > > > >     >         > > > > > >>> >> whatever
> > > > >     >         > > > > > >>> >> > > > >>> cluster the
> > > > >     >         > > > > > >>> >> > > > >>> > producer is sending to.
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > These two issues are
> > closely
> > > > > related.
> > > > >     > They are
> > > > >     >         > > both
> > > > >     >         > > > > > >>> resolved
> > > > >     >         > > > > > >>> >> by not
> > > > >     >         > > > > > >>> >> > > > >>> > coordinating or
> committing
> > > via
> > > > > the
> > > > >     > source
> > > > >     >         > cluster.
> > > > >     >         > > > And
> > > > >     >         > > > > > in
> > > > >     >         > > > > > >>> >> fact,
> > > > >     >         > > > > > >>> >> > > this
> > > > >     >         > > > > > >>> >> > > > >>> is the
> > > > >     >         > > > > > >>> >> > > > >>> > general model of
> > > > > SourceConnectors
> > > > >     > anyway, since
> > > > >     >         > > most
> > > > >     >         > > > > > >>> >> > > SourceConnectors
> > > > >     >         > > > > > >>> >> > > > >>> > _only_ have a
> destination
> > > > > cluster.
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > If there is a lot of
> > > interest
> > > > > here, I
> > > > >     > can
> > > > >     >         > expound
> > > > >     >         > > > > > further
> > > > >     >         > > > > > >>> on
> > > > >     >         > > > > > >>> >> this
> > > > >     >         > > > > > >>> >> > > > >>> aspect of
> > > > >     >         > > > > > >>> >> > > > >>> > MM2, but again I think
> > this
> > > is
> > > > >     > premature until
> > > > >     >         > > this
> > > > >     >         > > > > > first
> > > > >     >         > > > > > >>> KIP
> > > > >     >         > > > > > >>> >> is
> > > > >     >         > > > > > >>> >> > > > >>> approved.
> > > > >     >         > > > > > >>> >> > > > >>> > I intend to address each
> > of
> > > > > these in
> > > > >     > separate
> > > > >     >         > KIPs
> > > > >     >         > > > > > >>> following
> > > > >     >         > > > > > >>> >> this
> > > > >     >         > > > > > >>> >> > > > one.
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > Ryanne
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > On Wed, Oct 17, 2018 at
> > 7:09
> > > > AM
> > > > > Jan
> > > > >     > Filipiak <
> > > > >     >         > > > > > >>> >> > > > Jan.Filipiak@trivago.com
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > wrote:
> > > > >     >         > > > > > >>> >> > > > >>> >
> > > > >     >         > > > > > >>> >> > > > >>> > > This is not a
> > performance
> > > > >     > optimisation. Its a
> > > > >     >         > > > > > >>> fundamental
> > > > >     >         > > > > > >>> >> design
> > > > >     >         > > > > > >>> >> > > > >>> choice.
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>> > > I never really took a
> > look
> > > > how
> > > > >     > streams does
> > > > >     >         > > > exactly
> > > > >     >         > > > > > >>> once.
> > > > >     >         > > > > > >>> >> (its a
> > > > >     >         > > > > > >>> >> > > > trap
> > > > >     >         > > > > > >>> >> > > > >>> > > anyways and you
> usually
> > > can
> > > > > deal
> > > > >     > with at least
> > > > >     >         > > > once
> > > > >     >         > > > > > >>> >> donwstream
> > > > >     >         > > > > > >>> >> > > > pretty
> > > > >     >         > > > > > >>> >> > > > >>> > > easy). But I am very
> > > certain
> > > > > its
> > > > >     > not gonna get
> > > > >     >         > > > > > >>> somewhere if
> > > > >     >         > > > > > >>> >> > > offset
> > > > >     >         > > > > > >>> >> > > > >>> > > commit and record
> > produce
> > > > > cluster
> > > > >     > are not the
> > > > >     >         > > > same.
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>> > > Pretty sure without
> this
> > > > > _design
> > > > >     > choice_ you
> > > > >     >         > can
> > > > >     >         > > > > skip
> > > > >     >         > > > > > on
> > > > >     >         > > > > > >>> >> that
> > > > >     >         > > > > > >>> >> > > > exactly
> > > > >     >         > > > > > >>> >> > > > >>> > > once already
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>> > > Best Jan
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>> > > On 16.10.2018 18:16,
> > > Ryanne
> > > > > Dolan
> > > > >     > wrote:
> > > > >     >         > > > > > >>> >> > > > >>> > > >  >  But one big
> > obstacle
> > > > in
> > > > > this
> > > > >     > was
> > > > >     >         > > > > > >>> >> > > > >>> > > > always that group
> > > > > coordination
> > > > >     > happened on
> > > > >     >         > the
> > > > >     >         > > > > > source
> > > > >     >         > > > > > >>> >> cluster.
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > > Jan, thank you for
> > > > bringing
> > > > > up
> > > > >     > this issue
> > > > >     >         > with
> > > > >     >         > > > > > legacy
> > > > >     >         > > > > > >>> >> > > > MirrorMaker.
> > > > >     >         > > > > > >>> >> > > > >>> I
> > > > >     >         > > > > > >>> >> > > > >>> > > > totally agree with
> > you.
> > > > > This is
> > > > >     > one of
> > > > >     >         > several
> > > > >     >         > > > > > >>> problems
> > > > >     >         > > > > > >>> >> with
> > > > >     >         > > > > > >>> >> > > > >>> MirrorMaker
> > > > >     >         > > > > > >>> >> > > > >>> > > > I intend to solve in
> > > MM2,
> > > > > and I
> > > > >     > already
> > > > >     >         > have a
> > > > >     >         > > > > > design
> > > > >     >         > > > > > >>> and
> > > > >     >         > > > > > >>> >> > > > >>> prototype that
> > > > >     >         > > > > > >>> >> > > > >>> > > > solves this and
> > related
> > > > > issues.
> > > > >     > But as you
> > > > >     >         > > > pointed
> > > > >     >         > > > > > >>> out,
> > > > >     >         > > > > > >>> >> this
> > > > >     >         > > > > > >>> >> > > KIP
> > > > >     >         > > > > > >>> >> > > > is
> > > > >     >         > > > > > >>> >> > > > >>> > > > already rather
> > complex,
> > > > and
> > > > > I
> > > > >     > want to focus
> > > > >     >         > on
> > > > >     >         > > > the
> > > > >     >         > > > > > >>> core
> > > > >     >         > > > > > >>> >> feature
> > > > >     >         > > > > > >>> >> > > > set
> > > > >     >         > > > > > >>> >> > > > >>> > > > rather than
> > performance
> > > > >     > optimizations for
> > > > >     >         > now.
> > > > >     >         > > > If
> > > > >     >         > > > > we
> > > > >     >         > > > > > >>> can
> > > > >     >         > > > > > >>> >> agree
> > > > >     >         > > > > > >>> >> > > on
> > > > >     >         > > > > > >>> >> > > > >>> what
> > > > >     >         > > > > > >>> >> > > > >>> > > > MM2 looks like, it
> > will
> > > be
> > > > > very
> > > > >     > easy to
> > > > >     >         > agree
> > > > >     >         > > to
> > > > >     >         > > > > > >>> improve
> > > > >     >         > > > > > >>> >> its
> > > > >     >         > > > > > >>> >> > > > >>> performance
> > > > >     >         > > > > > >>> >> > > > >>> > > > and reliability.
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > > That said, I look
> > > forward
> > > > > to your
> > > > >     > support
> > > > >     >         > on a
> > > > >     >         > > > > > >>> subsequent
> > > > >     >         > > > > > >>> >> KIP
> > > > >     >         > > > > > >>> >> > > > that
> > > > >     >         > > > > > >>> >> > > > >>> > > > addresses consumer
> > > > > coordination
> > > > >     > and
> > > > >     >         > rebalance
> > > > >     >         > > > > > issues.
> > > > >     >         > > > > > >>> Stay
> > > > >     >         > > > > > >>> >> > > tuned!
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > > Ryanne
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > > On Tue, Oct 16, 2018
> > at
> > > > > 6:58 AM
> > > > >     > Jan
> > > > >     >         > Filipiak <
> > > > >     >         > > > > > >>> >> > > > >>> Jan.Filipiak@trivago.com
> > > > >     >         > > > > > >>> >> > > > >>> > > > <mailto:
> > > > > Jan.Filipiak@trivago.com>>
> > > > >     > wrote:
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     Hi,
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     Currently
> > > MirrorMaker
> > > > is
> > > > >     > usually run
> > > > >     >         > > > > collocated
> > > > >     >         > > > > > >>> with
> > > > >     >         > > > > > >>> >> the
> > > > >     >         > > > > > >>> >> > > > target
> > > > >     >         > > > > > >>> >> > > > >>> > > >     cluster.
> > > > >     >         > > > > > >>> >> > > > >>> > > >     This is all nice
> > and
> > > > > good.
> > > > >     > But one big
> > > > >     >         > > > > obstacle
> > > > >     >         > > > > > in
> > > > >     >         > > > > > >>> >> this was
> > > > >     >         > > > > > >>> >> > > > >>> > > >     always that
> group
> > > > >     > coordination happened
> > > > >     >         > on
> > > > >     >         > > > the
> > > > >     >         > > > > > >>> source
> > > > >     >         > > > > > >>> >> > > > cluster.
> > > > >     >         > > > > > >>> >> > > > >>> So
> > > > >     >         > > > > > >>> >> > > > >>> > > when
> > > > >     >         > > > > > >>> >> > > > >>> > > >     then network was
> > > > > congested,
> > > > >     > you
> > > > >     >         > sometimes
> > > > >     >         > > > > loose
> > > > >     >         > > > > > >>> group
> > > > >     >         > > > > > >>> >> > > > >>> membership and
> > > > >     >         > > > > > >>> >> > > > >>> > > >     have to
> rebalance
> > > and
> > > > > all
> > > > >     > this.
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     So one big
> request
> > > > from
> > > > > we
> > > > >     > would be the
> > > > >     >         > > > > support
> > > > >     >         > > > > > of
> > > > >     >         > > > > > >>> >> having
> > > > >     >         > > > > > >>> >> > > > >>> > > coordination
> > > > >     >         > > > > > >>> >> > > > >>> > > >     cluster !=
> source
> > > > > cluster.
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     I would
> generally
> > > say
> > > > a
> > > > > LAN
> > > > >     > is better
> > > > >     >         > > than a
> > > > >     >         > > > > WAN
> > > > >     >         > > > > > >>> for
> > > > >     >         > > > > > >>> >> doing
> > > > >     >         > > > > > >>> >> > > > >>> group
> > > > >     >         > > > > > >>> >> > > > >>> > > >     coordinaton and
> > > there
> > > > > is no
> > > > >     > reason we
> > > > >     >         > > > couldn't
> > > > >     >         > > > > > >>> have a
> > > > >     >         > > > > > >>> >> group
> > > > >     >         > > > > > >>> >> > > > >>> consuming
> > > > >     >         > > > > > >>> >> > > > >>> > > >     topics from a
> > > > different
> > > > >     > cluster and
> > > > >     >         > > > committing
> > > > >     >         > > > > > >>> >> offsets to
> > > > >     >         > > > > > >>> >> > > > >>> another
> > > > >     >         > > > > > >>> >> > > > >>> > > >     one right?
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     Other than that.
> > It
> > > > > feels
> > > > >     > like the KIP
> > > > >     >         > has
> > > > >     >         > > > too
> > > > >     >         > > > > > >>> much
> > > > >     >         > > > > > >>> >> > > features
> > > > >     >         > > > > > >>> >> > > > >>> where
> > > > >     >         > > > > > >>> >> > > > >>> > > many
> > > > >     >         > > > > > >>> >> > > > >>> > > >     of them are not
> > > really
> > > > > wanted
> > > > >     > and
> > > > >     >         > counter
> > > > >     >         > > > > > >>> productive
> > > > >     >         > > > > > >>> >> but I
> > > > >     >         > > > > > >>> >> > > > >>> will just
> > > > >     >         > > > > > >>> >> > > > >>> > > >     wait and see how
> > the
> > > > >     > discussion goes.
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     Best Jan
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > > >     On 15.10.2018
> > 18:16,
> > > > > Ryanne
> > > > >     > Dolan wrote:
> > > > >     >         > > > > > >>> >> > > > >>> > > >      > Hey y'all!
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >      > Please take a
> > > look
> > > > at
> > > > >     > KIP-382:
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>>
> > > > >     >         > > > > > >>> >> > > >
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > >
> > > > >     >         > > > >
> > > > >     >         > > >
> > > > >     >         > >
> > > > >     >         >
> > > > >     >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >      > Thanks for
> your
> > > > > feedback
> > > > >     > and support.
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >      > Ryanne
> > > > >     >         > > > > > >>> >> > > > >>> > > >      >
> > > > >     >         > > > > > >>> >> > > > >>> > > >
> > > > >     >         > > > > > >>> >> > > > >>> > >
> > > > >     >         > > > > > >>> >> > > > >>>
> > > > >     >         > > > > > >>> >> > > > >>
> > > > >     >         > > > > > >>> >> > > >
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> > > --
> > > > >     >         > > > > > >>> >> > > Best,
> > > > >     >         > > > > > >>> >> > > Alex Mironov
> > > > >     >         > > > > > >>> >> > >
> > > > >     >         > > > > > >>> >> >
> > > > >     >         > > > > > >>> >>
> > > > >     >         > > > > > >>> >
> > > > >     >         > > > > > >>>
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >> --
> > > > >     >         > > > > > >> Sönke Liebau
> > > > >     >         > > > > > >> Partner
> > > > >     >         > > > > > >> Tel. +49 179 7940878
> > > > >     >         > > > > > >> OpenCore GmbH & Co. KG -
> > Thomas-Mann-Straße
> > > 8
> > > > -
> > > > > 22880
> > > > >     > Wedel -
> > > > >     >         > > > Germany
> > > > >     >         > > > > > >>
> > > > >     >         > > > > > >
> > > > >     >         > > > > >
> > > > >     >         > > > > > --
> > > > >     >         > > > > > Sönke Liebau
> > > > >     >         > > > > > Partner
> > > > >     >         > > > > > Tel. +49 179 7940878
> > > > >     >         > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße
> > 8 -
> > > > > 22880
> > > > >     > Wedel -
> > > > >     >         > Germany
> > > > >     >         > > > > >
> > > > >     >         > > > >
> > > > >     >         > > >
> > > > >     >         > > >
> > > > >     >         > > > --
> > > > >     >         > > > Sönke Liebau
> > > > >     >         > > > Partner
> > > > >     >         > > > Tel. +49 179 7940878
> > > > >     >         > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 -
> > > 22880
> > > > > Wedel
> > > > >     > - Germany
> > > > >     >         > > >
> > > > >     >         > >
> > > > >     >         >
> > > > >     >         >
> > > > >     >         > --
> > > > >     >         > Sönke Liebau
> > > > >     >         > Partner
> > > > >     >         > Tel. +49 179 7940878
> > > > >     >         > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 -
> 22880
> > > > > Wedel -
> > > > >     > Germany
> > > > >     >         >
> > > > >     >
> > > > >     >
> > > > >     >
> > > > >     >
> > > > >     > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > for
> > > > >     > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > > are not
> > > > >     > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > > > others
> > > > >     > this message or any attachment. Please also notify the sender
> > by
> > > > > replying
> > > > >     > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > > the email
> > > > >     > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > > to the
> > > > >     > official business of this company shall be understood as
> > neither
> > > > > given nor
> > > > >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > company
> > > > >     > registered in England and Wales, company number 04008957) and
> > IG
> > > > > Index
> > > > >     > Limited (a company registered in England and Wales, company
> > > number
> > > > >     > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > Hill,
> > > > >     > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > > and IG
> > > > >     > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > > by the
> > > > >     > Financial Conduct Authority.
> > > > >     >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message