kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mickael Maison <mickael.mai...@gmail.com>
Subject Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions
Date Thu, 16 Jan 2020 14:00:57 GMT
Hi Manna,

In your example, the topic 'dummy' is not under replicated. It just
has 1 replica. A topic under replicated is a topic with less ISRs than
replicas.

Having under replicated topics is relatively common in a Kafka
cluster, it happens everytime is broker is down. However Kafka does
not permit it to happen at topic creation. Currently at creation,
Kafka requires to have at least as many brokers as the replication
factor. This KIP addresses this limitation.

Regarding your 2nd point. When rack awareness is enabled, Kafka tries
to distribute partitions across racks. When all brokers in a rack are
down (ie: a zone is offline), you can end up with partitions not well
distributed even with rack awareness. There are currently no easy way
to track such partitions so I decided to not attempt addressing this
issue in this KIP.

I hope that answers your questions.



On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
<kamal.chandraprakash@gmail.com> wrote:
>
> +1 (non-binding). Thanks for the KIP!
>
> On Mon, Jan 13, 2020 at 1:58 PM M. Manna <manmedia@gmail.com> wrote:
>
> > Hi Mikael,
> >
> > Apologies for last minute question, as I just caught up with it. Thanks for
> > your work on the KIP.
> >
> > Just trying to get your thoughts on one thing (I might have misunderstood
> > it) - currently it's possible (even though I am strongly against it) to
> > create Kafka topics which are under-replicated; despite all brokers being
> > online. This the the output of an intentionally under-replicated topic
> > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> >
> >
> > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > --create --topic dummy --partitions 6 --replication-factor 1
> > --bootstrap-server localhost:9092
> > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > --describe --topic dummy  --bootstrap-server localhost:9092
> > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> >
> > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > Isr: 3
> >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > Isr: 1
> >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > Isr: 2
> >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > Isr: 3
> >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > Isr: 1
> >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > Isr: 2
> >
> >  This is with respect to the following statement on your KIP (i.e.
> > under-replicated topic creation is also permitted when none is offline):
> >
> > *but note that this may already happen (without this KIP) when
> > > topics/partitions are created while all brokers in a rack are offline
> > (ie:
> > > an availability zone is offline). Tracking topics/partitions not
> > optimally
> > > spread across all racks can be tackled in a follow up KIP.  *
> >
> >
> >
> >
> > Did you mean to say that such under-replicated topics (including
> > human-created ones) will be handled in a separete KIP ?
> >
> > Regards,
> >
> >
> > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mickael.maison@gmail.com>
> > wrote:
> >
> > > Hi all.
> > >
> > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > votes would be nice.
> > >
> > > Thanks
> > >
> > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tbentley@redhat.com> wrote:
> > > >
> > > > +1 non-binding. Thanks!
> > > >
> > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > <soenke.liebau@opencore.com.invalid> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > thanks for your response! That all makes perfect sense and I cannot
> > > > > give any actual use cases for where what I asked about would be
> > useful
> > > > > :)
> > > > > It was more the idle thought if this might be low hanging fruit while
> > > > > changing this anyway to avoid having to circle back later on and
> > > > > wanted to at least mention it.
> > > > >
> > > > > I am totally happy either way!
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > mickael.maison@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks Sönke for the feedback.
> > > > > >
> > > > > > I debated this point quite a bit before deciding to base creation
> > > > > > around "min.insync.replicas".
> > > > > >
> > > > > > For me, the goal of this KIP is to enable administrators to
provide
> > > > > > higher availability. In a 3 node cluster configured for high
> > > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > > clusters should be fully usable even when 1 broker is down.
This
> > > > > > should cover all "normal" maintenance operations like a rolling
> > > > > > restart or just the recovery of a broker.
> > > > > >
> > > > > > At the moment, when creating a topic/partition, the assumption
is
> > > that
> > > > > > the resource will be fully functioning. This KIP does not change
> > this
> > > > > > assumption. If this is something someone wants, I think it should
> > be
> > > > > > handled in a different KIP that targets that use case. By relying
> > on
> > > > > > "min.insync.replicas", we don't break any assumptions the user
has
> > > and
> > > > > > this should be fully transparent from the user point of view.
> > > > > >
> > > > > > About "min.insync.replicas", one caveat that is not explicit
in the
> > > > > > KIP is that it's currently possible to create topics with less
> > > > > > replicas than this settings. For that reason, I think the
> > > > > > implementation will actually rely on min(replicas, min-isr)
instead
> > > of
> > > > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > > > mention this point.
> > > > > >
> > > > > > I hope that answers your question, let me know.
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > <soenke.liebau@opencore.com.invalid> wrote:
> > > > > > >
> > > > > > > Hi Michael,
> > > > > > >
> > > > > > > that sounds like a useful addition! I can't help but wonder
> > > whether by
> > > > > > > leaving in the restriction that "min.insync.replicas" has
to be
> > > > > > > satisfied we'll be back here in a years time because someone
has
> > a
> > > > > > > scenario where he or she wants to go below that :)
> > > > > > > I don't have a strong opinion either way to be honest,
just a
> > > random
> > > > > > > thought when reading the KIP.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Sönke
> > > > > > >
> > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > +1 non-binding, thx
> > > > > > > >
> > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Bumping this thread, I've not seen any votes
or feedback.
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison
> > > > > > > > > <mickael.maison@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to start a vote on KIP-409: Allow
creating
> > > > > under-replicated
> > > > > > > > > > topics and partitions
> > > > > > > > > >
> > > > > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sönke Liebau
> > > > > > > Partner
> > > > > > > Tel. +49 179 7940878
> > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
Wedel -
> > > Germany
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel
- Germany
> > > > >
> > > > >
> > >
> >

Mime
View raw message