kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions
Date Thu, 16 Jan 2020 14:03:11 GMT
MIckael,



On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mickael.maison@gmail.com>
wrote:

> Hi Manna,
>
> In your example, the topic 'dummy' is not under replicated. It just
> has 1 replica. A topic under replicated is a topic with less ISRs than
> replicas.
>
> Having under replicated topics is relatively common in a Kafka
> cluster, it happens everytime is broker is down. However Kafka does
> not permit it to happen at topic creation. Currently at creation,
> Kafka requires to have at least as many brokers as the replication
> factor. This KIP addresses this limitation.
>
> Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> to distribute partitions across racks. When all brokers in a rack are
> down (ie: a zone is offline), you can end up with partitions not well
> distributed even with rack awareness. There are currently no easy way
> to track such partitions so I decided to not attempt addressing this
> issue in this KIP.
>
> I hope that answers your questions.
>

 It does and I appreciate you taking time and explaining this.

 +1 (binding) if I haven't already.

>
>
>
> On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> <kamal.chandraprakash@gmail.com> wrote:
> >
> > +1 (non-binding). Thanks for the KIP!
> >
> > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <manmedia@gmail.com> wrote:
> >
> > > Hi Mikael,
> > >
> > > Apologies for last minute question, as I just caught up with it.
> Thanks for
> > > your work on the KIP.
> > >
> > > Just trying to get your thoughts on one thing (I might have
> misunderstood
> > > it) - currently it's possible (even though I am strongly against it) to
> > > create Kafka topics which are under-replicated; despite all brokers
> being
> > > online. This the the output of an intentionally under-replicated topic
> > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > >
> > >
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --create --topic dummy --partitions 6 --replication-factor 1
> > > --bootstrap-server localhost:9092
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > Topic:dummy     PartitionCount:6        ReplicationFactor:1
> > >
> > >
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > >         Topic: dummy    Partition: 0    Leader: 3       Replicas: 3
> > > Isr: 3
> > >         Topic: dummy    Partition: 1    Leader: 1       Replicas: 1
> > > Isr: 1
> > >         Topic: dummy    Partition: 2    Leader: 2       Replicas: 2
> > > Isr: 2
> > >         Topic: dummy    Partition: 3    Leader: 3       Replicas: 3
> > > Isr: 3
> > >         Topic: dummy    Partition: 4    Leader: 1       Replicas: 1
> > > Isr: 1
> > >         Topic: dummy    Partition: 5    Leader: 2       Replicas: 2
> > > Isr: 2
> > >
> > >  This is with respect to the following statement on your KIP (i.e.
> > > under-replicated topic creation is also permitted when none is
> offline):
> > >
> > > *but note that this may already happen (without this KIP) when
> > > > topics/partitions are created while all brokers in a rack are offline
> > > (ie:
> > > > an availability zone is offline). Tracking topics/partitions not
> > > optimally
> > > > spread across all racks can be tackled in a follow up KIP.  *
> > >
> > >
> > >
> > >
> > > Did you mean to say that such under-replicated topics (including
> > > human-created ones) will be handled in a separete KIP ?
> > >
> > > Regards,
> > >
> > >
> > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison <mickael.maison@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all.
> > > >
> > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > votes would be nice.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley <tbentley@redhat.com>
> wrote:
> > > > >
> > > > > +1 non-binding. Thanks!
> > > > >
> > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > <soenke.liebau@opencore.com.invalid> wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > thanks for your response! That all makes perfect sense and I
> cannot
> > > > > > give any actual use cases for where what I asked about would
be
> > > useful
> > > > > > :)
> > > > > > It was more the idle thought if this might be low hanging fruit
> while
> > > > > > changing this anyway to avoid having to circle back later on
and
> > > > > > wanted to at least mention it.
> > > > > >
> > > > > > I am totally happy either way!
> > > > > >
> > > > > > Best regards,
> > > > > > Sönke
> > > > > >
> > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> > > mickael.maison@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Thanks Sönke for the feedback.
> > > > > > >
> > > > > > > I debated this point quite a bit before deciding to base
> creation
> > > > > > > around "min.insync.replicas".
> > > > > > >
> > > > > > > For me, the goal of this KIP is to enable administrators
to
> provide
> > > > > > > higher availability. In a 3 node cluster configured for
high
> > > > > > > availability (3 replicas, 2 min ISR), by enabling this
feature,
> > > > > > > clusters should be fully usable even when 1 broker is down.
> This
> > > > > > > should cover all "normal" maintenance operations like a
rolling
> > > > > > > restart or just the recovery of a broker.
> > > > > > >
> > > > > > > At the moment, when creating a topic/partition, the assumption
> is
> > > > that
> > > > > > > the resource will be fully functioning. This KIP does not
> change
> > > this
> > > > > > > assumption. If this is something someone wants, I think
it
> should
> > > be
> > > > > > > handled in a different KIP that targets that use case.
By
> relying
> > > on
> > > > > > > "min.insync.replicas", we don't break any assumptions the
user
> has
> > > > and
> > > > > > > this should be fully transparent from the user point of
view.
> > > > > > >
> > > > > > > About "min.insync.replicas", one caveat that is not explicit
> in the
> > > > > > > KIP is that it's currently possible to create topics with
less
> > > > > > > replicas than this settings. For that reason, I think the
> > > > > > > implementation will actually rely on min(replicas, min-isr)
> instead
> > > > of
> > > > > > > simply min.insync.replicas. I have updated the KIP to
> explicitly
> > > > > > > mention this point.
> > > > > > >
> > > > > > > I hope that answers your question, let me know.
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > > > > <soenke.liebau@opencore.com.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Michael,
> > > > > > > >
> > > > > > > > that sounds like a useful addition! I can't help but
wonder
> > > > whether by
> > > > > > > > leaving in the restriction that "min.insync.replicas"
has to
> be
> > > > > > > > satisfied we'll be back here in a years time because
someone
> has
> > > a
> > > > > > > > scenario where he or she wants to go below that :)
> > > > > > > > I don't have a strong opinion either way to be honest,
just a
> > > > random
> > > > > > > > thought when reading the KIP.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Sönke
> > > > > > > >
> > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan <
> > > ryannedolan@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 non-binding, thx
> > > > > > > > >
> > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison
<
> > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Bumping this thread, I've not seen any votes
or feedback.
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael
Maison
> > > > > > > > > > <mickael.maison@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start a vote on KIP-409:
Allow creating
> > > > > > under-replicated
> > > > > > > > > > > topics and partitions
> > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sönke Liebau
> > > > > > > > Partner
> > > > > > > > Tel. +49 179 7940878
> > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8
- 22880 Wedel -
> > > > Germany
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sönke Liebau
> > > > > > Partner
> > > > > > Tel. +49 179 7940878
> > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel
-
> Germany
> > > > > >
> > > > > >
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message