kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephane Maarek <steph...@simplemachines.com.au>
Subject Re: Minimum Replication Factor
Date Fri, 23 Jun 2017 10:32:44 GMT
That’s the first time I see this setting, wow it was burried!
I think it makes sense to implement one to get full control. 

I wonder if it’s still not worth implementing a simple setting, or implementing a few “simple”
topic creation policies that users can just reference. I don’t see that interface being
implemented anywhere

On 23/6/17, 6:43 pm, "Edoardo Comar" <ECOMAR@uk.ibm.com> wrote:

    Hi Stephane,
    we enforce the constraint in a custom create topic policy (see '
    Edoardo Comar
    IBM Message Hub
    IBM UK Ltd, Hursley Park, SO21 2JN
    From:   Stephane Maarek <stephane@simplemachines.com.au>
    To:     "dev@kafka.apache.org" <dev@kafka.apache.org>
    Date:   23/06/2017 01:48
    Subject:        Minimum Replication Factor
    Hi all,
    Interested in getting people’s opinion on something.
    The problem I have is that some people launch streams app in our cluster 
    but forget to set a replication factor > 1. Then it’s a pain to increase 
    the topic’s RF, when we do notice some topic partitions go offline because 
    we reboot brokers. 
    I have two solutions for this, which I’m interested in hearing:
    Make the replication.factor in Kafka Streams “opiniated / smart” by 
    changing the default to a dynamic min(3, # brokers).
    Create a “minimum.replication.factor” in Kafka broker settings. If any 
    topic is trying to be created using a RF less than the min, Kafka says no 
    and doesn’t create the topic. That would ensure no topics get “miscreated” 
    in production clusters and ease the pain on both devs, devops and support.
    My preference goes towards 2). 
    Unless stated otherwise above:
    IBM United Kingdom Limited - Registered in England and Wales with number 
    Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

View raw message