kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Ozeritsky <aozerit...@yandex.ru>
Subject Re: Prevent broker from leading topic partitions
Date Mon, 22 Aug 2016 18:38:10 GMT
Hi

22.08.2016, 19:16, "Tom Crayford" <tcrayford@heroku.com>:
> Hi,
>
> I don't think I understand *why* you need this. Kafka is by default a
> distributed HA system that balances data and leadership over nodes. Why do
> you need to change this?

In fact, kafka does not balance anything. It uses static partition distribution.
In most cases the first replica in replica list is always a leader.

If a replica is under heavy load (for example hard disk was replaced or so on) it is a bad
idea to do automatic leader "rebalance".

>
> You could accomplish something like this with mirror maker, that may make
> more sense.
>
> Thanks
>
> Tom Crayford
>
> Heroku Kafka
>
> On Mon, Aug 22, 2016 at 4:05 PM, Jason Aliyetti <j.aliyetti@gmail.com>
> wrote:
>
>>  I have a use case that requires a 2 node deployment with a Kafka-backed
>>  service with the following constraints:
>>
>>  - All data must be persisted to node 1. If node 1 fails (regardless of the
>>  status of node 2), then the system must stop.
>>  - If node 2 is up, then it must stay in synch with node 1.
>>  - If node 2 fails, then service must not be disrupted, but as soon as it
>>  comes back up and rejoins ISR it must stay in synch.
>>
>>  The deployment is basically a primary node and a cold node with real time
>>  replication, but no failover to the cold node.
>>
>>  To achieve this I am considering adding a broker-level configuration option
>>  that would prevent a broker from becoming a leader for any topic partition
>>  it hosts - this would allow me to enforce that the cold node never take
>>  leadership for any topics. In conjunction with manipulating a topic's
>>  "min.insync.replicas" setting at runtime, I should be able to achieve the
>>  behavior desired (2 if both brokers up, 1 if the standby goes down).
>>
>>  I know this sounds like an edgy use case, but does this sound like a
>>  reasonable approach? Are there any valid use cases around such a broker or
>>  topic level configuration (i.e. does this sound like a feature that would
>>  make sense to open a KIP against)?

Mime
View raw message