kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Brugidou <maxime.brugi...@gmail.com>
Subject Re: Kafka 0.8 without replication - handle broker failure/availability
Date Wed, 09 Jan 2013 16:43:32 GMT
Thanks for your response. I think the work-around is not really acceptable
for me since it will consume 3x the resources (because replication of 3 is
the minimum acceptable) and it will still make the cluster less available
anyway (unless i have only 3 brokers).

The thing is that 0.7 was making the cluster 100% available (for my use
case, accepting data loss) as long a single broker was alive.

A way to handle this would be to:
1. Have a lot of partitions per topic (more than the # of brokers)
2. Have something that rebalances the partitions and make sure a broker has
a at least a partition for each topic (to make every topic "available")
3. Have a setting in the consumer/producer that say "I don't care about
partitioning, just produce/consume wherever you can"

This is probably not simple to implement, I'll add these ideas in the JIRA
and will pursue the discussion there.

Maxime

On Wed, Jan 9, 2013 at 5:18 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> As a work around in the meantime you can probably run with
> replication--although it sounds like you don't really need it, it shouldn't
> hurt.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message