kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Brugidou <maxime.brugi...@gmail.com>
Subject Re: Kafka 0.8 without replication - handle broker failure/availability
Date Wed, 09 Jan 2013 16:43:32 GMT
Thanks for your response. I think the work-around is not really acceptable
for me since it will consume 3x the resources (because replication of 3 is
the minimum acceptable) and it will still make the cluster less available
anyway (unless i have only 3 brokers).

The thing is that 0.7 was making the cluster 100% available (for my use
case, accepting data loss) as long a single broker was alive.

A way to handle this would be to:
1. Have a lot of partitions per topic (more than the # of brokers)
2. Have something that rebalances the partitions and make sure a broker has
a at least a partition for each topic (to make every topic "available")
3. Have a setting in the consumer/producer that say "I don't care about
partitioning, just produce/consume wherever you can"

This is probably not simple to implement, I'll add these ideas in the JIRA
and will pursue the discussion there.


On Wed, Jan 9, 2013 at 5:18 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> As a work around in the meantime you can probably run with
> replication--although it sounds like you don't really need it, it shouldn't
> hurt.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message