kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Jackson <jack...@twitter.com>
Subject How can I set topic partition count less than total # of brokers on 0.7.2?
Date Fri, 05 Apr 2013 20:40:08 GMT
We're setting up a new kafka cluster with many brokers. How can we set a
topic partition count to be less than total # of brokers in the cluster.
The reason for this is we figure our cluster will have more sequential IO
and and thus significantly higher performance than a setup where there's 1
partition per broker and a dozen+ topics.

I asked this question to someone on #kafka IRC, and they mentioned we could
use the Producer Partitioner class to accomplish this. Has anyone run this
in production, what is the complexity involved in this approach long term,
and/or any other approaches we could take besides splitting up our single
kafka cluster into many smaller ones.

Thanks,
Jason

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message