kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Juarez <mjua...@gmail.com>
Subject Consumer Offsets partition skew on Kafka 0.10.1.1
Date Tue, 03 Oct 2017 00:55:25 GMT
I was investigating some performance issues we're issues in one of our
production clusters, and I ran into extremely unbalanced offset partitions
for the __consumer_offsets topic.  I only pasted the top 8 below, out of 50
total.  As you can see, between the top 5 partitions, those servers have to
handle 83% of the commit volume, and brokers 9 and 10 show up repeatedly on
as leader as well as replicas.


Partition Offsets Percentage Leader Replicas ISR
6 52,761,610,477 34.24% 10 (10,6,7) (7,6,10)
5 46,196,021,230 29.98% 9 (9,5,6) (5,6,9)
42 17,530,298,423 11.38% 10 (10,9,11) (10,11,9)
31 12,927,081,106 8.39% 11 (11,9,10) (10,11,9)
0 8,557,903,671 5.55% 4 (4,12,1) (4,12,1)
2 3,969,232,652 2.58% 6 (6,2,3) (6,3,2)
49 3,555,754,347 2.31% 5 (5,11,7) (5,7,11)
33 2,273,951,745 1.48% 1 (1,11,12) (1,12,11)
Those brokers (9, 10 and 11) also happen to be the ones we're having
performance issues with.  We can't be sure yet if this is the cause of the
performance issues, but it's looking extremely likely.

So, I was wondering, what can be done to "rebalance" these consumer
offsets?  This was something, as far as I know, automatically decided, I
don't believe we ever changed a setting related to this.  I also don't
believe we can influence which partition gets which offsets when
consuming.

It would also be interesting to know what is the algorithm/pattern used to
decide the consumer offset partition, and is this something we can change
or influence?

Thanks,

Marcos Juarez

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message