Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
From: Marcos Juarez <mjuarez@gmail.com>
Date: Mon, 2 Oct 2017 18:55:25 -0600
Message-ID: <CALaekbxBp+EVnO-0qGvN5w4YR1+Zb=ME4eg0rXtnLpj2QFEtRg@mail.gmail.com>
Subject: Consumer Offsets partition skew on Kafka 0.10.1.1
To: dev@kafka.apache.org
Content-Type: multipart/alternative; boundary="f403043885e85d2879055a99f2ae"
archived-at: Tue, 03 Oct 2017 00:55:32 -0000

--f403043885e85d2879055a99f2ae
Content-Type: text/plain; charset="UTF-8"

I was investigating some performance issues we're issues in one of our
production clusters, and I ran into extremely unbalanced offset partitions
for the __consumer_offsets topic.  I only pasted the top 8 below, out of 50
total.  As you can see, between the top 5 partitions, those servers have to
handle 83% of the commit volume, and brokers 9 and 10 show up repeatedly on
as leader as well as replicas.


Partition Offsets Percentage Leader Replicas ISR
6 52,761,610,477 34.24% 10 (10,6,7) (7,6,10)
5 46,196,021,230 29.98% 9 (9,5,6) (5,6,9)
42 17,530,298,423 11.38% 10 (10,9,11) (10,11,9)
31 12,927,081,106 8.39% 11 (11,9,10) (10,11,9)
0 8,557,903,671 5.55% 4 (4,12,1) (4,12,1)
2 3,969,232,652 2.58% 6 (6,2,3) (6,3,2)
49 3,555,754,347 2.31% 5 (5,11,7) (5,7,11)
33 2,273,951,745 1.48% 1 (1,11,12) (1,12,11)
Those brokers (9, 10 and 11) also happen to be the ones we're having
performance issues with.  We can't be sure yet if this is the cause of the
performance issues, but it's looking extremely likely.

So, I was wondering, what can be done to "rebalance" these consumer
offsets?  This was something, as far as I know, automatically decided, I
don't believe we ever changed a setting related to this.  I also don't
believe we can influence which partition gets which offsets when
consuming.

It would also be interesting to know what is the algorithm/pattern used to
decide the consumer offset partition, and is this something we can change
or influence?

Thanks,

Marcos Juarez

--f403043885e85d2879055a99f2ae--