Mailing-List: contact users-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@kafka.apache.org
Received-SPF: pass (nike.apache.org: domain of manish@ooyala.com designates
 209.85.217.181 as permitted sender)
MIME-Version: 1.0
Date: Tue, 5 Feb 2013 17:41:02 -0800
Message-ID: 
 <CAAVeZCGMvNznSQE15qxjJN6MDnMfP9kiTkE678_uf_wtUdO7uA@mail.gmail.com>
Subject: Consumers constantly rebalancing
From: Manish Khettry <manish@ooyala.com>
To: kafka-users@incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8f22beb938e9d304d5046be5

--e89a8f22beb938e9d304d5046be5
Content-Type: text/plain; charset=ISO-8859-1

We are trying to trouble shoot a problem wherein our system just cannot
seem to read messages fast enough from Kafka. We are on kafka 0.6 and are
using the simple consumer.

>From looking at the logs, and we see a lot (almost constant chatty
messages) about rebalancing. So for instance every minute, we see messages
like this:


Consumer rookery-vacuum-prod_<first_ip>.internal-1360106018385
rebalancing the following partitions: List(0-0, 0-1, 0-10, 0-11, 0-12,
0-13, 0-14, 0-15, 0-16, 0-17, 0-18, 0-19, 0-2, 0-3, 0-4, 0-5, 0-6,
0-7, 0-8, 0-9, 1-0, 1-1, 1-10, 1-11, 1-12, 1-13, 1-14, 1-15, 1-16,
1-17, 1-18, 1-19, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8, 1-9) for topic
compact-player-logs with consumers:


I also see zookeeper timeouts like so:

Unable to reconnect to ZooKeeper service, session 0x33c981ab95100ed
has expired, closing socket connection


We increased the zookeeper session timeout from 6 seconds to 12 seconds and
this seems to have helped somewhat but I'm not sure  if these zookeeper
timeouts at 6 seconds are symptomatic of a problem with our zookeeper
cluster and/or connectivity between the consumers and zk. Any thoughts?

Manish

--e89a8f22beb938e9d304d5046be5--