Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4B4CFA57.2030008@apache.org>
Date: Tue, 12 Jan 2010 14:40:23 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: zookeeper-user@hadoop.apache.org, nicholas.bailey@rackspace.com
Subject: Re: Killing a zookeeper server
References: <1263333167.308920598@192.168.2.230>
In-Reply-To: <1263333167.308920598@192.168.2.230>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

12 servers? That's alot, if you dont' mind my asking why so many? 
Typically we recommend 5 - that way you can have one down for 
maintenance and still have a failure that doesn't bring down the cluster.

The "electing a leader" is probably the restarted machine attempting to 
re-join the ensemble (it should join as a follower if you have a leader 
already elected, given that it's xid is behind the existing leader.) 
Hard to tell though without the logs.

You might also be seeing the initLimit exceeded, is the data you are 
storing in ZK large? Or perhaps network connectivity is slow?
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_clusterOptions
again the logs would give some insight on this.


Patrick

Nick Bailey wrote:
> We are running zookeeper 3.1.0
> 
> Recently we noticed the cpu usage on our machines becoming
> increasingly high and we believe the cause is
> 
> https://issues.apache.org/jira/browse/ZOOKEEPER-427
> 
> However our solution when we noticed the problem was to kill the
> zookeeper process and restart it.
> 
> After doing that though it looks like the newly restarted zookeeper
> server is continually attempting to elect a leader even though one
> already exists.
> 
> The process responses with 'imok' when asked, but the stat command
> returns 'ZooKeeperServer not running'.
> 
> I belive that killing the current leader should trigger all servers
> to do an election and solve the problem, but I'm not sure. Should
> that be the course of action in this situation?
> 
> Also we have 12 servers, but 5 are currently not running according to
> stat.  So I guess this isn't a problem unless we lose another one.
> We have plans to upgrade zookeeper to solve the cpu issue but haven't
> been able to do that yet.
> 
> Any help appreciated, Nick Bailey
>