Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of traviscrawford@gmail.com
 designates 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WEJMFb488dOKMrQgrXZMol5p+HzwZ2eHeeIb0NphTeu9UxPLCjNZlbxuVBnvz1nwiB
         bf6cc+MokxjcjizHAmWU0yaqr03dceXCfCyrMlikcXPTOlvTPP2eWeEmkJmCtNbAgCWu
         E1iUvxn6S1wkuEwpb6G7JhpU92lr9oTP3Btd0=
MIME-Version: 1.0
Date: Wed, 30 Jun 2010 23:13:53 -0700
Message-ID: <AANLkTinJnl9B7LSnawQGm3MXrnu3B0ooxlTg6XxrXqJS@mail.gmail.com>
Subject: Zookeeper outage recap & questions
From: Travis Crawford <traviscrawford@gmail.com>
To: zookeeper-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey zookeepers -

We just experienced a total zookeeper outage, and here's a quick
post-mortem of the issue, and some questions about preventing it going
forward. Quick overview of the setup:

- RHEL5 2.6.18 kernel
- Zookeeper 3.3.0
- ulimit raised to 65k files
- 3 cluster members
- 4-5k connections in steady-state
- Primarily C and python clients, plus some java

In chronological order, the issue manifested itself as alert about RW
tests failing. Logs were full of too many files errors, and the output
of netstat showed lots of CLOSE_WAIT and SYN_RECV sockets. CPU was
100%. Application logs showed lots of connection timeouts. This
suggests an event happened that caused applications to dogpile on
Zookeeper, and eventually the CLOSE_WAIT timeout caused file handles
to run out and basically game over.

I looked through lots of logs (clients+servers) and did not see a
clear indication of what happened. Graphs show a sudden decrease in
network traffic when the outage began, zookeeper goes cpu bound, and
runs our of file descriptors.

Clients are primarily a couple thousand C clients using default
connection parameters, and a couple thousand python clients using
default connection parameters.

Digging through Jira we see two issues that probably contributed to this outage:

    https://issues.apache.org/jira/browse/ZOOKEEPER-662
    https://issues.apache.org/jira/browse/ZOOKEEPER-517

Both are tagged for the 3.4.0 release. Anyone know if that's still the
case, and when 3.4.0 is roughly scheduled to ship?

Thanks!
Travis