Return-Path: Delivered-To: apmail-hadoop-zookeeper-user-archive@minotaur.apache.org Received: (qmail 52978 invoked from network); 27 Apr 2010 23:18:33 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 Apr 2010 23:18:33 -0000 Received: (qmail 23539 invoked by uid 500); 27 Apr 2010 23:18:33 -0000 Delivered-To: apmail-hadoop-zookeeper-user-archive@hadoop.apache.org Received: (qmail 23509 invoked by uid 500); 27 Apr 2010 23:18:33 -0000 Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: zookeeper-user@hadoop.apache.org Delivered-To: mailing list zookeeper-user@hadoop.apache.org Received: (qmail 23501 invoked by uid 99); 27 Apr 2010 23:18:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:18:33 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of phunt@apache.org) Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:18:25 +0000 Received: from [10.73.135.249] ([10.73.135.249]) by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3RNH9Tg027036; Tue, 27 Apr 2010 16:17:09 -0700 (PDT) Message-ID: <4BD77075.7070603@apache.org> Date: Tue, 27 Apr 2010 16:17:09 -0700 From: Patrick Hunt User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: zookeeper-user@hadoop.apache.org CC: Satish Bhatti Subject: Re: Bizarre ZooKeeper Client Behaviour References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Are you sure you're closing the ZooKeeper object before creating a new one? I see names like: "task-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread" this most likely means you are creating ZK sessions in your watcher. This is fine, but you just need to make sure you close the old one. Perhaps some corner case in your code got triggered and it started creating ZK sessions recursively w/o bound? By default the server will only allow 10 sessions to be active from a particular client (ip), perhaps you hit this and it triggered the storm? Patrick On 04/27/2010 03:52 PM, Satish Bhatti wrote: > One of our processes that uses ZooKeeper was behaving oddly so I ran a > jstack on it. It is attached to this email. Note the "EventThread" > lines. Over 6000 of them. Has it _really_ spun up 6000 event threads? > > Satish >