Return-Path: Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: (qmail 29424 invoked from network); 5 Jan 2011 02:47:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Jan 2011 02:47:51 -0000 Received: (qmail 86744 invoked by uid 500); 5 Jan 2011 02:47:51 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 86560 invoked by uid 500); 5 Jan 2011 02:47:49 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 86546 invoked by uid 99); 5 Jan 2011 02:47:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 02:47:48 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.41 as permitted sender) Received: from [209.85.161.41] (HELO mail-fx0-f41.google.com) (209.85.161.41) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 02:47:43 +0000 Received: by fxm12 with SMTP id 12so8420751fxm.14 for ; Tue, 04 Jan 2011 18:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=MJJ0uhpRAqFySLmsf6DAeeXTzKhnjGI5AQvHbq9E7io=; b=MVgMmWlkDssvqK/aBDhTlTK7iBDmV9Mi8DZBw6Kj4mW6+ye7P+xjXzohJJdrXlgxee 5RHVjXkEMzDzDimfqkHw+zaK4ppAV/ydvvmQBNdgEiQ00/fTsoELe/LTnFV+r6QEp84P yVeTFOAOnEO/oCCErgpuTJ42d9hzFYAqQvBBU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=afiDplOsZeLlzZf2o6LyARD6aMOUnzbq20hQAMSs8/BPtBeujLjhD7ZwYlCM2axIf/ 6bRyTCnNmJxkbe+9gOx598QviwVK5Wx/SLpD+RbfdxLf1GnNeyvHYrB4Y5Ka4JGJ252o Xjx+Ii9LV0iv+OsYnvPd5LDK64FouIgSxR2ws= MIME-Version: 1.0 Received: by 10.223.100.15 with SMTP id w15mr2840589fan.121.1294195641949; Tue, 04 Jan 2011 18:47:21 -0800 (PST) Received: by 10.223.83.200 with HTTP; Tue, 4 Jan 2011 18:47:21 -0800 (PST) In-Reply-To: References: Date: Tue, 4 Jan 2011 18:47:21 -0800 Message-ID: Subject: Re: consistent KeeperException$ConnectionLossException From: Ted Yu To: dev@hbase.apache.org Content-Type: multipart/alternative; boundary=20cf3054a2e9821dde04991067ea --20cf3054a2e9821dde04991067ea Content-Type: text/plain; charset=ISO-8859-1 I increased max connections to 40. I still got: 2011-01-04 21:30:05,701 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.79 - max is 40 2011-01-04 21:30:06,072 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.80 - max is 40 2011-01-04 21:30:06,458 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.80 - max is 40 2011-01-04 21:30:06,944 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.79 - max is 40 2011-01-04 21:30:07,273 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.80 - max is 40 2011-01-04 21:30:07,665 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.202.50.79 - max is 40 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x12d52be9c2b001b, likely client has closed socket 2011-01-04 21:30:07,876 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.202.50.79:43150 which had sessionid 0x12d52be9c2b001b 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x12d52be9c2b008b, likely client has closed socket 2011-01-04 21:30:07,876 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.202.50.79:26104 which had sessionid 0x12d52be9c2b008b 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x12d52be9c2b010c, likely client has closed socket I verified maxClientCnxns of 30 in 0.20.6 where we didn't experience this problem. More comment is welcome. On Tue, Jan 4, 2011 at 11:47 AM, Ted Yu wrote: > So I should be using HTablePool. > For 0.20.6, I didn't see ConnectionLossException this often. > > I wonder if something changed from 0.20.6 to 0.90 > > On Tue, Jan 4, 2011 at 11:29 AM, Stack wrote: > >> Are you passing the same Configuration instance when creating your >> HTables? See >> http://people.apache.org/~stack/hbase-0.90.0-candidate-2/docs/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html >> if not. It explains how we figure whether zk client, rpc connections, >> etc. are shared or not. >> >> St.Ack >> >> On Tue, Jan 4, 2011 at 11:12 AM, Jean-Daniel Cryans >> wrote: >> > It's a zookeeper setting, you cannot have by default more than 30 >> > connections from the same IP per ZK peer. >> > >> > If HBase is starting ZK for you, do change >> > hbase.zookeeper.property.maxClientCnxns >> > >> > J-D >> > >> > On Tue, Jan 4, 2011 at 11:09 AM, Ted Yu wrote: >> >> Hi, >> >> I am using HBase 0.90 and our job fails consistently with the following >> >> exception: >> >> >> >> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: >> >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >> KeeperErrorCode = ConnectionLoss for /hbase >> >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:147) >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1035) >> >> ... 19 more >> >> Caused by: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >> KeeperErrorCode = ConnectionLoss for /hbase >> >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >> >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> >> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) >> >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902) >> >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:133) >> >> ... 20 more >> >> >> >> Zookeeper quorum runs on the same node as NameNode. HMaster is on >> another >> >> node. Hadoop is cdh3b2. >> >> >> >> In zookeeper log, I see (10.202.50.79 is the same node where the >> exception >> >> above happened): >> >> >> >> 2011-01-04 18:47:40,633 WARN org.apache.zookeeper.server.NIOServerCnxn: >> Too >> >> many connections from /10.202.50.79 - max is 30 >> >> 2011-01-04 18:47:41,187 WARN org.apache.zookeeper.server.NIOServerCnxn: >> Too >> >> many connections from /10.202.50.79 - max is 30 >> >> 2011-01-04 18:47:42,375 WARN org.apache.zookeeper.server.NIOServerCnxn: >> Too >> >> many connections from /10.202.50.79 - max is 30 >> >> 2011-01-04 18:47:42,447 WARN org.apache.zookeeper.server.NIOServerCnxn: >> Too >> >> many connections from /10.202.50.79 - max is 30 >> >> 2011-01-04 18:47:43,113 WARN org.apache.zookeeper.server.NIOServerCnxn: >> >> EndOfStreamException: Unable to read additional data from client >> sessionid >> >> 0x12d5220eb970025, likely client has closed socket >> >> 2011-01-04 18:47:43,113 INFO org.apache.zookeeper.server.NIOServerCnxn: >> >> Closed socket connection for client /10.202.50.79:37845 which had >> sessionid >> >> 0x12d5220eb970025 >> >> 2011-01-04 18:47:43,113 WARN org.apache.zookeeper.server.NIOServerCnxn: >> >> EndOfStreamException: Unable to read additional data from client >> sessionid >> >> 0x12d5220eb970087, likely client has closed socket >> >> >> >> Please advise what parameter I should tune. >> >> >> >> Thanks >> >> >> > >> > > --20cf3054a2e9821dde04991067ea--