Return-Path: Delivered-To: apmail-hadoop-hbase-user-archive@minotaur.apache.org Received: (qmail 6328 invoked from network); 2 Mar 2010 18:00:13 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Mar 2010 18:00:13 -0000 Received: (qmail 37120 invoked by uid 500); 2 Mar 2010 18:00:08 -0000 Delivered-To: apmail-hadoop-hbase-user-archive@hadoop.apache.org Received: (qmail 36984 invoked by uid 500); 2 Mar 2010 18:00:07 -0000 Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-user@hadoop.apache.org Delivered-To: mailing list hbase-user@hadoop.apache.org Received: (qmail 36970 invoked by uid 99); 2 Mar 2010 18:00:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 18:00:07 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 74.125.83.48 as permitted sender) Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 17:59:59 +0000 Received: by gwaa11 with SMTP id a11so224261gwa.35 for ; Tue, 02 Mar 2010 09:59:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to :content-type:content-transfer-encoding; bh=sE40PDA+wHoOdow2aZ+q51CbkHKeJArc9FLiFXkPAK0=; b=j8HArMCjEpXNQWZ+DUjA4+Q1Bla6gzNRqnmvo072o5B/LfZOSIZ+YmIEFA3dZTj9tS UEVbuYtqypdcth/CulEJ5x+xj23BmnNzCPE6wBcFMxuqFJi57qMsEmBKc7KLGjBqNrNa i1zlxVhB7BJJi1AfS/kQWZVYKcGv55cM8Hgjg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=coz5S3n+aFJ7gS7j6OPXyeDx3s3Pd9MTck2sOaUQ0vyjv4o3c0zNYwvMzdGcnyzL6b zp9ShDyd0mKgUYdCQHK3sWdRvm+czbykKLgOf79SblpHEOGX8DnnqUI3aTh36rp0DQZN jYE+2wAm7kd+BnN1pJZkZTQhaZJ4dfD1FRtPk= MIME-Version: 1.0 Sender: jdcryans@gmail.com Received: by 10.91.163.2 with SMTP id q2mr42598ago.33.1267552756152; Tue, 02 Mar 2010 09:59:16 -0800 (PST) In-Reply-To: References: Date: Tue, 2 Mar 2010 09:59:14 -0800 X-Google-Sender-Auth: 46ffbcb1b0ff8f85 Message-ID: <31a243e71003020959o283b215ctbee0c13ca153d23@mail.gmail.com> Subject: Re: fail to startup regionserver From: Jean-Daniel Cryans To: hbase-user@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org This is http://issues.apache.org/jira/browse/HBASE-1946 Fixed in 0.20.2, since 0.20.3 was released a while ago I really recommend upgrading. It's all backward compatible. J-D On Tue, Mar 2, 2010 at 3:38 AM, Zheng Lv wrote: > Hello Everyone, > =A0We added a node to our cluster, and startup the datanode, tasktracker, > regionserver, but the regionserver failed.And we noted that there was som= e > exception in hbase log as following: > > =A02010-03-02 19:18:06,565 INFO org.apache.zookeeper.ClientCnxn: Attempti= ng > connection to server cactus208/127.0.0.1:2222 > 2010-03-02 19:18:06,570 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7d95d4fe > java.net.ConnectException: Connection refused > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method= ) > =A0 =A0 =A0 =A0at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-03-02 19:18:06,572 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > =A0 =A0 =A0 =A0at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > =A0 =A0 =A0 =A0at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.ja= va:360) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-03-02 19:18:06,572 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > =A0 =A0 =A0 =A0at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > =A0 =A0 =A0 =A0at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.j= ava:368) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-03-02 19:18:06,689 WARN > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set watcher= on > ZNode /hbase/master > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode =3D ConnectionLoss for /hbase/master > =A0 =A0 =A0 =A0at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > =A0 =A0 =A0 =A0at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > =A0 =A0 =A0 =A0at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:78= 0) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(Zoo= KeeperWrapper.java:304) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRe= gionServer.java:385) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.reinitializeZooKeeper(= HRegionServer.java:315) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.reinitialize(HRegionSe= rver.java:306) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.j= ava:276) > =A0 =A0 =A0 =A0at sun.reflect.NativeConstructorAccessorImpl.newInstance0(= Native > Method) > =A0 =A0 =A0 =A0at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc= cessorImpl.java:39) > =A0 =A0 =A0 =A0at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst= ructorAccessorImpl.java:27) > =A0 =A0 =A0 =A0at java.lang.reflect.Constructor.newInstance(Constructor.j= ava:513) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.doMain(HRegionServer.j= ava:2472) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.jav= a:2540) > 2010-03-02 19:18:06,689 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to set watcher= on > ZooKeeper master address. Retrying. > 2010-03-02 19:18:07,417 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server cactus209/172.16.1.209:2222 > 2010-03-02 19:18:07,418 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=3D/ > 172.16.1.208:39575 remote > =3Dcactus209/172.16.1.209:2222] > 2010-03-02 19:18:07,421 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > ... > ... > ... > > 2010-03-02 19:23:37,084 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at > 172.16.1.207:60000 that we are up > 2010-03-02 19:23:37,102 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: Unhandled exception. > Aborting... > java.lang.NullPointerException > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java= :459) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619) > 2010-03-02 19:23:37,103 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=3D0.0, regions=3D0, stores=3D0, storefiles=3D0, storefileInd > exSize=3D0, memstoreSize=3D0, usedHeap=3D25, maxHeap=3D2991, blockCacheSi= ze=3D5147928, > blockCacheFree=3D622254440, blockCacheCount=3D0, blockCacheHitRatio=3D0 > 2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping = IPC > Server listener on 60020 > 2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 6 on 60020: exiting > 2010-03-02 19:23:37,104 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2010-03-02 19:23:37,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 0 on 60020: exiting > 2010-03-02 19:23:37,107 INFO org.apache.hadoop.ipc.HBaseServer: Stopping = IPC > Server Responder > 2010-03-02 19:23:37,110 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 2 on 60020: exiting > 2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 3 on 60020: exiting > 2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 4 on 60020: exiting > 2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 5 on 60020: exiting > 2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 7 on 60020: exiting > 2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 8 on 60020: exiting > 2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 1 on 60020: exiting > 2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 9 on 60020: exiting > 2010-03-02 19:23:37,114 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > regionserver/127.0.0.1:60020.compactor exiting > =A0The version we are using is hbase0.20.1.Anyone can give some > suggestions?Thank you very much. > =A0 =A0LvZheng >