hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhong, Sheng" <Sheng.Zh...@searshc.com>
Subject RE: Hbase startup error: NoNode for /hbase/master after running out of space
Date Tue, 07 Jun 2011 14:37:40 GMT

Joey,

Yeah, the master did not start properly, even though I could see the process and listener
port: 60020, but could not have access. Thank you for looking into this. Thanks, - Andy Zhong

hbase-hadoop-master-XXX.log:
Mon Jun  6 21:53:14 CDT 2011 Starting master on hofvmecln21.ecom.sears.com
ulimit -n 1024
Mon Jun  6 22:08:04 CDT 2011 Starting master on hofvmecln21.ecom.sears.com
ulimit -n 1024

Hbase.log:
2011-06-07 09:33:16,996 WARN  regionserver.HRegionServer - Unable to read master address from
ZooKeeper. Retrying. Error was:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
= NoNode for /hbase/master
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
        ... 5 more
2011-06-07 09:33:17,999 WARN  regionserver.HRegionServer - Unable to read master address from
ZooKeeper. Retrying. Error was:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
= NoNode for /hbase/master
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
        ... 5 more



 

-----Original Message-----
From: Joey Echeverria [mailto:joey@cloudera.com] 
Sent: Tuesday, June 07, 2011 9:30 AM
To: user@hbase.apache.org
Subject: Re: Hbase startup error: NoNode for /hbase/master after running out of space

> 2011-06-06 18:40:11,833 WARN  regionserver.HRegionServer - Unable to 
> read master address from ZooKeeper. Retrying. Error was:

This means that your master didn't start. Can you post the log for the master?

-Joey

On Tue, Jun 7, 2011 at 9:47 AM, Zhong, Sheng <Sheng.Zhong@searshc.com> wrote:
>
> Hey,
>
> The system works perfect until receiving device out of space and more 
> space is added. Hope anyone here could provide help on the hbase issue 
> we got after our of space. I could bring up HDFS via one of post found 
> by manually modifying namenode's /var/hdfs/current/edits file: $ 
> printf "\xff\xff\xff\xee\xff" > edits. However, the issue came to 
> hbase startup. We could see HMaster and HRegion processes running, but 
> could not access Hbase. The errors is described below:
>
> I am appreciated by any help and suggestion. We are even okay to lose 
> recent unprocessed data. P.S: we're using apache hadoop 0.20.2 and 
> hbase 0.20.3, and zookeeper is running via zookeeper-3.2.2 (not 
> managed by Hbase).
>
> Hbase.log:
> 2011-06-06 18:40:11,833 WARN  regionserver.HRegionServer - Unable to 
> read master address from ZooKeeper. Retrying. Error was:
> java.io.IOException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode 
> = NoNode for /hbase/master
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(
> Zo
> oKeeperWrapper.java:394)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr
> Th
> row(ZooKeeperWrapper.java:298)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe
> rv
> er.java:1282)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi
> on
> Server.java:1314)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j
> av
> a:430)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(
> Zo
> oKeeperWrapper.java:390)
>        ... 5 more
>
> Zookeeper.log:
> 2011-06-06 21:48:36,174 - WARN
> [ProcessThread:-1:PrepRequestProcessor@402] - Got exception when 
> processing sessionid:0x23067fc51970002 type:create cxid:0x19a9 
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode = NodeExists
>        at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestP
> ro
> cessor.java:245)
>        at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProces
> so
> r.java:114)
>
>
> Regards,
> Andy (Sheng) Zhong
>
>
> This message, including any attachments, is the property of Sears Holdings Corporation
and/or one of its subsidiaries. It is confidential and may contain proprietary or legally
privileged information. If you are not the intended recipient, please delete it without reading
the contents. Thank you.
>



--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Mime
View raw message