hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <lhofha...@yahoo.com>
Subject Re: HBase Issue after Namenode Formating
Date Wed, 17 Oct 2012 22:39:58 GMT
No problem...

We need to fix this in HBase too.There's HBASE-6294, but no patch for it, yet.

-- Lars



________________________________
 From: Pankaj Misra <pankaj.misra@impetus.co.in>
To: "user@hbase.apache.org" <user@hbase.apache.org>; lars hofhansl <lhofhansl@yahoo.com>

Sent: Wednesday, October 17, 2012 3:18 PM
Subject: RE: HBase Issue after Namenode Formating
 
Hi Lars,

Thank you so much Lars for your perfect solution, it worked, you have been a life saver here,
worked like a charm. Thanks again Lars.

Regards
Pankaj Misra


________________________________________
From: lars hofhansl [lhofhansl@yahoo.com]
Sent: Wednesday, October 17, 2012 11:43 PM
To: user@hbase.apache.org
Subject: Re: HBase Issue after Namenode Formating

Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart
ZK)



----- Original Message -----
From: Pankaj Misra <pankaj.misra@impetus.co.in>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Cc:
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I did some more fact finding around this issue, and it seems to me that the Master is not
getting initialized and hence I am not able to create table. I tried connecting via the Java
client as well, which gave me the following exception. I am not sure what is causing Master
to not initialize at all. I am not able to figure out, where am I going wrong, and would request
help. Thanks.

java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
        at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0,
.META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
        at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
        at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
        at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
        at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
        at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)

Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream
exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
        at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009

Thanks and Regards
Pankaj Misra


________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: user@hbase.apache.org
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed
mode, and was running fine.

Today I had to reset and reformat the namenode and clear all the data as well to have a fresh
run, but since then Hbase is not working as expected. For instance I am able to list the tables
but cannot create new one, it reports that Master is initializing, as seen from the exception
below

hbase(main):004:0> create 'testrun','samplecf'

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException:
Master is initializing
        at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

In order to resolve this I have even tried to delete the root so that hbase can recreate it
fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect
via native java client, as it does not get reference of the Master.

Additionally I see the following error in the HBase Master logs

2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode
for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master
has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node.
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master
available. Notifying waiting threads
2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before
this master became active
2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads


Calling for help, as I think I am missing something here.

Thanks and Regards
Pankaj Misra

________________________________

Impetus Ranked in the Top 50 India's Best Companies to Work For 2012.

Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems'
available at http://lf1.me/0E/.


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________

Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.

Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’
available at http://lf1.me/0E/.


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.


________________________________

Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.

Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’
available at http://lf1.me/0E/.


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message