hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irwan Hadi <ihbli...@gmail.com>
Subject Issue with hbase
Date Wed, 25 Jul 2012 06:28:45 GMT
A few weeks ago we installed hbase 0.90.5 on our dev system, to play
around with it, along with Hadoop, and HDFS. All of these are
installed on a Centos 6 server.
Even though we specified hbase to use HDFS in hbase-site.xml, we had a
small configuration typo which then apparently caused it to default
storing data to /tmp/hbase-hadoop/hbase

As expected, after running for a couple days, the tmpwatch cron
cleaned up certain files in /tmp.  The issue wasn't apparent until
hbase had to be cycled, which shows that we have 0 tables.

The question now is:
We still have the tables intact, but lost the META, ROOT, and
.regioninfo under /tmp/hbase-hadoop/hbase/<table_name>/id/<number> ,
are these enough to somehow recover the tables, if so how can we do
this?
We tried hbase hbck already, but it does not want to fix the issue
=======================================================================================
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
Number of Tables: 0
Number of live region servers: 1
Number of dead region servers: 0
Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid =
0x138bca7fff5000a, negotiated timeout = 40000
12/07/25 06:23:11 INFO
client.HConnectionManager$HConnectionImplementation: Closed zookeeper
sessionid=0x138bca7fff5000a
12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Session: 0x138bca7fff5000a closed
12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
12/07/25 06:23:11 DEBUG
client.HConnectionManager$HConnectionImplementation: The connection to
null has been closed.
ERROR: Region file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
on HDFS, but not listed in META or deployed on any region server
ERROR: Region file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
on HDFS, but not listed in META or deployed on any region server
ERROR: Region file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
on HDFS, but not listed in META or deployed on any region server
Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  web01.dev.:52768
  .META. is okay.
    Number of regions: 1
    Deployed on:  web01.dev.:52768
3 inconsistencies detected.
Status: INCONSISTENT
=======================================================================================



We also tried rebuilding the META using the following "hbase
org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base
file:///tmp/hbase-hadoop/hbase/" , but got the following error:
=======================================================================================
12/07/25 05:25:58 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
{NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED =>
1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES
=> [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647',
BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
ERROR: Unable to load region info for table news!  It may be an
invalid format or version file.  You may want to remove
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table news!  It may be an invalid format or version file.  You may
want to remove file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905/.regioninfo
does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
        at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
        at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
{NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META =>
'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'NONE', TTL
=> '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
ERROR: Unable to load region info for table story!  It may be an
invalid format or version file.  You may want to remove
file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table story!  It may be an invalid format or version file.  You may
want to remove file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b/.regioninfo
does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
        at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
        at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
ERROR: Unable to load region info for table topic!  It may be an
invalid format or version file.  You may want to remove
file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
region from hdfs and retry.
12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
table topic!  It may be an invalid format or version file.  You may
want to remove file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
region from hdfs and retry.
java.io.FileNotFoundException: File
file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad/.regioninfo
does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
        at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
        at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
        at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable
to load region info for table news!  It may be an invalid format or
version file.  You may want to remove
file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
region from hdfs and retry.
        at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:292)
        at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
        at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
=======================================================================================

Mime
View raw message