hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin O'dell" <kevin.od...@cloudera.com>
Subject Re: Issue with hbase
Date Wed, 25 Jul 2012 12:42:43 GMT
Irwan,

  Just to be clear: You don't have the META, ROOT, or any .regioninfo
files, but the storefiles are still intact?

If so, you should be able to recreate your tables using the correct HBase
home and bulk load your data back in?  I think that would work, but I will
defer to Stack or Michael on this one.

On Wed, Jul 25, 2012 at 2:28 AM, Irwan Hadi <ihblist2@gmail.com> wrote:

> A few weeks ago we installed hbase 0.90.5 on our dev system, to play
> around with it, along with Hadoop, and HDFS. All of these are
> installed on a Centos 6 server.
> Even though we specified hbase to use HDFS in hbase-site.xml, we had a
> small configuration typo which then apparently caused it to default
> storing data to /tmp/hbase-hadoop/hbase
>
> As expected, after running for a couple days, the tmpwatch cron
> cleaned up certain files in /tmp.  The issue wasn't apparent until
> hbase had to be cycled, which shows that we have 0 tables.
>
> The question now is:
> We still have the tables intact, but lost the META, ROOT, and
> .regioninfo under /tmp/hbase-hadoop/hbase/<table_name>/id/<number> ,
> are these enough to somehow recover the tables, if so how can we do
> this?
> We tried hbase hbck already, but it does not want to fix the issue
>
> =======================================================================================
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
> Number of Tables: 0
> Number of live region servers: 1
> Number of dead region servers: 0
> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> 12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=180000
> watcher=hconnection
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Opening socket connection
> to server localhost/127.0.0.1:2181
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Socket connection
> established to localhost/127.0.0.1:2181, initiating session
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: Session establishment
> complete on server localhost/127.0.0.1:2181, sessionid =
> 0x138bca7fff5000a, negotiated timeout = 40000
> 12/07/25 06:23:11 INFO
> client.HConnectionManager$HConnectionImplementation: Closed zookeeper
> sessionid=0x138bca7fff5000a
> 12/07/25 06:23:11 INFO zookeeper.ZooKeeper: Session: 0x138bca7fff5000a
> closed
> 12/07/25 06:23:11 INFO zookeeper.ClientCnxn: EventThread shut down
> 12/07/25 06:23:11 DEBUG
> client.HConnectionManager$HConnectionImplementation: The connection to
> null has been closed.
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> on HDFS, but not listed in META or deployed on any region server
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> on HDFS, but not listed in META or deployed on any region server
> ERROR: Region
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> on HDFS, but not listed in META or deployed on any region server
> Summary:
>   -ROOT- is okay.
>     Number of regions: 1
>     Deployed on:  web01.dev.:52768
>   .META. is okay.
>     Number of regions: 1
>     Deployed on:  web01.dev.:52768
> 3 inconsistencies detected.
> Status: INCONSISTENT
>
> =======================================================================================
>
>
>
> We also tried rebuilding the META using the following "hbase
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base
> file:///tmp/hbase-hadoop/hbase/" , but got the following error:
>
> =======================================================================================
> 12/07/25 05:25:58 INFO util.HBaseFsck: Loading HBase regioninfo from
> HDFS...
> 12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
> {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED =>
> 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES
> => [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647',
> BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> ERROR: Unable to load region info for table news!  It may be an
> invalid format or version file.  You may want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table news!  It may be an invalid format or version file.  You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905/.regioninfo
> does not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
>         at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 DEBUG util.HBaseFsck: HRegionInfo read: REGION =>
> {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META =>
> 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'NONE', TTL
> => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE
> => 'false'}]}}
> ERROR: Unable to load region info for table story!  It may be an
> invalid format or version file.  You may want to remove
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table story!  It may be an invalid format or version file.  You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/story/89770459431a8e7558d952fc4ee7c60b/.regioninfo
> does not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
>         at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> ERROR: Unable to load region info for table topic!  It may be an
> invalid format or version file.  You may want to remove
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> region from hdfs and retry.
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Unable to load region info for
> table topic!  It may be an invalid format or version file.  You may
> want to remove
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad
> region from hdfs and retry.
> java.io.FileNotFoundException: File
>
> file:/tmp/hbase-hadoop/hbase/topic/cb411aee8e80c70b03b0ebdc574ba8ad/.regioninfo
> does not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
>         at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
> org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable
> to load region info for table news!  It may be an invalid format or
> version file.  You may want to remove
> file:/tmp/hbase-hadoop/hbase/news/349a8254e3fe134eb0ec82693e8ff905
> region from hdfs and retry.
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:292)
>         at
> org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
>         at
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
> 12/07/25 05:25:59 ERROR util.HBaseFsck: Bailed out due to:
>
> =======================================================================================
>



-- 
Kevin O'Dell
Customer Operations Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message