hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1342) Add to filesystem info needed to rebuild .META.
Date Fri, 01 May 2009 21:13:30 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705119#action_12705119
] 

stack commented on HBASE-1342:
------------------------------

Patch coming.  Here is what it does meantime.

It adds a file on HRegion initialization.  If already there, does nothing.  File is named
.regioninfo.  Presumption is that the families list is gotten from schema, not by doing a
listing of whats under the region directory (I've added code to stop a user creating a family
named .regioninfo).

File looks like this:

{code}
^@^@^@^@^@^@^@^@^@^@^@  -ROOT-,,0^@^@^@^@^@^D^F-ROOT-^A^A^@^@^@^B^@^@^@^GIS_ROOT^@^@^@^Dtrue^@^@^@^GIS_META^@^@^@^Dtrue^@^@^@^A^G^Dinfo^@^@^@^H^@^@^@^KBLOOMFILTER^@^@^@^Efalse^@^@^@^KCOMPRESSION^@^@^@^DNONE^@^@^@^HVERSIONS^@^@^@^B10^@^@^@^FLENGTH^@^@^@
2147483647^@^@^@^CTTL^@^@^@^B-1^@^@^@   BLOCKSIZE^@^@^@^D8192^@^@^@     IN_MEMORY^@^@^@^Efalse^@^@^@
BLOCKCACHE^@^@^@^Dtrue^@^@^@^@cخ^O

REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,
TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10',
LENGTH => '2147483647', TTL => '-1', BLOCKSIZE => '8192', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}], INDEXES => []}}
{code}

Its the serialized HRI followed by text version.  Two newlines separates them.

To read the binary part to reconstruct an HRI, just do something like this (using shell):

{code}
durruti:trunk stack$ ./bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Version: 0.20.0-dev, r770006, Fri May  1 10:52:11 PDT 2009
hbase(main):001:0> import org.apache.hadoop.fs.FileSystem
=> Java::OrgApacheHadoopFs::FileSystem
hbase(main):002:0> fs = FileSystem.get(@configuration)
=> #<Java::OrgApacheHadoopFs::LocalFileSystem:0x875de82 @java_object=#<Java::JavaObject:0x314dd379>>
hbase(main):003:0> import org.apache.hadoop.hbase.HRegionInfo
file:/Users/stack/checkouts/hbase/trunk/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:34
warning: already initialized constant HRegionInfo
=> Java::OrgApacheHadoopHbase::HRegionInfo
hbase(main):004:0> hri = HRegionInfo.new 
=> #<Java::OrgApacheHadoopHbase::HRegionInfo:0x12082fd2 @java_object=#<Java::JavaObject:0x45cfa4ed>>
hbase(main):006:0> import org.apache.hadoop.fs.Path
=> Java::OrgApacheHadoopFs::Path
hbase(main):007:0> p = Path.new("/tmp/hbase-stack/hbase/-ROOT-/70236052/.regioninfo/")
=> #<Java::OrgApacheHadoopFs::Path:0x779dfb5a @java_object=#<Java::JavaObject:0x70db5d33>>
hbase(main):009:0> fin = fs.open(p)
=> #<Java::OrgApacheHadoopFs::FSDataInputStream:0x3dbe5cb8 @java_object=#<Java::JavaObject:0x32d7970b>>
hbase(main):011:0> hri.readFields(fin)
hbase(main):012:0> fin.close()
hbase(main):014:0> p hri.toString()
"REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,
TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '10', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', BLOCKSIZE => '8192', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}], INDEXES => []}}"
{code}

That should be some help reconstructing .META.

Let me review the master logging now to make sure good stuff shows there at info level.

> Add to filesystem info needed to rebuild .META.
> -----------------------------------------------
>
>                 Key: HBASE-1342
>                 URL: https://issues.apache.org/jira/browse/HBASE-1342
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>
> From Cosmin up on list:
> {code}
> Hi,
> Is there any way to rebuild the .META. Table with information in HDFS?
> Long story short, the namenode ran out of disk and crashed and we couldn't
> have it load the edits.new file at startup (the edits.new file seemed
> corrupted and it failed to read some Long value) so we removed the edits.new
> file. Apparently the edits file had 10 - 15 entries. Some edit data on
> -ROOT-, some edit data on .META., some data on a table compaction
> (comapction.dir) and some data on some hbase logs. Only last entry seem to
> be bogus, having almost a megabyte of trailing zeroes.
> After renaming edits.new HDFS started ok and HBase data is present. However
> when we started Hbase one table was missing entirely and one table was
> empty. The HBase master report shows Regions On FS: 94, but the report on
> region servers shows only 10 regions on all the region servers.
> Is there any way to cleanup hbase and rebuild the .META. table from HDFS?
> This would be ok even if it implies throwing up some data.
> If we would clean the edits.new file and restart HDFS or if we would
> manually apply the edits would that still make consistent changes to HBase
> data?
> Thanks,
> Cosmin
> {code}
> Lets log the info needed to rebuild your .META. or add things like HRegionInfo made creating
a region as a file under each region directory.  That don't seem hard and the info could be
used rebuilding.  What else would we need?
> Marking as blocker/critical for 0.20.0.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message