hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vrodio...@carrieriq.com>
Subject RE: OfflineMetaRepair?
Date Fri, 06 Jan 2012 01:37:09 GMT
I cp'ed hdfs-site.xml into HBASE_CONF_DIR and was able tun the tool.

The tool found a lot of abandoned regions:

like this one:

12/01/06 01:18:15 ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info
for table TRIAL-DIMENSIONS-1324576713641!  It may be an invalid format or version file.  You
may want to remove hdfs://us01-ciqps1-name01.carrieriq.com:9000/hbase/TRIAL-DIMENSIONS-1324576713641/ff6031e6472d10bac8517314179acb33
region from hdfs and retry.
        at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:292)
        at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
        at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)

There are hundreds of such regions, literally.

Region directories contain only .tmp subdir, like this one:

/hbase/M2M-INTEGRATION-MM_ERRORS-1324575562966/fd480b2c39f7d3333308bf1d9a304510/.tmp

No .regioninfo

These dirs are left-overs of a tables which have been deleted already and they confuse this
tool.  If we delete table we should wipe out the whole directory, is not it?
Is there any scenario which can explain this?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Todd Lipcon [todd@cloudera.com]
Sent: Thursday, January 05, 2012 4:50 PM
To: dev@hbase.apache.org
Subject: Re: OfflineMetaRepair?

Are you sure you have fs.default.name set properly to hdfs://yournn/
in your hbase-site.xml?

You shouldn't *have* to do this, but I bet it will fix the issue.

-Todd

On Thu, Jan 5, 2012 at 4:26 PM, Jonathan Hsieh <jon@cloudera.com> wrote:
> Hey Vlad,
>
> I wrote the tool -- and I've used it to repair a fairly messed up META
> table.  I must of used on a local filesystem copy of META (just got all the
> .regioninfo files in their directory paths), and then shipped the repaired
> version of the .META. dir to the customer.
>
> This is definitely a bug.  FIle the jira and I'll try to fix in the next
> few days.
>
> Jon.
>
> On Thu, Jan 5, 2012 at 4:16 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com>wrote:
>
>> Ted,
>>
>> "fs.default.name" is a standard config property name which is described
>> here:
>> http://hadoop.apache.org/common/docs/current/core-default.html
>>
>> It is not CDH -specific. If you are right than this tool has never been
>> tested.
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ________________________________________
>> From: Ted Yu [yuzhihong@gmail.com]
>> Sent: Thursday, January 05, 2012 4:06 PM
>> To: dev@hbase.apache.org
>> Subject: Re: OfflineMetaRepair?
>>
>> Vlad:
>> In the future, please drop unrelated discussion from bottom of your email.
>>
>> I think what you saw was caused by FS default name not being set correctly.
>> In hbck:
>>        conf.set("fs.defaultFS", conf.get(HConstants.HBASE_DIR));
>> But cdh3 uses:
>>    conf.set("fs.default.name", "hdfs://localhost:0");
>> ./src/test/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
>>
>> You can try adding the following line after line 77 of
>> OfflineMetaRepair.java:
>>    conf.set("fs.default.name", path);
>> and rebuilding hbase 0.90.6 (tip of 0.92 branch)
>>
>> If the above works, please file a JIRA.
>>
>> Thanks
>>
>> On Thu, Jan 5, 2012 at 3:30 PM, Vladimir Rodionov
>> <vrodionov@carrieriq.com>wrote:
>>
>> > 0.90.5
>> >
>> > I am trying to repair .META. table using this tool
>> >
>> > 1.  HBase cluster was shutdown
>> >
>> > Then I ran:
>> >
>> > 2. [name01 bin]$ hbase
>> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
>> > -base hdfs://us01-ciqps1-name01.carrieriq.com:9000/hbase -details
>> >
>> >
>> > This is waht I got:
>> >
>> > 12/01/05 23:23:15 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > HDFS...
>> > 12/01/05 23:23:30 ERROR util.HBaseFsck: Bailed out due to:
>> > java.lang.IllegalArgumentException: Wrong FS: hdfs://
>> >
>> us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-MM_TION-1325190318714/0003d2ede27668737e192d8430dbe5d0/.regioninfo
>> ,
>> > expected: file:///
>> >        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
>> >        at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> >        at
>> >
>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:126)
>> >        at
>> > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284)
>> >        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398)
>> >        at
>> > org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
>> >        at
>> > org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
>> >        at
>> > org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
>> >        at
>> >
>> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
>> >
>> >
>> > Q: What am I doing wrong?
>> >
>> > Best regards,
>> > Vladimir Rodionov
>> > Principal Platform Engineer
>> > Carrier IQ, www.carrieriq.com
>> > e-mail: vrodionov@carrieriq.com
>> >
>> >
>>
>> Confidentiality Notice:  The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited.  If you have received this message in error, please
>> immediately notify the sender and/or Notifications@carrieriq.com and
>> delete or destroy any copy of this message and its attachments.
>>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com



--
Todd Lipcon
Software Engineer, Cloudera

Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.

Mime
View raw message