hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: java.io.IOException: HRegionInfo was null or empty in .META.
Date Tue, 14 Jul 2009 23:09:34 GMT
It's a known problem with the .META. table involving old deleted rows
and new rows.  There is a brief summary here:

http://issues.apache.org/jira/browse/HBASE-1383

I believe that we are making truncate in the shell actually issue the
major_compact itself too now.


On Tue, Jul 14, 2009 at 3:40 PM, Haijun Cao<haijuncao@ymail.com> wrote:
> Ryan,
>
> It worked. thank you!
>
> I have used truncate command on TestTable before, but I then decided to drop the table
completely and start the test from scratch. This problem seem to happen after the hbase server
crashed. I experienced both hbase-1634/HBASE-1638 and this count issue. Running the script
in 1638 resolve the NPE problem (Thanks stack), running flush/major_compact meta table resolved
the count issue.
>
> For my education, do you mind explaining the cause/fix for this count issue I am seeing?
>
> Thanks
> Haijun
>
>
>
>
> ________________________________
> From: Ryan Rawson <ryanobjc@gmail.com>
> To: hbase-dev@hadoop.apache.org
> Sent: Tuesday, July 14, 2009 3:14:33 PM
> Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META.
>
> Try doing this on the shell:
>
> flush '.META.'
> major_compact '.META.'
>
> did you use 'truncate' command at any point?
>
> On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<haijuncao@ymail.com> wrote:
>>
>> Hi
>>
>> I am running hbase PE test, loaded TestTable with 10M records without any problem,
later hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable,
it always get stuck at 1.1 mil records.
>>
>>
>> I am wondering if anybody has encountered the same problem (data got corrupted after
server crash)? How do you recover? At this point, it is just test data, so it is ok for me
to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have
a auto-repair tool like fsck?
>>
>> Your help is greatly appreciated.
>>
>> Haijun
>>
>>
>> Error stack:
>>
>> Current count: 1100000, row: 0001099999
>> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
>> Exceptions:
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>> java.io.IOException: HRegionInfo was null or empty in .META.
>>
>>        from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
>>        from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
>>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>>        from java/lang/reflect/Method.java:597:in `invoke'
>>        from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
>>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>>        from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
>>        from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
>>        from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
>>        from org/jruby/ast/WhileNode.java:127:in `interpret'
>>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>>        from org/jruby/ast/BlockNode.java:71:in `interpret'
>>        from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in
`call'
>>        from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
>>        from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
>>        from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
>>
>>
>> I scan the .META. table and noticed that region 0001192603 has no info columns, only
historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log
file in its directory (other regions don't):
>>
>> Found 3 items
>> -rw-r--r--   3 bamboo supergroup        619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
>> drwxr-xr-x   - bamboo supergroup          0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
>> -rw-r--r--   3 bamboo supergroup   39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
>>
>>
>>
>
>
>
>

Mime
View raw message