hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haijun Cao <haijun...@ymail.com>
Subject Re: java.io.IOException: HRegionInfo was null or empty in .META.
Date Tue, 14 Jul 2009 22:40:32 GMT
Ryan,

It worked. thank you!

I have used truncate command on TestTable before, but I then decided to drop the table completely
and start the test from scratch. This problem seem to happen after the hbase server crashed.
I experienced both hbase-1634/HBASE-1638 and this count issue. Running the script in 1638
resolve the NPE problem (Thanks stack), running flush/major_compact meta table resolved the
count issue.

For my education, do you mind explaining the cause/fix for this count issue I am seeing?

Thanks
Haijun




________________________________
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Tuesday, July 14, 2009 3:14:33 PM
Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META.

Try doing this on the shell:

flush '.META.'
major_compact '.META.'

did you use 'truncate' command at any point?

On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<haijuncao@ymail.com> wrote:
>
> Hi
>
> I am running hbase PE test, loaded TestTable with 10M records without any problem, later
hbase crashed (during another sequentialWrite test), after restart, I can't count TestTable,
it always get stuck at 1.1 mil records.
>
>
> I am wondering if anybody has encountered the same problem (data got corrupted after
server crash)? How do you recover? At this point, it is just test data, so it is ok for me
to lose some data. Is there a way to drop/repair the bad region (0001192603)? Does hbase have
a auto-repair tool like fsck?
>
> Your help is greatly appreciated.
>
> Haijun
>
>
> Error stack:
>
> Current count: 1100000, row: 0001099999
> NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server null for region , row '0001192603', but failed after 5 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
> java.io.IOException: HRegionInfo was null or empty in .META.
>
>        from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
>        from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>        from java/lang/reflect/Method.java:597:in `invoke'
>        from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>        from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
>        from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
>        from org/jruby/ast/WhileNode.java:127:in `interpret'
>        from org/jruby/ast/NewlineNode.java:104:in `interpret'
>        from org/jruby/ast/BlockNode.java:71:in `interpret'
>        from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
>        from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:273:in `cacheAndCall'
>        from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
>
>
> I scan the .META. table and noticed that region 0001192603 has no info columns, only
historian columns. I also check the hadoop file system, region 0001192603 has a oldlogfile.log
file in its directory (other regions don't):
>
> Found 3 items
> -rw-r--r--   3 bamboo supergroup        619 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/.regioninfo
> drwxr-xr-x   - bamboo supergroup          0 2009-07-13 16:28 /user/bamboo/hbase/TestTable/954997373/info
> -rw-r--r--   3 bamboo supergroup   39417817 2009-07-13 16:52 /user/bamboo/hbase/TestTable/954997373/oldlogfile.log
>
>
>



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message